This talk will give a brief introduction of deep learning models and the energy they consume for training and inference. We then discuss what methods currently exist for handling their complexity, and how neural network parameter counts could grow by orders of magnitude, despite the end of Moore's law.