Optimization for Machine Learning
Enseignant responsable :
Volume horaire : 24Description du contenu de l'enseignement :
Optimization has long been a fundamental component for modeling and solving classical machine learning problems such as linear regression and SVM classification. It also plays a key role in the training of neural networks, thanks to the development of efficient numerical tools tailored to deep learning. This course is concerned with developing optimization algorithms for learning tasks, and will consist of both lectures and hands-on sessions in Python. The course will begin by an introduction to the various problem formulations arising in machine and deep learning, together with a refresher on key mathematical concepts (linear algebra, convexity, smoothness). The course will then describe the main algorithms for optimization in data science (gradient descent, stochastic gradient) and their theoretical properties. Finally, the course will focus on the challenges posed by implementing these methods in a deep learning and large-scale environment (automatic differentiation, distributed calculations, regularization).
Compétence à acquérir :
- Understand the nature and structure of optimization problems arising in machine learning.
- Select an algorithm tailored to solving a particular instance among those seen in class based on theoretical and practical concerns.
- Experience the practical challenges in implementing an optimization scheme in a learning setting.
Bibliographie, lectures recommandées
- L. Bottou, F. E. Curtis and J. Nocedal. Optimization Methods for Large-Scale Machine Learning. SIAM Review, 2018.
- S. J. Wright and B. Recht. Optimization for Data Analysis. Cambridge University Press, 2022.