Optimization for Machine Learning
Enseignant responsable :
Volume horaire : 24Description du contenu de l'enseignement :
This course delves into the mathematical underpinnings and algorithmic strategies essential for understanding and applying Machine Learning techniques. Central to the course is the exploration of optimization, a pivotal element in contemporary advancements in machine learning. This exploration encompasses fundamental approaches such as linear regression, SVMs, and kernel methods, and extends to the dynamic realm of deep learning. Deep learning has become a leading methodology for addressing a variety of challenges in areas like imaging, vision, and natural language processing. The course content is structured to provide a comprehensive overview of the mathematical foundations, algorithmic methods, and a variety of modern applications utilizing diverse optimization techniques. Participants will engage in both traditional lectures and practical numerical sessions using Python. The curriculum is divided into three parts: The first focuses on smooth and convex optimization techniques, including gradient descent and duality. The second part delves into advanced methods like non-smooth optimization and proximal methods. Lastly, the third part addresses large-scale methods such as stochastic gradient descent and automatic differentiation, with a special focus on their applications in neural networks, including both shallow and deep architectures.
Detailed Syllabus:
1. Foundational Concepts in Differential Calculus and Gradient Descent: - Introduction to differential calculus - Principles of gradient descent - Application of gradient descent in optimization
2. Automatic Differentiation and Its Applications: - Understanding the mechanics of automatic differentiation - Implementing automatic differentiation using modern Python frameworks
3. Advanced Gradient Descent Techniques: - In-depth study of gradient descent theory - Accelerated gradient methods - Stochastic gradient algorithms and their applications
4. Exploring Convex and Non-Convex Optimization: - Fundamentals of convex analysis - Strategies and challenges in non-convex optimization
5. Special Topics in Optimization: - Introduction to non-smooth optimization methods - Study of semidefinite programming (SDP) - Exploring interior points and proximal methods
6. Large-Scale Optimization Methods and Neural Networks: - Techniques in large-scale methods, focusing on stochastic gradient descent - Applications of automatic differentiation in neural networks - Overview of neural network architectures: shallow and deep networks
Bibliographie, lectures recommandées
- Theory and algorithms: Convex Optimization, Boyd and Vandenberghe
- Introduction to matrix numerical analysis and optimization, Philippe Ciarlet
- Proximal algorithms, N. Parikh and S. Boyd
- Introduction to Nonlinear Optimization - Theory, Algorithms and Applications, Amir Beck
- Numerics: Python and Jupyter installation: use only Python 3 with Anaconda distribution.
- The Numerical Tours of Signal Processing, Gabriel Peyré
- Scikitlearn tutorial, Fabian Pedregosa, Jake VanderPlas