Optimization for Data Science

• explain fundamental concepts in convex analysis, including convex sets and functions, conjugate functions, and subdifferentiability
• characterize optimization problems based on their mathematical properties (e.g., smooth/nonsmooth, convex/nonconvex, continuous/discrete, unconstrained/constrained) and recognize the implications of these properties on problem-solving approaches
• formulate optimization problems and derive optimality conditions and Lagrange dual problems
• explain how changes in constraints impact the optimal solution
• apply surrogation and convex relaxation techniques
• explain explicit and implicit regularization techniques
• analyze and apply stochastic optimization methods
• implement scalable algorithms for solving optimization problems within data science
• apply hyperparameter tuning strategies
• compare various optimization algorithms and evaluate trade-offs in terms of convergence speed, robustness, and scalability

The role of optimization in data science (model assumptions, approximation-estimation-optimization trade-off, regularization techniques). Optimization theory (global and local optimizers, surrogate models, optimality conditions), convex analysis (convex sets and functions, conjugate function, subdifferential), and problem transformations. Optimization problems and their properties (linear/convex/nonlinear, smooth/nonsmooth, continuous/discrete, unconstrained/constrained). Surrogation and relaxation techniques. Optimization algorithms (first-order methods, proximal operator, quasi-Newton methods, Newton’s method, stochastic methods, Bayesian optimization) and convergence analysis. Randomized numerical linear algebra and implementation of scalable solution approaches for specific optimization problems encountered in data science.

01005/01006/01020/01025/01034/01035/01037/02402/02403/02601, coursework in calculus and linear algebra (e.g. 01005/01006/01020 and 01025/01034/01035/01037), statistics (e.g., 02402/02403), and numerical algorithms (e.g., 02601). Proficiency in a high-level programming language such as Python, Julia, or MATLAB is expected. While not essential, coursework in probability theory (e.g., 02405) and mathematical optimization (e.g., 02610) is recommended.

Lectures and assignments