Optimization
Training a model means minimizing a loss. Start from gradient descent on a 2D loss surface and build up to momentum, SGD, Adam and learning-rate schedules, constraints and duality, Newton-type methods, nonconvex landscapes and the matrix optimizers used to train large models.
For students, engineers and researchers who know gradients, eigenvalues and a little probability, and want to understand why optimizers work, when they fail, and how to tune them in practice.
Quick recap
The key takeaways of all lessons on one page
All interactive figures
Every figure of the track on one page
Videos and books
The best free courses
Glossary
Every technical term of the track with a short definition
Concept map
How the lessons of this track connect to each other and to other tracks.
Keeps every lesson page and figure of this track in this browser, so you can read it without internet.
Gradient descent and curvature
Loss surfaces, step sizes, convexity, smoothness and the condition number, and how fast gradient descent converges.
-
01
Loss surfaces and gradient descentTraining a model means walking downhill on a loss surface. Why the negative gradient points downhill, how the learning rate decides between slow progress, zig-zags and divergence, and when to stop.
55 min
4 figures
Beginner -
02
Convexity, smoothness and the condition numberConvex functions have no bad local minima. Two parabolas around the loss define the smoothness L and the strong convexity μ; their ratio, the condition number κ = L/μ, decides how fast gradient descent is, and feature scaling changes it.
75 min
5 figures
Intermediate -
03
Convergence rates of gradient descent and line searchHow fast gradient descent converges, in numbers. The factor ρ(η) on quadratics, the linear rate for strongly convex functions, the O(1/k) rate for convex ones, and how a line search (Armijo, Wolfe) chooses the step without knowing L.
70 min
4 figures
Intermediate
Momentum and noise
Why momentum needs about the square root of the condition number of steps, and what minibatch noise does to gradient descent.
-
04
Momentum and Nesterov accelerationHeavy-ball momentum and Nesterov's method need about √κ steps per digit of accuracy instead of κ. A 2×2 matrix per eigenvalue explains why, and also why heavy ball can fail on functions that are not quadratic.
75 min
4 figures
Intermediate -
05
Stochastic gradient descent, the noise ball and batch sizeSGD replaces the full gradient by the gradient on a few examples. Why it stops in a noise ball at a constant step, how a decaying step reaches the minimum, and what the batch size does to the learning rate and to the cost.
75 min
4 figures
Intermediate
Optimizers for training neural networks
AdaGrad, RMSProp, Adam and AdamW exactly, and the learning-rate schedules, warmup and clipping that make training work.
-
06
Adaptive methods: AdaGrad, RMSProp, Adam and AdamWAdaptive methods give every parameter its own step size, from a moving average of its squared gradients. The exact update rules of AdaGrad, RMSProp, Adam and AdamW, why Adam needs bias correction, what it can and cannot fix, and why weight decay is not L2 regularization inside Adam.
70 min
4 figures
Intermediate -
07
Learning-rate schedules, warmup and gradient clipping in practiceHow the learning rate should change during training, why warmup and a final decay help, how to pick the peak learning rate, and how clipping by the global norm protects a run from rare large gradients.
80 min
4 figures
Intermediate
Constraints and nonsmooth problems
Lagrange multipliers and the KKT conditions as a picture, duality, and proximal methods for the lasso and beyond.
-
08
Lagrange multipliers, KKT conditions and dualityAt a constrained minimum a level set of the loss touches the constraint, so the two gradients are parallel. This gives Lagrange multipliers, the KKT conditions for inequality constraints, multipliers as prices, and the dual problem, whose value is a lower bound and a stopping certificate.
95 min
5 figures
Intermediate -
09
Proximal methods (ISTA, FISTA) and ADMMFor a smooth loss plus a simple nonsmooth term, take a gradient step on the smooth part and a proximal step on the rest. This gives soft thresholding and exact zeros, projected gradient, ISTA and FISTA for the lasso, and ADMM, which splits a problem into easy pieces.
75 min
4 figures
Advanced
Curvature, landscapes and modern optimizers
Newton and quasi-Newton methods, saddle points and sharpness in neural networks, and the matrix optimizers K-FAC, Shampoo and Muon.
-
10
Newton, quasi-Newton and Gauss–Newton methodsMethods that use curvature. Newton's method and its local quadratic convergence, damping and affine invariance; BFGS and L-BFGS, which learn the curvature from gradients; Gauss–Newton and Levenberg–Marquardt for least squares; and why deep learning uses only approximations of them.
95 min
5 figures
Advanced -
11
Nonconvex landscapes: saddle points, sharpness and the edge of stabilityWhat gradient descent can promise on a nonconvex loss, why saddle points rarely trap it, how to measure curvature with Hessian-vector products, why training runs at the edge of stability, and what sharp and flat minima mean.
85 min
5 figures
Advanced -
12
Matrix optimizers: K-FAC, Shampoo and MuonOptimizers that treat a weight matrix as a matrix. K-FAC approximates the Fisher by a Kronecker product; steepest descent under the spectral norm gives the orthogonalized gradient UVᵀ; Muon computes it with Newton–Schulz iterations; Shampoo and SOAP reach it through inverse roots. With the evidence as of 2025–2026.
80 min
4 figures
Advanced