learn.mohebi.ai

    Type at least 2 letters. Enter opens the first result, Esc closes.
    Optimization

    Videos and books

    The free courses, lecture series, books and short reads that pair best with this track, with a note on what each one is good for.

    Every resource on this page is free to watch or read, from the author's or the publisher's own page. Where a standard book is not free, we say so and point to its official page. Each lesson of the track also ends with a Watch and read section that names the exact lecture, chapter or paper.

    How to use it. Watch for the picture, read for the proofs, and run code to check both. A good default: one short video before a lesson, then the lesson and its notebook, then one book section from the table at the end of this page. For lessons 06, 07, 11 and 12 the best reading is the original papers; the lessons list them.

    Video courses #

    For the picture #

    The loss surface of a real network and why its gradient tells each weight how to move.
    Open on YouTube · the player loads from youtube-nocookie.com when you press play
    • Steve Brunton, Optimization Bootcamp (YouTube playlist). Short videos on convexity, gradient descent, Newton's method and constrained optimization, with examples from science and engineering. Pairs with lessons 01–03, 08 and 10.
    • Andrew Ng, Improving Deep Neural Networks (DeepLearning.AI, course 2 of the Deep Learning Specialization; YouTube playlist). Ten-minute videos on minibatches, exponentially weighted averages, bias correction, momentum, RMSProp, Adam and learning-rate decay. Pairs with lessons 04–07.
    • Constantine Caramanis, Optimization Algorithms (UT Austin; YouTube playlist). Clear board lectures on convex functions, smoothness and strong convexity, gradient descent rates, proximal methods, ISTA and FISTA. Pairs with lessons 02, 03 and 09.

    University courses #

    The narrow valley, the zig-zag of gradient descent and the factor (κ−1)/(κ+1)(\kappa - 1)/(\kappa + 1).
    Open on YouTube · the player loads from youtube-nocookie.com when you press play
    • MIT 18.065, Matrix Methods in Data Analysis, Signal Processing, and Machine Learning (Strang, 2018) (course, YouTube playlist). Lectures 21 (Newton's method and minimizing step by step), 22 (gradient descent), 23 (momentum) and 25 (stochastic gradient descent, by Suvrit Sra) cover lessons 01–05 and 10.
    • Stanford EE364A, Convex Optimization I (Stephen Boyd) (course page, 2023 lectures on YouTube). The standard course on convex sets and functions, duality and the KKT conditions, with many applications. Pairs with lessons 02 and 08.
    • CMU 10-725, Convex Optimization (Ryan Tibshirani) (course page with notes, Fall 2016 lectures on YouTube). The algorithms side of convex optimization: gradient descent, subgradients, proximal gradient and acceleration, duality, KKT, Newton, quasi-Newton and ADMM. The lecture notes are short and precise. Pairs with lessons 03, 08, 09 and 10.
    • UCLA ECE236C, Optimization Methods for Large-Scale Systems (Lieven Vandenberghe) (course page with slides). Slides only, no videos. Complete proofs of the rates of gradient, proximal gradient, accelerated and Newton-type methods. Pairs with lessons 03, 04, 09 and 10.

    For training neural networks #

    SGD with momentum, Nesterov, AdaGrad, RMSProp and Adam on one slide each, then learning-rate schedules.
    Open on YouTube · the player loads from youtube-nocookie.com when you press play
    • Andrej Karpathy, Let's reproduce GPT-2 (124M) (video). From 2:14:55 to 2:26:20: AdamW settings, gradient clipping, warmup and cosine decay in a real training script. Pairs with lessons 06 and 07.
    • Stanford CS231n course notes: Optimization and Learning and evaluation, with gradient checks, update rules and learning-rate schedules. Pairs with lessons 01 and 04–07.

    Free books #

    Main textbooks #

    • Boyd and Vandenberghe, Convex Optimization (Cambridge, 2004; free PDF from the authors). Chapter 3 (convex functions), chapter 5 (duality, KKT) and chapter 9 (gradient descent, backtracking, Newton's method). The reference for lessons 02, 03, 08 and 10.
    • Sébastien Bubeck, Convex Optimization: Algorithms and Complexity (2015; free on arXiv). A short monograph with clean proofs of the rates of gradient descent, Nesterov's method, mirror descent and stochastic methods. Lessons 03–05 and 09.
    • Nocedal and Wright, Numerical Optimization (Springer, 2nd ed. 2006; not free, publisher page). The standard book on line searches, trust regions, Newton, quasi-Newton (BFGS, L-BFGS) and nonlinear least squares. Lessons 03, 08 and 10 follow its notation.

    Optimization for machine learning #

    • Bottou, Curtis and Nocedal, Optimization Methods for Large-Scale Machine Learning (SIAM Review, 2018; free on arXiv). Why SGD works for machine learning, its rates with fixed and decaying steps, noise reduction, and second-order methods. Lessons 05 and 10.
    • Garrigos and Gower, Handbook of Convergence Theorems for (Stochastic) Gradient Methods (2023; free on arXiv). One theorem and one short proof for each combination of method (GD, SGD, proximal, momentum) and assumption (smooth, strongly convex, PL). Lessons 02, 03, 05 and 09.
    • Parikh and Boyd, Proximal Algorithms (2014; free from the authors) and Boyd et al., Distributed Optimization and Statistical Learning via ADMM (2011; free from the authors). Lesson 09.
    • Goodfellow, Bengio and Courville, Deep Learning (MIT Press, 2016; free online). Chapter 8, "Optimization for training deep models": local minima and saddle points, momentum, adaptive methods. Lessons 05, 06 and 11.
    • Simon Prince, Understanding Deep Learning (MIT Press, 2023; free PDF from the author). Chapter 6 (fitting models: SGD, momentum, Adam) with clear figures. Lessons 04–06.
    • Madsen, Nielsen and Tingleff, Methods for Non-Linear Least Squares Problems (DTU, 2004; free PDF). Gauss–Newton and Levenberg–Marquardt with the gain-ratio rule used in lesson 10.

    Short reads #

    Which lesson pairs with what #

    Lesson Watch Read
    01 Loss surfaces and gradient descent 3Blue1Brown, Deep Learning ch. 2; 18.065 lecture 22 Goh, Why Momentum Really Works (first section); CS231n optimization notes
    02 Convexity, smoothness, conditioning Brunton, Convexity 101; Caramanis 2.3 and 3.2; EE364A lecture 3 Boyd–Vandenberghe ch. 3; Garrigos–Gower §2
    03 GD rates and line search 18.065 lecture 22; 10-725 lecture 5 Bubeck §3.1–3.4; Boyd–Vandenberghe §9.2–9.3; Nocedal–Wright ch. 3
    04 Momentum and Nesterov 18.065 lecture 23; Ng C2W2L06 Goh, Why Momentum Really Works; Bubeck §3.5–3.7
    05 SGD and batch size 18.065 lecture 25 (Sra); StatQuest SGD Bottou–Curtis–Nocedal §3–4; McCandlish et al. (2018)
    06 AdaGrad, RMSProp, Adam, AdamW Ng C2W2L03–L08; CS231n lecture 7 Kingma–Ba (2015); Loshchilov–Hutter (2019); Ruder (2016)
    07 Schedules, warmup and clipping Karpathy, GPT-2 (2:14:55–2:26:20); Ng C2W2L09 Hägele et al. (2024); Zhang et al. (2020); Tuning Playbook
    08 Lagrange multipliers, KKT, duality Khan Academy on Lagrange multipliers; EE364A lecture 8 Boyd–Vandenberghe ch. 5; Tibshirani, KKT notes
    09 Proximal methods and ADMM Caramanis 5.1 and 5.4; 10-725 lectures 8 and 23 Parikh–Boyd; Boyd et al., ADMM §3
    10 Newton, BFGS and Gauss–Newton 18.065 lecture 21; 10-725 lecture 14; Harvard AM205 4.9 and 1.8 Boyd–Vandenberghe §9.5; Nocedal–Wright ch. 6–7, 10; Madsen–Nielsen–Tingleff
    11 Saddles, sharpness and stability Welch Labs, How Models Learn part 1; Jeremy Cohen, CMU AI Seminar Cohen et al. (2021); Lee et al. (2016); Foret et al. (2021); Goodfellow et al. ch. 8
    12 K-FAC, Shampoo and Muon Jia-Bin Huang on Muon; Bernstein, Depths of First Order Optimization Bernstein–Newhouse (2024); Jordan, Muon blog; Martens–Grosse (2015)