Videos and books

Every resource on this page is free to watch or read, from the author's or the publisher's own page. Where a classic book is not free, we say so and point to its official page. Each lesson of the track also ends with a Watch and read section that names the exact lecture or section.

How to use it. Watch for the picture, read for the proofs, and write code to check every derivative. A good default: one short video before a lesson, then the lesson and its notebook, then one book section from the table at the end of this page.

Video courses #

For the picture #

• 3Blue1Brown, Essence of calculus (course page, YouTube playlist). Short animated chapters on derivatives, the chain and product rules, implicit differentiation, limits, integrals and Taylor series. No proofs, but the pictures behind lessons 01, 09 and 10.
• 3Blue1Brown, Neural networks, chapters 3–4 (course page). Backpropagation first as a picture, then as the chain rule on a tiny network. Watch before lessons 04 and 06.
• Khan Academy, Multivariable calculus (taught by Grant Sanderson; YouTube playlist, course). Short videos on partial derivatives, the gradient, directional derivatives, the Jacobian, the Hessian and quadratic approximation. Pairs with lessons 02–03.

University courses #

• MIT 18.S096, Matrix Calculus for Machine Learning and Beyond (Edelman and Johnson, 2023) (course, YouTube playlist). The closest course to this track: derivatives as linear operators, finite-difference checks, forward mode with dual numbers, reverse mode on computational graphs, Hessians and second derivatives. Pairs with lessons 03–08.
• MIT 18.02, Multivariable Calculus (Auroux, 2007) (course, YouTube playlist). A full first course: partial derivatives, the gradient, the tangent plane, double integrals, polar coordinates and change of variables. Pairs with lessons 02 and 10.
• CMU 10-414/714, Deep Learning Systems (Chen and Kolter) (course). How an autodiff framework is built: computational graphs, reverse mode by extending the graph, and the implementation of a small framework. Pairs with lessons 05 and 08.
• Stanford CS231n (course notes). Lecture 4 and the notes "Backpropagation, intuitions" teach computational graphs and local gradients on small examples. Pairs with lessons 04 and 06.

Code-first #

• Andrej Karpathy, Neural Networks: Zero to Hero (course page, YouTube playlist). The first video builds the reverse-mode engine of lesson 05. "Becoming a Backprop Ninja" writes the full backward pass of an MLP with cross-entropy and batch normalization by hand, as in lesson 06.
• Steve Brunton, Physics Informed Machine Learning (YouTube playlist). How physics enters a loss: PINNs, conservation laws, symmetries. Pairs with lesson 13.

Free books #

Calculus and matrix calculus #

• Gilbert Strang, Calculus (MIT OpenCourseWare, free PDF). A complete one-variable and multivariable textbook. Chapters 2–4 for lesson 01.
• OpenStax, Calculus, volumes 1–3 (volume 1, volume 3). Free and careful, with many worked examples and exercises. Volume 1 for lesson 01, volume 3 (chapters 4–5) for lessons 02 and 10.
• Deisenroth, Faisal and Ong, Mathematics for Machine Learning (free PDF). Chapter 5 covers gradients, Jacobians, backpropagation and Taylor series; chapter 6 covers expectations and the change of variables. Note that the book writes the gradient as a row vector; this track uses a column vector.
• Bright, Edelman and Johnson, Matrix Calculus (for Machine Learning and Beyond) (arXiv). The notes of MIT 18.S096: derivatives as linear operators, the chain rule, forward and reverse mode, second derivatives. Lessons 03–08.

Automatic differentiation and backpropagation #

• Blondel and Roulet, The Elements of Differentiable Programming (arXiv). A modern book on the whole second half of this track: forward and reverse mode (ch. 8), second-order AD and Hessian-vector products (ch. 9), implicit differentiation (ch. 11) and gradient estimators. Lessons 05, 08, 09, 12.
• Baydin, Pearlmutter, Radul and Siskind, "Automatic differentiation in machine learning: a survey" (JMLR 2018; arXiv). AD versus symbolic and numerical differentiation, and forward and reverse mode on one example graph. Lessons 04–05.
• Michael Nielsen, Neural Networks and Deep Learning, chapter 2 (online). The four equations of backpropagation, with proofs. Lesson 06.
• Goodfellow, Bengio and Courville, Deep Learning, chapter 6 (online). §6.5 on back-propagation and other differentiation algorithms. Lesson 06.
• Kevin Murphy, Probabilistic Machine Learning: An Introduction (free PDF). §13.3 on backpropagation and the VJPs of common layers. Lessons 04 and 06.
• Griewank and Walther, Evaluating Derivatives, 2nd edition (SIAM 2008; official page). Not free, but the standard reference on AD, including checkpointing. Find it in a library for lesson 05.

Numerical methods, integrals and probability #

• Driscoll and Braun, Fundamentals of Numerical Computation (free online). §5.4–5.5 on finite differences and the best step size; chapter 5 also covers numerical integration. Lessons 07 and 10.
• Art Owen, Monte Carlo theory, methods and examples (free chapters). Simple Monte Carlo and its error, variance reduction and control variates. Lessons 10 and 12.
• David MacKay, Information Theory, Inference, and Learning Algorithms (free PDF). Chapter 27 on Laplace's method and §28.1 on the evidence and the Occam factor. Lesson 11.
• Rasmussen and Williams, Gaussian Processes for Machine Learning (free PDF). Gaussian integrals at work: the marginal likelihood (§2.2) and the Laplace approximation for classification (§3.4). Lesson 11.
• Peter Olver, The Calculus of Variations (free lecture notes). The Euler–Lagrange equation, boundary conditions, the second variation and several variables. Lesson 13.
• Nocedal and Wright, Numerical Optimization, 2nd edition (Springer 2006). Not free. §2.1 on Taylor's theorem and optimality conditions, chapter 10 on Gauss-Newton and Levenberg-Marquardt. Lessons 03 and 08.

Surveys and tutorials #

• Kolter, Duvenaud and Johnson, Deep Implicit Layers (NeurIPS 2020 tutorial). The implicit function theorem, fixed-point layers, deep equilibrium models and neural ODEs, with code. Lesson 09.
• Papamakarios et al., "Normalizing Flows for Probabilistic Modeling and Inference" (JMLR 2021; arXiv). The change of variables for densities and how flow layers keep the Jacobian determinant cheap. Lesson 10.
• Mohamed, Rosca, Figurnov and Mnih, "Monte Carlo Gradient Estimation in Machine Learning" (JMLR 2020; arXiv). Score-function and pathwise estimators, their conditions and variances, and control variates. Lesson 12.

Short reads and tools #

• Chris Olah, "Calculus on Computational Graphs: Backpropagation" (online). Paths that multiply and add, and why reverse mode is fast. Lesson 04.
• Parr and Howard, "The Matrix Calculus You Need For Deep Learning" (online). Jacobians of vector functions and the vector chain rule, with neural-network examples. Lessons 03–04.
• CS231n, "Backpropagation, intuitions" (notes) and Justin Johnson, "Derivatives, backpropagation, and vectorization" (PDF). Local gradients and the shapes of tensor derivatives. Lessons 04 and 06.
• JAX, "The Autodiff Cookbook" (online). grad, jvp, vjp, jacfwd, jacrev and Hessian-vector products, with their costs. Lessons 05 and 08.
• PyTorch, "Autograd mechanics" (online). How PyTorch records the graph, saves tensors for the backward pass and handles no_grad. Lesson 05.
• Andrej Karpathy, micrograd (GitHub). A complete reverse-mode engine in about 100 lines. Read it after lesson 05.
• Nick Higham, "What is the complex step approximation?" (online). Two pages on derivatives without cancellation. Lesson 07.
• matrixcalculus.org (online). Type a matrix expression and get its derivative. Use it to check hand computations in lessons 03 and 06.

Which lesson pairs with what #

The first choice for each lesson. The Watch and read section at the end of each lesson gives more, with exact section numbers.