Calculus for ML
A derivative is the best linear approximation of a function. Start from slopes and contour plots, then climb to Jacobians and Hessians, the chain rule behind backpropagation, forward- and reverse-mode automatic differentiation, implicit gradients, and the integrals inside probabilistic ML: expectations, densities, Gaussian integrals and the Laplace approximation.
For students, engineers and researchers who want to derive, compute and check every gradient in machine learning. You need school calculus in one variable (we review it in lesson 1) and matrices as linear maps (linear algebra lessons 1 and 3); later lessons also use eigenvalues and positive definite matrices.
Quick recap
The key takeaways of all lessons on one page
All interactive figures
Every figure of the track on one page
Videos and books
The best free courses
Glossary
Every technical term of the track with a short definition
Concept map
How the lessons of this track connect to each other and to other tracks.
Keeps every lesson page and figure of this track in this browser, so you can read it without internet.
Derivatives in one and many variables
Slopes and linear approximations, partial derivatives and the gradient on contour plots, Jacobians, Hessians and Taylor's theorem.
-
01
Derivatives in one variable: slope, linear approximation and the rulesThe derivative is the slope of the tangent line and the best linear approximation of a function. The rules, the derivatives that ML uses, curvature, Taylor polynomials and the first use of all of this, gradient descent in one variable.
60 min
4 figures
Beginner -
02
Partial derivatives, the gradient and directional derivativesFunctions of several variables as surfaces and contour plots. Partial derivatives as slopes of slices, the gradient as a vector in input space, the directional derivative , steepest descent, and why partial derivatives alone do not make a function differentiable.
60 min
4 figures
Beginner -
03
The derivative as a linear map: Jacobians, Hessians and Taylor's theoremNear a point, a smooth map looks like a matrix (the Jacobian), and a smooth loss looks like a quadratic (gradient plus Hessian). This gives the Newton step, the test for minima and saddles, and the Taylor test for gradients.
75 min
5 figures
Intermediate
Chain rule, automatic differentiation and backpropagation
The chain rule as a product of Jacobians, forward and reverse mode, backprop through losses and layers, and how to check a gradient.
-
04
The chain rule: products of Jacobians and computational graphsThe derivative of a composition is the product of the Jacobians, in the right order. Gradients flow back through a map as vector-Jacobian products, contributions add where a value is used twice, and a computational graph lets you run the chain rule by hand.
70 min
4 figures
Intermediate -
05
Automatic differentiation: forward mode and reverse modeAutomatic differentiation computes exact derivatives of a program for a small constant times its cost. Forward mode carries tangents (dual numbers) and gives one Jacobian column per pass; reverse mode records a tape and gives one row per pass, which makes gradients of scalar losses cheap but costs memory.
80 min
6 figures
Intermediate -
06
Backpropagation by hand: MLPs, losses and layer normalizationThe backward pass of a multilayer perceptron, derived step by step and in batch form; the gradients and Hessians of least squares, logistic regression and softmax cross-entropy; the backward pass of layer normalization; and the tricks that keep them finite.
90 min
5 figures
Intermediate -
07
Numerical differentiation and gradient checkingFinite differences have two errors with opposite trends, truncation and rounding, so the best step is not the smallest one. The complex step avoids the subtraction. With float64, a relative error and no kinks, finite differences become a reliable unit test for gradients.
60 min
3 figures
Intermediate
Second-order and implicit derivatives
Hessian-vector products, Gauss-Newton and trace estimation without forming a matrix, and gradients through solvers and fixed points.
-
08
Hessian-vector products, Gauss-Newton and trace estimationCurvature without the matrix. Compute exactly for the cost of a few gradients, use the Gauss-Newton matrix as a curvature model that is never negative, and estimate from random probes.
70 min
4 figures
Advanced -
09
The implicit function theorem and gradients through solversDifferentiate the solution of an equation, a fixed point or an argmin without differentiating the solver. One linear solve at the solution gives the exact gradient, with memory that does not grow with the number of iterations.
70 min
4 figures
Advanced
Integrals for ML
Expectations as integrals, change of variables and densities under transformations, Gaussian integrals and Laplace's method.
-
10
Integrals, expectations and the change of variablesAn integral is a limit of sums, and an expectation is an integral. When you change variables, small volumes scale by |det| of the Jacobian. This one rule gives polar coordinates, the density of a transformed random variable, and normalizing flows.
85 min
5 figures
Intermediate -
11
Gaussian integrals and Laplace's methodThe integral of exp(−½xᵀAx + bᵀx) has a closed form, and it gives the Gaussian normalizer, moments and marginal likelihoods. When the integral is not Gaussian, Gauss–Hermite quadrature computes low-dimensional Gaussian expectations, and Laplace's method replaces a peaked integrand by a Gaussian. This gives Stirling's formula, the Laplace approximation of a posterior and the BIC.
85 min
5 figures
Advanced
Gradients of expectations and functionals
Score-function and pathwise gradient estimators, and the calculus of variations behind energy and physics-informed losses.
-
12
Differentiating expectations: score-function and pathwise gradientsWhen the parameters sit inside the distribution, you cannot backpropagate through a sample. Two unbiased Monte Carlo estimators fix this, the score function (REINFORCE) and the pathwise gradient (reparameterization trick). We derive both, state their conditions, and compare their variances.
75 min
4 figures
Advanced -
13
Calculus of variations, Euler–Lagrange and physics-informed lossesA functional maps a whole function to a number. Its derivative, the first variation, turns "J is smallest" into a differential equation, the Euler–Lagrange equation. This gives the Poisson equation from the Dirichlet energy and two kinds of physics losses for neural networks, energy (Ritz) losses and residual (PINN) losses.
80 min
4 figures
Advanced