Calculus for ML · 57 figures

Interactive figures

Every figure in the track. Each one is a small experiment: drag, slide, and watch the numbers. The link under each figure takes you to the lesson that explains it.

Lesson 1 · Derivatives in one variable

In context: Lesson 1

Lesson 2 · Partial derivatives and the gradient

In context: Lesson 2

Predict first: at which angle of $v$ does the curve on the right have its peak? Then drag the tip of $v$ around the circle. The slope $D_v f$ follows a cosine. Its peak $\|\nabla f\|$ is at the direction of the gradient (blue line), its lowest value $-\|\nabla f\|$ at the opposite direction (orange line), and it is 0 along the contour. The limit check computes $\big(f(x + tv) - f(x)\big)/t$ with a small $t$ and agrees with $\nabla f^\top v$ . The dashed segment drops from the gradient tip onto the line of $v$ : the part of $\nabla f$ along $v$ is $\|\nabla f\|\cos\theta$ .

In context: Lesson 2

Lesson 3 · Jacobians, Hessians and Taylor

Make the square smaller with the δ slider. The image of the square under $f$ (solid) and its image under the linear map $f(x_0) + Df(x_0)h$ (dashed) become the same shape. The right panel is magnified so that the image always fills it. Drag the point $x_0$ and try the other maps. For polar coordinates, move $x_0$ to $r = 0$ : $\det Df = r$ becomes 0, and the parallelogram becomes flat.

In context: Lesson 3

Switch between the tangent plane and the quadratic model, and move $x_0$ . The tangent plane (order 1) and the quadratic model (order 2) touch the surface at $x_0$ . The magenta segments show the error at the corners of the patch. Halve the patch size $\rho$ (from 0.6 to 0.3): the largest error falls by about 4 for the plane and by about 8 for the quadratic model. On the bump, find a point where the model is a dome and one where it is a saddle; on the cubic, find a bowl.

In context: Lesson 3

Press "One step" a few times, then switch to the cubic. On the stretched bowl, the Newton step (dashed arrow) goes straight to the minimum, and gradient descent zig-zags. Change $\eta$ above $0.1 = 2/20$ : gradient descent diverges along $y$ . On the cubic, drag the start point near the saddle at $(0, 0)$ (for example to $(0.3, -0.3)$ ): Newton goes to the saddle, not to the minimum. The readout says when $H$ is indefinite.

In context: Lesson 3

Switch the gradient to "buggy", then the direction to $(1, 0)$ . With the correct gradient, $r_1$ has slope 2 and $r_2$ has slope 3. With the buggy gradient, $r_1$ has slope 1. With $d = (1, 0)$ the bug is invisible, because the error $\tilde g - \nabla f = (0, 1)$ is orthogonal to $d$ . At small $t$ all curves stop at the rounding level of $f$ ; switch to float32 to see this floor rise.

In context: Lesson 3

Lesson 4 · Chain rule and computational graphs

Move the Δx slider toward 0. A small step on the $x$ line becomes a step on the $u$ line and then on the $y$ line. The violet bars are the true changes; the dashed blue bars are the linear predictions $u'(x)\Delta x$ and $y'(u)\,u'(x)\Delta x$ . The magenta gap shrinks faster than $\Delta x$ , so $\Delta y/\Delta x$ tends to the product $y'(u)\,u'(x)$ . Try the other presets: the rates are now different at every point.

In context: Lesson 4

Make the square smaller with the ε slider. The violet square around $x$ is mapped by $f$ (a rotation and scaling near each point) and then by $g$ (a shear). The true images are slightly curved. The dashed blue parallelograms are the Jacobian predictions $Df\cdot\text{square}$ and $Dg\,Df\cdot\text{square}$ ; for small $\varepsilon$ they match the true images. Press show wrong order: the magenta parallelogram $Df\,Dg\cdot\text{square}$ does not match, even for tiny squares. Each panel is zoomed around its point.

In context: Lesson 4

Press Next step and predict each number before it appears. The first 7 steps compute the values (first number under each node). Then the backward run fills the orange adjoints $\bar v_i = \partial f/\partial v_i$ from right to left. Watch $x_2$ and $x_1$ : each is used twice, so it receives two contributions, and the second one is added (magenta). At the end, $\bar x_1 = 5.5$ and $\bar x_2 = 1.716$ are the two partial derivatives. Switch to forward (tangents) to get one directional derivative instead.

In context: Lesson 4

Lesson 5 · Forward and reverse mode autodiff

Press Next step and predict each blue number. Each step computes a value and its tangent together, from left to right. With the seed $\dot x = e_1 = (1, 0)$ the output tangent is $\partial f/\partial x_1 = 5.5$ . Change the seed to $e_2$ and run again to get $\partial f/\partial x_2 = 1.716$ : one forward pass gives one directional derivative, so two inputs need two passes. With $v = (1, 2)$ you get $\nabla f^\top v$ in one pass.

In context: Lesson 5

Step through both phases. In the forward run, each step writes one line of the tape (the table): the operation and its value. In the backward run, the tape is read from the bottom up. Each step sends $\bar v_i$ times the local derivative to the parents. The local derivatives use stored values: $\partial v_2/\partial v_0 = v_{-1} = 2$ needs the value of $x_1$ from the forward run. Contributions to $x_1$ and $x_2$ add because both fan out.

In context: Lesson 5

Compare the three bars (log scale). For the loss shape $1000\to100\to100\to1$ , the output side needs $1.1\cdot10^5$ multiplications and the input side $1.01\cdot10^7$ : about 92 times more. Switch to simulator and the input side wins. Try bottleneck ( $100\to1000\to1\to1000\to100$ ): both ends cost about $10^7$ , but the best order $(J_4J_3)(J_2J_1)$ needs only $2.1\cdot10^5$ . Move the width sliders and predict which bar is shortest before you look.

In context: Lesson 5

Lesson 6 · Backprop: MLPs, losses, layers

Step through one training step. Press Next step. The forward pass fills the nodes with values (blue); the backward pass fills them with error signals $\delta$ (orange). A grey hidden unit is "off" (ReLU with $z \le 0$ ): its slope is 0, so no error passes through it, and its row of $\partial\ell/\partial W^1$ is zero. Move $x$ with the sliders to switch units on and off, and switch to the softmax loss to see $\delta^2 = p - e_y$ .

In context: Lesson 6

Drag the weight vector $w$ or press Run gradient descent. The aqua lines are level sets of the logistic loss $L(w)$ in the plane of weights; the blue arrow is one gradient-descent step $-\eta\nabla L(w)$ with the step $\eta = 4n/\lambda_{\max}(X^\top X)$ of 3.5. The inset shows the data and the decision line $w^\top x = 0$ . On the data that are not separable, the level sets are closed curves around one minimizer $w^*$ . Switch to separable: the level sets open up, gradient descent runs off along a ray, the loss decreases slowly towards 0, and the Hessian eigenvalues (readout) shrink towards 0.

In context: Lesson 6

Compare the full backward pass with a naive one. The grey bars are $\hat g = \gamma \odot g$ , which is what you get (times $1/\sigma$ ) if you treat $\mu$ and $\sigma$ as constants. The orange bars are the true gradient, times $\sigma$ . The true one always sums to 0, and it is orthogonal to $\hat x$ while the row variance is much larger than $\varepsilon = 10^{-5}$ . Move the scale slider down until the variance is close to $\varepsilon$ and watch the cosine readout.

In context: Lesson 6

Change the depth, the activation and the weight scale. The orange line is the norm of the error signal $\|\delta^l\|$ at each layer of a random MLP of width 64, on a log scale; the backward pass starts at the right with $\|\delta^L\| = 1$ . With tanh and scale 1 the norm decreases slowly; with scale 0.5 it vanishes fast. With ReLU, scale $\sqrt 2 \approx 1.41$ (He initialization) keeps it nearly flat, and scale 2 explodes. The sigmoid vanishes even at scale 1.

In context: Lesson 6

Lesson 7 · Finite differences and gradient checks

In context: Lesson 7

Move the probe from right to left. On the right arm the error falls as $h$ falls. At the dashed lines (the optimal steps predicted in section 3) the forward and central errors turn around. Left of them, the digits of $f(x+h)$ and $f(x)$ in the readout are mostly the same: the subtraction cancels the grey digits and keeps only the pink ones, which are rounding noise. Below $h \approx 10^{-16}$ , $x + h$ rounds to $x$ and the difference is 0. The complex step subtracts nothing and stays at full accuracy. Switch to float32: the V moves up and to the right.

In context: Lesson 7

Make the step $h$ smaller than the distance $|x|$ to the kink. While $|x| < h$ , the points $x - h$ and $x + h$ lie on different sides of 0, and the central secant mixes the two slopes: the check fails although the analytic derivative is correct. As soon as $h \le |x|$ , the check passes. Only the ratio $x/h$ matters, so the axes are in units of the step (for example $10^{-6}$ ).

In context: Lesson 7

Lesson 8 · Hessian-vector products

Drag the violet point $x$ and the violet direction handle $v$ . Left: the contour lines of $f$ , the dashed line $x + tv$ and the gradient arrows along it. Right: the path of the gradient's tip as $t$ moves. Its tangent at $t = 0$ is the exact $Hv$ ; the secant through the two points at $t = \pm\eta$ is the finite-difference estimate. Move the $\eta$ slider and watch the error chart: large $\eta$ gives truncation error, tiny $\eta$ gives rounding error. For the cubic preset the gradient is quadratic in $t$ , so the secant is exact up to rounding.

In context: Lesson 8

Move the slider $\theta$ . A one-parameter model $\sin(\theta t)$ is fitted to 12 points. Top: the loss $L(\theta)$ with two quadratic models at $\theta$ : the Newton model (uses the true second derivative $L''$ ) and the Gauss-Newton model (uses $G$ ). Bottom: $L''$ $=$ $G$ $+$ residual term. Notice that $G \ge 0$ everywhere, while $L''$ is negative between the valleys. There the Newton model opens downward and its step goes uphill. Set the noise to 0: at the best fit the residuals vanish and $L'' = G$ .

In context: Lesson 8

Step through the trace of $f(x) = x_1^3 + x_1 x_2^2$ at $x = (1, 2)$ . Rows 1–6 are the forward pass, rows 7–12 the backward pass that computes $\nabla f = (\bar x_1, \bar x_2)$ . In forward-over-reverse, every row also carries a tangent (the dot). The tangents of $\bar x_1, \bar x_2$ are $Hv$ . In reverse-over-reverse, a third sweep goes back up through all 12 rows and fills the second adjoints; the values at $x_1, x_2$ are $Hv$ . Notice that this third sweep needs all 12 rows stored. Choose $v = e_1$ to get the first column of $H$ .

In context: Lesson 8

Press "New probes" a few times. The lines are running averages of $z^\top A z$ for a fixed $30\times30$ symmetric matrix: Rademacher and Gaussian probes. The dashed curves are $\operatorname{tr} A \pm$ one standard deviation from the variance formula; they shrink like $1/\sqrt S$ . This matrix is diagonal-heavy, so the Rademacher band is narrow and the Gaussian band is wide. Switch to the off-diagonal-heavy matrix: both bands are wide and almost equal. For the diagonal matrix, every Rademacher probe gives the exact trace.

In context: Lesson 8

Lesson 9 · Implicit differentiation

Drag the violet point along the curve $F(x, \theta) = 0$ (horizontal axis $\theta$ , vertical axis $x$ ). The blue line has the implicit slope $-F_\theta/F_x$ . The orange secant joins the solutions at $\theta \pm h$ , found by Newton's method from the current $x$ . They agree everywhere except near a fold, where $F_x \to 0$ , the tangent turns vertical and the secant has no second point. Try the S-shaped cubic: it has three solutions for some $\theta$ , and each branch has its own derivative.

In context: Lesson 9

Move $\theta$ and watch the cobweb. The iteration $x_{k+1} = f(x_k, \theta)$ walks between the curve $y = f(x, \theta)$ and the diagonal $y = x$ , and stops at the fixed point. The dashed curve is $f(\cdot, \theta + 0.3)$ : its fixed point moved by about $0.3 \cdot f_\theta/(1 - f_x)$ . The readout compares this implicit slope with the derivative of the $K$ -step iteration (unrolled). For $f = (x^2 + \theta)/4$ , the upper fixed point repels the iteration, but the implicit formula still gives its slope.

In context: Lesson 9

Move $K$ (the number of solver iterations) and the contraction. The chart shows three errors on a log scale: the forward error $|x_K - x^*|$ (grey), the error of the unrolled gradient (backprop through $K$ steps) and the error of the implicit gradient evaluated at $x_K$ . Both gradients improve geometrically; the implicit one is about as accurate as the forward solve. The bars show the memory: unrolling stores $K$ iterates, the implicit gradient stores one point.

In context: Lesson 9

Drag $\log_{10}\lambda$ with the slider. Top: the validation loss $V$ against $\log_{10}\lambda$ with the tangent line whose slope is the hypergradient from one linear solve. Bottom: the weights $w^*_i(\lambda)$ and the norm $\|w^*\|$ . Notice that $\|w^*\|$ always falls as $\lambda$ grows, but a single weight can rise. Press Hypergradient step to take gradient-descent steps on $\log\lambda$ .

In context: Lesson 9

Lesson 10 · Integrals and change of variables

Change $n$ and the method, and watch the error chart. The shaded strips are the Riemann sum; the black curve is $f$ . In the log-log chart, each line is the error of one method against $n$ : left sum, midpoint, trapezoid and the typical Monte Carlo error. The left sum has slope $-1$ (error $\propto 1/n$ ). The midpoint and trapezoid rules have slope $-2$ (error $\propto 1/n^2$ ). Monte Carlo has slope $-1/2$ : it is the slowest in 1-D.

In context: Lesson 10

Drag the violet point, then make the square small. Left: the small square in the $y$ -coordinates. Right: its image under $T$ and, dashed, the image under the linear map $DT(y_0)$ . The readout compares the ratio of the two areas with $|\det DT(y_0)|$ ; they agree better as the side $s$ shrinks. In the polar map, $y = (r, \theta)$ and $\det DT = r$ : the same square gives a bigger image far from the origin. With "linear, det < 0" the map flips the orientation, and the area factor is still the absolute value.

In context: Lesson 10

Move the violet strip and compare the two violet areas. Top: samples of $X$ and its density. Bottom: the same samples after the map, $Y = g(X)$ , with the formula $p_X(x)/|g'(x)|$ on top. The violet strip holds the same probability in both charts, so where its image is wider, it is lower. Then choose a wrong formula in the menu: it no longer fits the histogram, and its integral is not 1. Try $X^2$ with "one branch" and $\sigma(2X)$ , where the density piles up near 0 and 1.

In context: Lesson 10

Add layers with the slider, then drag the violet point $y$ . The shading is the exact density $p_Y$ , computed by inverting the layers. The dots are samples of the base Gaussian pushed through the layers. The dashed path goes back from $y$ through the layers to the base point $z_0$ . The readout lists each layer's $\log|\det J| = s_k$ and subtracts their sum from $\log p_Z(z_0)$ . The sum of $p_Y$ times the cell area stays close to 1: the layers move probability, they do not create it.

In context: Lesson 10

Lesson 11 · Gaussian integrals and Laplace

Drag the violet point to choose a ring, then increase the number of rings $n$ . Top: the bump $e^{-(x^2 + y^2)}$ seen from above (darker = higher) and the selected ring. Bottom: each bar is the volume of one ring, divided by $\Delta r$ ; the black curve is $2\pi r\,e^{-r^2}$ . With more rings the bars fill the area under the curve. The sum tends to $\pi(1 - e^{-9}) \approx \pi$ (the figure stops at $r = 3$ ), so $I \approx \sqrt\pi = 1.7725$ .

In context: Lesson 11

Change $\lambda_1$ , $\lambda_2$ and the angle, then drag the violet peak. The shading is the bump $\exp(-\tfrac12 x^\top A x + b^\top x)$ , with level sets at 1, 2 and 3 "widths". The arrows are the eigenvectors scaled to the widths: $v_1/\sqrt{\lambda_1}$ and $v_2/\sqrt{\lambda_2}$ . The readout splits $Z$ into a width part $2\pi/\sqrt{\det A}$ and a height part $e^{\frac12 b^\top A^{-1} b}$ , and compares $Z$ with a grid sum. Notice: the angle does not change $Z$ ; dragging the peak changes only the height part. Try the preset "λ₂ < 0": then the integral is infinite.

In context: Lesson 11

Increase $n$ , then switch to the zoomed axis. The black curve is the integrand $e^{-n(\varphi - \varphi^*)}$ (scaled to peak 1); its shaded area is the exact integral. The dashed curve is the Laplace Gaussian. In the zoomed axis $u = \sqrt n\,(x - x^*)$ the Gaussian stays fixed and the true curve moves onto it. The lower chart shows the relative error against $n$ , with the slope $-1$ prediction $|c|/n$ . Try the four functions: " $x^2/2$ " is exact for every $n$ ; "Bernoulli" puts Gaussian mass outside $(0, 1)$ ; " $(x^2 - 1)^2/4$ " has two minima, and the error does not go to 0.

In context: Lesson 11

Increase $K$ and switch between the functions. Top: $f$ (black), the shape of the density of $X$ , and the nodes $x_k = \mu + \sqrt2\,\sigma t_k$ ; a bigger dot means a bigger weight. Bottom: the Gauss–Hermite error against $K$ , and the typical Monte Carlo error $\sigma_f/\sqrt K$ for the same number of evaluations. Notice: for $x^6$ the error is exactly 0 from $K = 4$ on ( $2K - 1 \ge 6$ ); for $e^x$ it reaches rounding level near $K = 12$ ; for ReLU it falls slowly.

In context: Lesson 11

Move the total count $S$ and the number of windows $m$ ; then try "no events". Poisson counts in $m$ time windows with total $S$ and the prior $p(\lambda) = e^{-\lambda}$ . The black curve is the exact posterior (a Gamma density). The orange dashed curve is the Laplace approximation in $\lambda$ ; the blue dashed curve is the Laplace approximation in $\theta = \log\lambda$ , drawn as a density in $\lambda$ . Notice: with little data the orange Gaussian is too narrow, is centred at the mode, and puts mass at $\lambda < 0$ . The evidence ratio depends only on $S$ . With $S = 0$ the mode is on the boundary, and only the $\log\lambda$ version works.

In context: Lesson 11

Lesson 12 · Gradients of expectations

In context: Lesson 12

Two pictures of the same samples. In "move the points", each sample is $x_i = \mu + \sigma\epsilon_i$ with fixed noise $\epsilon_i$ : change $\mu$ or $\sigma$ and every sample slides along its own path (grey dots: the positions at $\mu = 0$ , $\sigma = 1$ ). In "reweight the points", the samples stay where they were drawn at $\mu = 0$ , $\sigma = 1$ , and their weights $p_\theta(x_i)/p_{\theta_0}(x_i)$ change. The derivative of the weight at $\theta_0$ is the score. Notice that far from $\theta_0$ a few weights become very large.

In context: Lesson 12

Where does the probability go? When $\theta$ grows by $\Delta\theta$ , the density on $[0, \theta]$ goes down (magenta: the change of the weights, which the score sees) and a new strip $[\theta, \theta + \Delta\theta]$ gets probability (blue: the boundary term, which the score misses). Compare the two Monte Carlo estimates with the exact slope. Try $f = 1$ : the true slope is $0$ , but the score-function estimate is $-1/\theta$ .

In context: Lesson 12

Three estimators of the same gradient. Each histogram shows 4000 estimates of $\partial F/\partial\mu$ for $x \sim N(\mu, 1)$ , $f = x^2$ : pathwise, score function and score function with baseline $b$ . All three have the same mean, the true value $2\mu$ (black line), but the score-function histograms are wide and skewed. Move $b$ and find the narrowest violet histogram; the button $b = b^*$ sets $b^* = \mu^2 + 3$ . Increase $N$ : every histogram gets narrower by a factor $\sqrt N$ and more symmetric, but the order stays.

In context: Lesson 12

Lesson 13 · Calculus of variations

Change N in both views. For $J[u] = \int_0^1 (\tfrac12 u'^2 - u)\,dx$ and a fixed function $u$ , the raw gradient $\partial J_h/\partial U_i$ gets smaller as the grid gets finer. Switch to " $(1/h)\,\partial J_h/\partial U_i$ ": the dots now lie on the curve $\delta J/\delta u = -u'' - 1$ (black). For the minimizer $u = x(1 - x)/2$ the gradient is zero at every grid point.

In context: Lesson 13

Drag the violet handle. It moves a smooth bump $\eta$ (dashed) along the curve and changes $\epsilon$ . The functional is $J[u] = \int_0^1 u'^2\,dx$ with $u(0) = 0$ and $u(1) = 1$ . The chart shows $\varphi(\epsilon) = J[u + \epsilon\eta]$ , a parabola, and its tangent at $\epsilon = 0$ . For $u = x^2$ the slope at $0$ is not zero, so $u = x^2$ is not a minimizer. Switch to $u = x$ : the slope is $0$ for every bump you try.

In context: Lesson 13