Optimization · 52 figures

Interactive figures

Every figure in the track. Each one is a small experiment: drag, slide, and watch the numbers. The link under each figure takes you to the lesson that explains it.

Lesson 1 · Loss surfaces and gradient descent

Drag the violet point and the small grey handle $u$ . The violet curve is the level set through the point, the orange arrow is $\nabla f(x)$ and the blue arrow is $-\nabla f(x)$ , the direction gradient descent takes. Notice that the gradient always crosses the level set at a right angle, so in the narrow valley it points mostly across the valley, not toward the minimum (the black star). Turn $u$ and watch $\nabla f(x)^\top u$ : it is largest along $\nabla f$ and zero along the level set.

In context: Lesson 1

Drag the blue start point and change the step. The slider gives $\eta$ in units of $2/L$ , where $L$ is the largest curvature (here $L = \kappa$ ). The blue path is gradient descent; the chart shows $f(x_k) - f^\star$ on a log scale, with the theory bound dashed. Use the Try buttons: "too small" crawls, "best" uses $\eta = 2/(1 + \kappa)$ , "just above $2/L$ " diverges in the steep direction. Then raise $\kappa$ , and switch to the rotated quadratic: the valley no longer follows an axis, and the picture is the same.

In context: Lesson 1

Click the chart or move the slider to pick $\eta$ . The blue curve is the loss after 50 steps, $f(x_{50})$ , for every step size, from the same start $(2, 1)$ . It is flat for tiny $\eta$ , has its lowest point near the best step $2/(\mu + L)$ (grey dashed line), and then rises sharply at the stability limit $2/L$ (yellow line). The small contour plot shows the chosen run. Switch $\kappa$ : the cliff moves with $2/L$ , and the good region gets narrower as $\kappa$ grows.

In context: Lesson 1

Lesson 2 · Convexity, smoothness, conditioning

Drag the violet chord ends $a$ , $b$ and the aqua tangent point $t$ . Where the chord lies below the graph, or the graph falls below the tangent, the gap is shaded magenta. For the double well both tests fail for some choices. For $\tfrac12 x^2$ , the logistic loss and $x^4/4$ no choice fails: they are convex. For $|x|$ the chord test passes everywhere, but at $t = 0$ there is no tangent line (no derivative), only many supporting lines.

In context: Lesson 2

Drag the point on the graph and move the sliders $L$ and $\mu$ . The orange parabola has curvature $L$ , the blue parabola curvature $\mu$ ; both touch $f$ at the point. Where $f$ leaves the band, the gap is shaded magenta. For $\tfrac12x^2 + \log\cosh x$ , find the smallest $L$ and the largest $\mu$ that work at every point (answer: $L = 2$ , $\mu = 1$ ). Then try $x^4/4$ : whatever $L$ you choose, some point far out breaks the upper bound.

In context: Lesson 2

Move the eigenvalues $\lambda_1$ , $\lambda_2$ and the angle $\varphi$ . The blue ellipses are level sets of $\tfrac12 x^\top H x$ ; read $\mu$ , $L$ , $\kappa$ and the axis ratio $\sqrt\kappa$ in the panel. Think of $H$ as $\frac1n X^\top X$ , the covariance of two centered features. Press Standardize the features: the aqua dashed ellipses show the loss after each feature is divided by its standard deviation. With $\varphi = 0$ they are circles ( $\kappa = 1$ ). With $\varphi = 45°$ nothing changes: the features are correlated, and rescaling each feature alone cannot remove a correlation.

In context: Lesson 2

Move $t$ along the segment from $\theta_A$ ( $t = 0$ ) to $\theta_B$ ( $t = 1$ ). The two ends compute the same function and have loss 0. The violet curve is the loss along the segment; a convex loss would stay on or below the dashed chord, here at 0. In the two-unit tanh network the midpoint has equal hidden units whose output weights cancel, so it outputs 0 everywhere. Switch to $\tfrac12(uv - 1)^2$ to see the segment leave the set of minimizers (aqua).

In context: Lesson 2

Lesson 3 · GD rates and line search

Press Play, then try the three presets (the slider $\eta \cdot L/2$ is the step in units of $2/L$ ). The contours belong to $f = \tfrac12 x^\top A x$ with $\mu = 1$ and $L = \kappa$ ; the dashed lines are the eigen-directions $q_1$ (flat) and $q_2$ (steep). The blue path is gradient descent. The bars show the two error components with their sign. With $\eta = 1/L$ the steep component dies in one step, and the flat one shrinks by only $1 - 1/\kappa$ per step. With $\eta = 0.97 \cdot 2/L$ the steep component flips sign at every step (zig-zag). On the log chart, the blue curve $\|e_k\|$ always ends up parallel to the slower dashed line.

In context: Lesson 3

Drag on the chart to move the step size $\eta$ (or use the slider $\eta \cdot L/2$ , the step in units of $2/L$ ). The aqua V is $|1 - \eta\mu|$ , the magenta V is $|1 - \eta L|$ , and the thick dark curve is their maximum $\rho(\eta)$ . Above the dashed line $\rho = 1$ there is no convergence: this starts at the yellow line $2/L$ . The lowest point of the thick curve is the best step $\eta^\star = 2/(\mu + L)$ . Change $\mu$ and $L$ : the best factor depends only on $\kappa = L/\mu$ .

In context: Lesson 3

Press Next trial to run backtracking one trial at a time. The black curve is $\varphi(\alpha)$ , the grey dashed line is the tangent at 0, and the yellow line is the Armijo line. The violet circles are the trials $\alpha_0, \tau\alpha_0, \dots$ ; the filled one is accepted. The bands below show where Armijo holds, where the curvature condition holds, and where both hold (the Wolfe set); $c_2$ is the curvature constant defined below. Try the wavy slice: the Wolfe set has several pieces. Move $c_1$ toward 0.5 and see the Armijo set shrink.

In context: Lesson 3

Compare the fixed step (blue) with Armijo backtracking (violet, dashed), both started at $(-1.2, 1)$ . Try the three presets for $\eta$ . With $\eta = 0.0021$ , just above $2/\lambda_{\max}$ at the minimizer, the fixed step never settles: it stays near the minimizer and bounces across the valley. Look at the long violet jump: with $\alpha_0 = 1$ , backtracking sometimes accepts a long step across the bend of the valley, because it lowers $f$ enough. Then switch $\alpha_0$ to "2 × last step" and compare the number of function evaluations.

In context: Lesson 3

Lesson 4 · Momentum and Nesterov

Press Play and watch the three paths: gradient descent (blue), heavy ball (orange) and Nesterov (violet), each with its theory parameters. The chart shows $\|x_k - x^\star\|/\|x_0 - x^\star\|$ ; the dashed lines are the theory factors $\rho^k$ . Heavy ball first overshoots far across the valley, then wins by a wide margin. Increase $\kappa$ and see the gap grow. With same η and β all three share one step (the slider $\eta \cdot L$ is the step in units of $1/L$ ) and one $\beta$ . Then choose Rosenbrock: heavy ball with the parameters that are best near the minimizer diverges, while Nesterov converges.

In context: Lesson 4

Move $\eta\lambda$ and $\beta$ , or use the presets. The orange dots are the heavy-ball iterates $x_k$ , the faint blue line is gradient descent with the same $\eta\lambda$ , and the dashed lines are $\pm\rho^k$ , where $\rho$ is the size of the largest root. In the under-damped case the roots are complex and their size is exactly $\sqrt\beta$ , whatever $\eta\lambda$ is. Find the critically damped point for $\beta = 0.5$ : it is at $\eta\lambda = (1 - \sqrt{0.5})^2 \approx 0.086$ .

In context: Lesson 4

The roots in the complex plane. For many eigenvalues $\lambda$ between $\mu$ and $L$ , the figure draws both roots of $z^2 - (1 + \beta - \eta\lambda)z + \beta$ as orange dots. The solid circle is the unit circle (outside it: divergence); the dashed circle has radius $\sqrt\beta$ . With Polyak's parameters every root lies on the dashed circle. Make $\beta$ smaller: the roots for $\lambda$ near $\mu$ leave the circle along the real axis, and the largest root size grows. Press β = 0 to see plain gradient descent, whose roots $1 - \eta\lambda$ all lie on the real axis.

In context: Lesson 4

Steps against $\kappa$ , on log-log axes. Move the marker. Gradient descent (best step) lies on a line of slope 1: steps $\propto \kappa$ . Heavy ball and Nesterov lie on lines of slope $1/2$ : steps $\propto \sqrt\kappa$ . The dashed lines are the formulas $\ln 10^6/\ln\frac{\kappa + 1}{\kappa - 1}$ and $\ln 10^6/\ln\frac{\sqrt\kappa + 1}{\sqrt\kappa - 1}$ . Conjugate gradient (grey) is never worse than heavy ball and is often better, because it adapts to the actual eigenvalues (folded part at the end of section 3).

In context: Lesson 4

Lesson 5 · SGD and batch size

Drag the blue point $x$ from the right into the shaded region. Each arrow is the step direction of one example: blue if it agrees with the full step $-f'(x)$ (thick dark arrow), magenta if it points the other way. Outside the data range all arrows agree. Inside, they disagree, and the full step becomes short. The chart below runs SGD with one random example per step from the current $x$ : it moves fast at first, then jumps around $x^\star$ inside the dashed band $x^\star \pm \sqrt{\eta\sigma^2/(2-\eta)}$ (derived in section 3).

In context: Lesson 5

Many SGD runs from one start. Each faint blue line is one SGD run; the dark line is gradient descent with the same step. Press Play. The runs first follow gradient descent, then stop in a cloud: the noise ball. Change the step $\eta$ and the batch $B$ and watch the cloud size: it scales like $\eta/B$ . The yellow dashed circle has the radius of the bound $\sqrt{\eta\sigma^2/(\mu B)}$ ; the dotted dark circle has the radius $\sqrt{m_\infty}$ , where $m_\infty$ is the exact floor. Then choose decaying: the cloud keeps shrinking.

In context: Lesson 5

Move the batch size $B$ and watch the two costs. The blue curve is the number of steps $S/S_{\min} = 1 + B_{\text{noise}}/B$ ; the violet curve is the number of examples $E/E_{\min} = 1 + B/B_{\text{noise}}$ . Below $B_{\text{noise}}$ (yellow line) the steps fall almost like the dashed line of perfect linear scaling, at almost no extra cost in examples. Above it, the steps flatten and the examples grow. Change $B_{\text{noise}}$ : the whole picture slides along the axis.

In context: Lesson 5

Count work, not steps. Least squares with $d = 5$ and $\kappa \approx 11$ . The x-axis is the number of example gradients. Gradient descent (dark, one dot per pass) needs a full pass for one step; minibatch SGD makes progress after a few hundred examples, then stops at its noise floor. Switch $n$ from 10,000 to 100,000: the GD curve moves right by a factor of 10, the SGD curve does not move. Choose decaying to see SGD go below its floor, slowly. The yellow line marks one pass.

In context: Lesson 5

Lesson 6 · AdaGrad, RMSProp, Adam, AdamW

Slide $\beta$ and watch the two averages of the noisy gradients (grey dots). The orange EMA without correction starts near 0 and needs about $1/(1-\beta)$ steps to catch up; the aqua corrected average is right from the first step. With jump at t = 50, a large $\beta$ reacts slowly: long memory means smooth but late. The bars show the weight of each past gradient in the last average; the dashed line marks $1/(1-\beta)$ steps back.

In context: Lesson 6

Race four optimizers. Gradient descent (blue), momentum (orange), RMSProp (aqua, dashed) and Adam (aqua, solid) start from the same point (drag it). On the axis-aligned quadratic, Adam and RMSProp go almost straight to the minimum. Switch to rotated 45°: with the same learning rates they lose their advantage and stall, while gradient descent is unchanged. Then try the saddle (Adam escapes in a few steps, because it rescales the tiny gradient along $x_2$ ), Rosenbrock and the double well, and turn on gradient noise. Each optimizer has its own learning-rate slider; Default learning rates resets them.

In context: Lesson 6

Type a gradient (or pick an example) and read the first three steps of each method, in units of $\eta$ . SGD copies the gradient: steps of $1000\eta$ and $0.001\eta$ . RMSProp (light aqua) starts with $10\eta$ ; Adam without correction (grey) starts with about $3.16\eta$ ; Adam (aqua) moves by $\eta$ in both coordinates. With (100, 0.01), then flip, the coordinate whose sign flips moves back by only 5% of $\eta$ . Set $\varepsilon = 1$ to see Adam lose its adaptivity.

In context: Lesson 6

Two groups of weights, no reason to move. All weights start at 1 and see zero-mean gradient noise: group A with RMS 10 (solid lines), group B with a smaller RMS (dashed). Only the decay moves them on average. With AdamW (aqua) both groups shrink alike, like $(1 - \eta\lambda)^t$ . With Adam + L2 (magenta), group A barely shrinks and group B shrinks fast, like $\exp(-\eta\lambda t/\text{RMS})$ . Change the RMS of group B and the decay $\lambda$ .

In context: Lesson 6

Lesson 7 · Schedules, warmup and clipping

In context: Lesson 7

Compare the curves. All runs first fall to the same plateau: the noise ball for $\eta_{\max}$ . The constant run stays there. The WSD runs stay there too, until the cooldown starts (dashed line), and then drop fast. The cosine run decays slowly from the start. The cosine planned for $2T$ and stopped at $T$ (violet, dotted) is still high. Set σ = 0 (no noise): now the constant learning rate ends lowest, because decay only slows progress when there is no noise to remove.

In context: Lesson 7

Run the test on a least-squares problem. The smoothed loss falls, reaches a minimum and blows up. The yellow dashed line is $2/\lambda_{\max}$ , the stability limit of full-batch gradient descent on this problem. The blow-up comes a little later, because divergence needs a few steps to show and the moving average lags. Switch the features to the same scale: $\lambda_{\max}$ drops and the whole curve moves right.

In context: Lesson 7

Lesson 8 · Lagrange multipliers, KKT, duality

Drag the black point along the yellow constraint curve and find where $f$ is smallest. The blue arrow is the direction of $\nabla f$ , the yellow arrow the direction of $\nabla h$ , and the magenta arrow the downhill direction along the curve. The dashed blue curve is the level set of $f$ through the point. At the minimum the magenta arrow vanishes, the angle between the gradients is $0°$ or $180°$ , and the dashed level set touches the curve. The chart shows $f$ along the curve. Try the circle: it has a minimum and a maximum, and both satisfy the condition.

In context: Lesson 8

Move the slider to walk along the lifted curve, and drag the figure to rotate it. The blue surface is $f(x) = x_1^2 + x_2^2$ ; the yellow curve on the floor is the constraint, and the vertical lines connect it to its lift. The bottom of the bowl, $(0, 0)$ , is not on the curve. Press Go to the lowest point: for the line, it is $(0.5, 0.5)$ with $f = 0.5$ , the same answer as the calculation above.

In context: Lesson 8

Move $c$ to slide the constraint line. The left plot shows the contours of $f$ and the constrained minimizer $x^\star$ ; the right plot shows the optimal value $p^\star(c)$ (black) and its tangent line, whose slope is $-\lambda$ . For the inequality $x_1 + x_2 \le c$ , the multiplier is $\lambda = 2$ at $c = 1$ and becomes 0 when $c \ge 3$ : then the unconstrained minimizer $(2, 1)$ is feasible and the constraint costs nothing. Switch to the equality to see a multiplier $\nu$ that changes sign.

In context: Lesson 8

Drag the blue point $a$ , the unconstrained minimizer of $f_0(x) = \|x - a\|^2$ , inside and outside the yellow feasible set $f_1(x) \le 0$ . Inside, $\lambda = 0$ and $x^\star = a$ . Outside, $x^\star$ (black star) is on the boundary, and the push of the wall $-\lambda\nabla f_1$ (yellow arrow) cancels the pull of $f_0$ , $-\nabla f_0$ (blue arrow). The four KKT conditions are listed with ticks. In the mode choose λ yourself, the orange point minimizes $f_0 + \lambda f_1$ for your $\lambda$ : too small and it is infeasible, too large and complementary slackness fails.

In context: Lesson 8

Move $\lambda$ . Each thin line is $f_0(x) + \lambda f_1(x)$ for one fixed $x$ : yellow for feasible $x$ (slope $f_1(x) \le 0$ ), grey for infeasible $x$ . Their lower envelope is the dual function $g(\lambda)$ (thick blue), and the orange line belongs to the $x$ that attains the minimum. The blue dot never rises above the dashed line $p^\star$ . In the convex problem the best bound reaches $p^\star = 4$ at $\lambda = 4$ . In the nonconvex problem the best bound is $-3$ while $p^\star = -1$ : a gap of 2.

In context: Lesson 8

Lesson 9 · Proximal methods and ADMM

Move $v$ and $\eta$ . Left: the two parts of the prox objective, $g(u)$ (violet) and $\frac{1}{2\eta}(u - v)^2$ (blue, dashed), and their sum (black). The orange point is the minimizer $p = \operatorname{prox}_{\eta g}(v)$ . Right: the prox map $v \mapsto p$ with the identity dashed. For $g = |u|$ , the yellow dead zone $[-\eta, \eta]$ is sent to exactly 0: the kink at 0 holds the minimizer there. For $\tfrac12 u^2$ the map is a line through 0 with slope $1/(1 + \eta)$ ; for the indicator it is a clip.

In context: Lesson 9

Drag the blue start point and press Play. Each step is a gradient step (blue, dashed), which may leave the set, followed by a projection back onto the yellow set $C$ . The iterates reach the boundary and then slide along it to the constrained minimizer $x^\star$ (black star). The grey star is the unconstrained minimizer, outside $C$ . At $x^\star$ , $-\nabla f(x^\star)$ (grey arrow) points straight out of $C$ : no feasible direction goes downhill. Try the disc, and a step $\eta L$ above 2.

In context: Lesson 9

Drag the start point, move $\lambda$ , and press Play. ISTA (blue) and FISTA (violet) minimize a 2D lasso whose contours (grey) have kinks on the axes. At $\lambda = 0.2$ the minimizer is $x^\star = (2, 0)$ : both methods reach $x_2 = 0$ exactly after a finite number of steps. FISTA overshoots past $x_1 = 2$ and its curve in the chart goes up and down, but it ends far ahead of ISTA. The dashed lines are the two worst-case bounds. The worked example has $\kappa = 3$ ; there acceleration helps little.

In context: Lesson 9

Move $\lambda$ and watch which coefficients are exactly 0. The curves (colours 1, 2, 3, 4, 5) are the lasso coefficients of a small data set (20 samples, 5 standardized features), computed in your browser by coordinate descent. Right of $\lambda_{\max}$ all are 0. Coefficient 3 enters first, grows, shrinks, is exactly 0 again for $\lambda$ between about $0.02\,\lambda_{\max}$ and $0.07\,\lambda_{\max}$ , and then turns negative: a coefficient can leave the model when correlated features take over.

In context: Lesson 9

Lesson 10 · Newton, BFGS and Gauss–Newton

Press Step several times. The grey curve is $f$ ; the dashed magenta parabola is the local model $m_k$ at the current point, and the step goes to its stationary point. On $x^4/4 - x$ from $x_0 = 2$ , the log chart of $|x_k - 1|$ falls faster and faster: the exponent doubles (quadratic convergence). Then try the presets. From $x_0 = 0.1$ the curvature is tiny and the jump goes far away. On the double well from $x_0 = 0.3$ the curvature is negative: the model opens downward and pure Newton moves towards the maximum at 0. Switch to "damped" to see the fixes of section 3.

In context: Lesson 10

Drag the point $x$ in the left plot. On the left are the elliptic level sets of $f = \tfrac12 x^\top A x$ . The blue arrow is a gradient step $-\tfrac1L\nabla f$ : it is perpendicular to the level set, so it does not point at the minimum. The magenta arrow is the Newton step: it points straight at the minimum. On the right the same point is drawn in the coordinates $z = A^{1/2}x$ ; the grey grid is the left grid after the change of variables. The level sets are circles, and the gradient step $-z$ is the magenta arrow again. Set $\kappa = 1$ : both arrows on the left point the same way.

In context: Lesson 10

Pick a surface, then drag the large start dot. Blue: gradient descent (best fixed step on the quadratic, backtracking elsewhere). Magenta: pure Newton. Violet: damped Newton. Dashed magenta: BFGS. On the rotated quadratic, Newton needs one step for every $\kappa$ , while gradient descent needs more steps as $\kappa$ grows. On Rosenbrock, pure Newton needs 6 steps and gradient descent about 19,000. On the double well from the default start, pure Newton stops at the saddle point $(0, 0)$ : the Hessian there is not positive definite. Damped Newton (which also fixes negative curvature) and BFGS reach a minimum.

In context: Lesson 10

Press Step twice. Left: BFGS with exact line searches on $f = \tfrac12 x^\top A x$ , $\kappa$ from the slider, starting from $H_0 = I$ . Right: the ellipse $\{H_k u : \|u\| = 1\}$ of the current estimate (magenta) and of the true inverse Hessian $A^{-1}$ (dashed). After one step the two ellipses pass through the same point $H_1\hat y = A^{-1}\hat y$ (orange arrow, $\hat y = y/\|y\|$ ): this is the secant condition. After two steps the ellipses coincide and $x_2 = x^\star$ .

In context: Lesson 10

Press Step, then try the presets. We fit $y = a\,e^{-bt}$ to 9 data points. Left: contours of $F(a, b) = \tfrac12\|r\|^2$ with the paths of Gauss–Newton (magenta) and Levenberg–Marquardt (violet, adaptive $\lambda$ ). Right: the data and the current fits, and $F(x_k) - F^\star$ on a log scale. With small residuals both methods converge in a few steps. "Start far away" ( $b_0 = 3$ ): the first Gauss–Newton step jumps to $b \approx -16$ and the fit explodes, while LM rejects bad steps, increases $\lambda$ and converges. "Large residuals" (data that no decaying exponential fits well): the dropped term $S$ is no longer small, and the log chart shows a slower, linear rate. Choose "fixed $\lambda$ " and move the slider: a huge $\lambda$ gives tiny gradient steps.

In context: Lesson 10

Lesson 11 · Saddles, sharpness and stability

Rotate the surface by dragging it. The surface is aqua where $f$ is above $f(0)$ and magenta where it is below. The orange and blue curves follow the eigenvectors $q_1$ and $q_2$ of the Hessian. Use the presets for a minimum, a maximum and a saddle. Then set $\lambda_2 = 0$ , and try the degenerate functions $x^4 + \tfrac12 y^2$ (a minimum) and $x^3 + \tfrac12 y^2$ (not a minimum): both have the same Hessian at the origin, so the Hessian alone cannot tell them apart.

In context: Lesson 11

Make the offset $\delta$ smaller. The gradient-descent path comes closer to the saddle (the cross) and stays near it longer, but it still escapes. The chart shows $|y_k|$ on a log scale: a straight line with slope $\log(1 + \eta)$ , next to the linear estimate (dashed). Each factor 10 in $\delta$ adds the same number of steps. Press δ = 0 exactly: now GD converges to the saddle. Add noise and it escapes again. Show Newton: Newton's method (magenta) jumps to the saddle. Switch to the double well $(x^2 - 1)^2 + \tfrac12 y^2$ to see the same on a function with two minima.

In context: Lesson 11

Press "One HVP" a few times. The blue vector $v_k$ turns toward the eigenvector $q_1$ (yellow dashed line); the orange vector is $Hv_k$ , the only thing the method computes. The dashed ellipse is the image of the unit circle under $H$ . The chart shows the error of $r_k$ next to $(\lambda_2/\lambda_1)^{2k}$ (dashed). Try close eigenvalues: convergence slows down. Try near a saddle: the iteration finds $\lambda_2 = -3$ , not $\lambda_{\max} = 2$ .

In context: Lesson 11

Compare three step sizes. A small network (97 parameters) is trained with full-batch GD on 20 points, from the same start. The bottom chart shows the sharpness and the line $2/\eta$ . With $\eta = 0.02$ the sharpness rises (progressive sharpening) and stays below the line. With $\eta = 0.1$ or $0.2$ it rises to $2/\eta$ and stays there, while the loss (top) has small up-and-down steps but keeps decreasing.

In context: Lesson 11

Change the radius $\rho$ . The loss (grey) has a deeper sharp minimum on the left and a flat one on the right. The dashed violet curve is the exact SAM objective: the worst loss in the yellow ball around each point. For $\rho = 0$ it is the loss. As $\rho$ grows, the narrow well is filled first, and the table shows that the SAM objective becomes lower at the flat minimum. Drag the point $w$ : the arrows below the axis compare the GD step and the SAM step, which uses the gradient at the ascent point $\hat w$ .

In context: Lesson 11

Lesson 12 · K-FAC, Shampoo and Muon

Drag the tip of $g$ . The three shaded shapes are unit balls: the circle ( $\ell_2$ ), the square ( $\ell_\infty$ ) and the diamond ( $\ell_1$ ). Each arrow is the point of its ball where $\langle g, \Delta\rangle$ is most negative. Choose one norm under "Show": the dashed line is the level line $\langle g, \Delta\rangle = -\|g\|_*$ , and it touches the ball exactly at the steepest point. Its distance from the origin gives the dual norm $\|g\|_*$ . Try $g = (2, 2)$ : the steepest $\ell_1$ step is no longer unique.

In context: Lesson 12

Move the sliders, then try the presets. The dashed circle is the set of unit inputs, with the input directions $v_1, v_2$ (grey arrows). The blue ellipse is its image under the gradient $G$ (drawn with $\sigma_1 = 1$ ): the blue arrows $Gv_i = \sigma_i u_i$ have lengths $\sigma_1$ and $\sigma_2$ . The magenta circle is the image under $UV^\top$ : the same directions $u_1, u_2$ , both of length 1. With " $\sigma = (100, 1)$ " the blue ellipse is almost a line, but the orthogonalized step still moves the weak direction $u_2$ fully. "Muon (5 steps)" shows the dashed result of Muon's iteration: close to the circle, but not exactly on it.

In context: Lesson 12

Move the slider from 1 example to 2, then to many. A tiny linear layer with 2 inputs and 2 outputs has 4 weights, so its Fisher block is $4 \times 4$ . Left: the exact $F$ , the mean of $(aa^\top) \otimes (\delta\delta^\top)$ over $n$ examples. Right: K-FAC's $A \otimes D$ from the same examples. Blue entries are positive, orange ones negative; the thick lines show the $2 \times 2$ blocks (block $(i, j)$ of $A \otimes D$ is $A_{ij}D$ ). With one example the two maps are equal. With independent $a$ and $\delta$ the error shrinks as $n$ grows (with some noise). When $\delta$ depends on $a$ , the error stays at about 0.6 for every large $n$ .

In context: Lesson 12

Move $k$ and switch the polynomial. The large chart shows the map $\sigma \mapsto p^{\circ k}(\sigma)$ after $k$ steps (magenta; earlier $k$ faint), on a log scale of the starting singular value. The small chart follows five starting values over the steps. With the cubic, values near 1 converge very fast but $\sigma = 0.001$ needs many steps. With the quintic, every value from 0.001 to 1 enters the band between the yellow lines within about 6 steps, and then keeps moving inside it.

In context: Lesson 12