Linear Algebra · 94 figures

Interactive figures

Every figure in the track. Each one is a small experiment: drag, slide, and watch the numbers. The link under each figure takes you to the lesson that explains it.

Lesson 0 · Linear algebra in plain words

In context: Lesson 0

Drag the orange tip $u$ and the blue tip $v$ . The thick segment is the shadow of $v$ on the line through $u$ . It is aqua when $u^\top v > 0$ and magenta when $u^\top v < 0$ , and it disappears when $u \perp v$ . Check that $u_1v_1 + u_2v_2$ and $\|u\|\|v\|\cos\theta$ always agree. Try perpendicular: $(1, 2)^\top (2, -1) = 0$ .

In context: Lesson 0

Drag $x$ around the circle and watch $Ax$ . Usually $Ax$ points off the line through $x$ . For $A = \begin{pmatrix} 1 & 2 \\ 1 & 0 \end{pmatrix}$ there are two directions where it does not: $(2, 1)$ is stretched by $\lambda = 2$ , and $(1, -1)$ is flipped, $A(1, -1) = (-1, 1)$ , so $\lambda = -1$ . Press Show eigenvectors to check. Then pick rotation 90°: no direction stays on its line.

In context: Lesson 0

Lesson 1 · Matrices as maps, span and basis

In context: Lesson 1

Lesson 2 · Rank and the four subspaces

In context: Lesson 2

All four subspaces of a $2 \times 3$ matrix. On the left is the input space $\mathbb{R}^3$ : the row space and the null space meet at a right angle. On the right is the output space $\mathbb{R}^2$ : the column space and the left null space. Move $x$ with the sliders: it splits into $x_{\text{row}} + x_{\text{null}}$ , and only $x_{\text{row}}$ changes $Ax$ . Switch to rank 1 and watch every dimension in the table move by one.

In context: Lesson 2

Step through the elimination. Press Next step to apply one row operation at a time. Each operation acts on $A$ and on $I$ together, so the right block becomes the matrix $E$ with $EA = R$ . Pivot columns and free columns are marked as soon as elimination decides them. At the end the widget reads off bases of all four subspaces, in the colours of the figures above; the last row of $E$ is the left null vector $(-1, -1, 1)$ . Try needs a row swap, or type your own matrix.

In context: Lesson 2

Lesson 3 · Inverse, determinant and projection

In context: Lesson 3

Drag the orange tip $u$ and the blue tip $v$ . The thick segment is the shadow of $v$ on the line through $u$ : aqua when $u^\top v > 0$ , magenta when $u^\top v < 0$ . The readout compares $u_1v_1 + u_2v_2$ with $\|u\|\|v\|\cos\theta$ . Notice that the shadow is never longer than $v$ itself: that is Cauchy–Schwarz. The two lengths are equal only when $v$ lies on the line through $u$ (try opposite).

In context: Lesson 3

Project $b$ onto a plane. The green patch is $C(A)$ for the columns $a_1 = (1, 0, 1)$ and $a_2 = (0, 1, 1)$ of section 4(c). Drag the figure to rotate it, and move $b$ with the sliders. The projection $p$ lies in the plane, and the error $e$ always meets the plane at a right angle, so $A^\top e = 0$ . Switch to a line for section 4(b), and to plane, the wrong way to see that the sum of the two line projections misses $p$ when $a_1^\top a_2 \ne 0$ .

In context: Lesson 3

Lesson 4 · Orthogonal matrices and change of basis

In context: Lesson 4

Drag the basis vectors and watch $B = M^{-1}AM$ . The map is $A = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix}$ of section 4(b). Column $j$ of $B$ holds the new coordinates of $Ab_j$ , drawn as a dashed walk. The table shows what never changes (green) and what changes (magenta). Choose eigenvectors, or drop $b_1$ and $b_2$ on the dashed lines: $B$ becomes $\operatorname{diag}(3, 2)$ . Choose orthonormal: now the singular values stay the same too.

In context: Lesson 4

Press Next step four times. The left panel shows coordinate vectors (black $c$ , violet output); its orange and blue axes mean "steps along $b_1$ " and "steps along $b_2$ ". The right panel shows the same arrows in standard coordinates. Watch $c \to Mc \to AMc \to M^{-1}AMc$ , then see the shortcut $Bc$ land on the same point. Drag $c$ and change the basis: the shortcut always agrees, but $B$ is diagonal only for the eigenvector basis $(2, 1), (1, 1)$ .

In context: Lesson 4

Drag $x$ and $y$ , or change $\theta$ . The default is the rotation $Q = \tfrac15\begin{pmatrix} 3 & -4 \\ 4 & 3 \end{pmatrix}$ of section 4(c), with $x = (1, 2)$ and $y = (3, 1)$ . Notice that lengths, the dot product and the angle stay the same (green), and the unit circle maps onto itself. Switch to the reflection $F_\theta$ : still $Q^\top Q = I$ , but $\det = -1$ and the F is mirrored. Then try not orthogonal to see everything change (magenta).

In context: Lesson 4

Lesson 5 · Eigenvalues and Jordan form

In context: Lesson 5

Slide the entry $a_{21}$ and watch the roots. The violet curve is $p(\lambda) = \lambda^2 - \tau\lambda + \delta$ ; its crossings with the $\lambda$ -axis are the eigenvalues (orange and blue dots). The lower panel shows the same two roots in the complex plane. Move $a_{21}$ from 2 down to $-2$ : the crossings move together, meet at the lowest point ( $\tau^2 = 4\delta$ ), and then leave the real line as a complex pair $a \pm bi$ . Try rotation 90° and rotate and scale.

In context: Lesson 5

Press Step and watch the dots line up. $A$ has the eigenvectors $v_1 = (1, 1)$ and $v_2 = (2, -1)$ of the example in section 2, with eigenvalues you set. The chart shows the sine of the angle between $x_k$ and the dominant eigenvector on a log scale: close to a straight line, because each step multiplies it by about $|\lambda_2/\lambda_1|$ (the dashed reference line). Try 1 and 0.9: about 0.05 digits per step. Try the tie $1, -1$ : no convergence. Make the dominant $\lambda$ negative: $x_k$ flips sign every step, but its line converges.

In context: Lesson 5

Slide $\varepsilon$ toward 0. The two eigenvector lines of $A(\varepsilon)$ close up like scissors, and at $\varepsilon = 0$ only one is left: the Jordan block. The right panel shows the eigenvalues $2 \pm \sqrt{\varepsilon}$ : they meet at 2 and, for $\varepsilon < 0$ , turn into a complex pair. Press $\varepsilon = 10^{-6}$ : the matrix barely changed, yet the eigenvalues moved by $10^{-3}$ and $\kappa(X) = 1000$ .

In context: Lesson 5

Lesson 6 · Spectral theorem and Rayleigh quotient

Change $a$ , $b$ , $c$ in $S = \begin{pmatrix} a & b \\ b & c \end{pmatrix}$ . The violet ellipse is the image of the unit circle. Its axes are always the eigenvectors $q_1$ and $q_2$ , at a right angle, with half-lengths $|\lambda_1|$ and $|\lambda_2|$ . Drag $x$ : the two faint arrows build $Sx$ from its pieces along $q_1$ and $q_2$ . Compare with a non-symmetric matrix in lesson 05: there the directions that only stretch need not be perpendicular.

In context: Lesson 6

Move the slider and watch the violet arrow. $S$ is a symmetric $3\times3$ matrix and $q_1$ is its top eigenvector. The aqua patch is the plane $q_1^\perp$ . As $x$ walks around the unit circle of this plane, $Sx$ never leaves the plane: the image of the circle is an ellipse inside the plane. Switch to not symmetric: $q_1$ is still an eigenvector of $M$ , but $Mx$ now leaves the plane (the dashed magenta segment is its height $q_1 \cdot Mx$ ).

In context: Lesson 6

Drag $x$ around the circle and watch the chart. For a unit $x$ , $R(x) = x \cdot Sx$ is the part of $Sx$ along $x$ (thick violet). The sideways part $Sx - R(x)x$ (dashed magenta) is $\tfrac12\nabla R$ : it points to where $R$ grows. It vanishes only at the eigenvectors, where the chart has its max $\lambda_1$ and its min $\lambda_2$ . The bar shows the weights $(q_1 \cdot x)^2$ and $(q_2 \cdot x)^2$ of the weighted average. This is the "sideways part" of lesson 05, now with a meaning: a gradient.

In context: Lesson 6

Turn and tilt the plane $U$ . Here $S = \operatorname{diag}(3, 2, 1)$ , so the eigenvectors are the axes. The blue plane is $W = \operatorname{span}(q_2, q_3)$ , where $R \le 2$ . However you move the orange plane $U$ , it shares a magenta line with $W$ , and the readout shows $\min_U R \le R(x) \le \lambda_2 = 2$ . Press best plane to reach $\min_U R = 2$ . Switch to a line ( $\dim U = 1$ ): $1 + 2 = 3$ is not more than 3, and the line usually meets $W$ only at 0.

In context: Lesson 6

Move $\varepsilon$ and watch the paths stay in their cones. $A$ is a fixed $4 \times 4$ symmetric matrix with eigenvalues $3, 2.2, 1.8, 0.4$ , and $E$ is symmetric with $\|E\|_2 = 1$ . Each coloured path is $\lambda_k(A + \varepsilon E)$ ; each shaded cone is Weyl's interval $\lambda_k(A) \pm \varepsilon$ . Press E = I to touch the cone edges, and New random E to see a typical perturbation use less of its budget. Then switch to not symmetric: the eigenvalues $\pm\sqrt{\varepsilon}$ leave the cone at once.

In context: Lesson 6

Lesson 7 · Positive definite matrices and Cholesky

Change $a$ , $b$ , $c$ in $S = \begin{pmatrix} a & b \\ b & c \end{pmatrix}$ and drag to rotate. The surface is the energy $x^\top S x$ : aqua where it is positive, magenta where it is negative. The arrows on the floor are the eigenvectors $q_1$ and $q_2$ . Go from bowl to trough to saddle and watch $\lambda_2$ cross 0. Then try $\begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix}$ : positive entries, but a saddle.

In context: Lesson 7

Switch between white noise z, μ + Lz, μ + √S z and both, and follow the magenta point. The round cloud of white noise becomes the aqua ellipse of $S$ . With both, the blue cloud ( $Lz$ ) and the orange cloud ( $S^{1/2}z$ ) fill the same ellipse and have almost the same sample covariance, but the same $z$ lands in two different places. The violet arrows are the columns of the map; for $L$ the second one is vertical, because $L$ is lower triangular. Change $\rho$ and drag $\mu$ .

In context: Lesson 7

Compare two ellipses $\mathcal E_A$ (blue) and $\mathcal E_B$ (orange). Turn the grey handle $u$ : the dashed lines touch each ellipse at its width $\sqrt{u^\top M u}$ in that direction. For $\operatorname{diag}(2, 1)$ and $\operatorname{diag}(1, 2)$ , each is wider somewhere, so they are incomparable. Switch to S vs S + 0.8I (ridge) or posterior vs prior (a Kalman update): one ellipse sits inside the other, $A \preceq B$ . Drag the half-axis handles to make your own pair.

In context: Lesson 7

Lesson 8 · The SVD

Press Play, or step through the three factors. $V^\top$ turns the orange $v_1$ and the blue $v_2$ onto the axes, $\Sigma$ stretches the axes by $\sigma_1$ and $\sigma_2$ , and $U$ turns the ellipse into place: it ends on the dashed violet ellipse $\{Ax : \|x\| = 1\}$ . Try diag(3, −2) and det < 0: the letter F comes out mirrored. The figure puts the mirror into $U$ (it could also go into $V$ ), never into $\Sigma$ . With rank 1 the circle is flattened to a segment.

In context: Lesson 8

Turn the pair with the slider, or drag the tip of $v_1$ . The magenta number is the angle between $Av_1$ and $Av_2$ . Find the turn where it is $90°$ , then press Find the orthogonal images to check. Notice in the lower chart: at that same turn, $\|Av_1\|$ reaches its maximum $\sigma_1$ (or its minimum $\sigma_2$ ). For the shear the answer is $t \approx 58.3°$ , not the eigenvector $(1, 0)$ of $A$ . Try 2 × rotation: every pair works, because $A^\top A = 4I$ .

In context: Lesson 8

Move the rank slider. The middle image is $A_k$ , the sum of the first $k$ layers $\sigma_i u_i v_i^\top$ . The chart shows the singular values you keep, the grey ones you drop, and the relative error, which is exactly $\sqrt{\sum_{i>k}\sigma_i^2/\sum_i\sigma_i^2}$ . Switch to axis-aligned blocks: a background and three rectangles need only rank 4, and the error drops to rounding level. The rotated square has the same kind of content but needs far more terms, because its edges are not aligned with the rows and columns. Set the third panel to k-th layer to see single rank-1 terms.

In context: Lesson 8

Two planes: the input space of $x$ and the output space of $b$ . With $\sigma_2 = 0$ the matrix has rank 1. Drag $b$ : $Ax^+$ is its projection onto the line $C(A)$ , and the dashed magenta residual is perpendicular to it. In the input space, every point of the thick violet line is a least-squares solution; $x^+$ is the one in the row space, closest to 0. Drag $x$ along the line and compare $\|x\|$ . Then raise $\sigma_2$ a little (0.05): $A$ becomes invertible and $x^+$ jumps far away. Switch on the cut-off: $\sigma_2 \le 0.1\,\sigma_1$ is treated as 0, as rcond does, and $x^+$ stays small at the price of a nonzero residual.

In context: Lesson 8

Lesson 9 · PCA and randomized SVD

Drag the orange handle to turn $w$ . Each grey point casts an orange shadow on the line; the blue segments are the distances to the line. The chart shows, for every angle, the shadow variance $w^\top S w$ and the squared distances, summed and divided by $N - 1$ like the variance. The two curves always add up to $\operatorname{tr} S$ . Find the angle where the orange curve is highest, then press Show principal axes: it is PC1, and its height is $\lambda_1$ .

In context: Lesson 9

Change one setting at a time. Here $A = \operatorname{diag}(3, 1, 0.1)$ , so the singular vectors are the axes. With $p = 0$ and $q = 0$ , the single sample $y_1 = A\omega_1$ misses the top direction $u_1$ by about $49°$ for this unlucky $\omega_1$ (grey). Set $p = 1$ : two samples span a blue plane that almost contains $u_1$ , and the estimate $\hat u_1$ is about $6°$ from it. Set $p = 0$ and $q = 1$ : the weights $3, 1, 0.1$ become $27, 1, 0.001$ , and the single sample turns toward $u_1$ . Press Run 1000 draws: the median angle is about $20°$ with $p = q = 0$ , and about $2°$ with $p = 1$ or $q = 1$ .

In context: Lesson 9

Drag the cloud by its mean $\bar x$ , and change the spreads and the correlation. The orange and blue arrows are the principal axes $\pm\sqrt{\lambda_i}\,q_i$ ; the violet ellipses are one and two standard deviations. The dashed magenta line is the top eigenvector of the uncentered matrix $X^\top X/(N-1)$ : move the mean away from 0 and it turns toward the mean. Press Mean to 0 and it agrees with PC1 again. The readout checks Theorem 2: $\sigma_i^2/(N-1) = \lambda_i$ .

In context: Lesson 9

Each dot is one randomized SVD; the lines are means. The chart shows the spectral error of the rank- $k$ result divided by the best possible error $\sigma_{k+1}$ (Eckart–Young), against the oversampling $p$ , for $q = 0, 1, 2$ power iterations. With the slow spectrum, oversampling alone stays far from 1, while one power iteration almost reaches it. With the fast spectrum, about ten extra columns are enough. The noise floor case is the worst for $q = 0$ . Here $A$ is $\operatorname{diag}(\sigma_1, \dots, \sigma_{200})$ : the algorithm only multiplies by $A$ and $A^\top$ , and a rotated Gaussian matrix is still Gaussian, so the errors have the same distribution for $U\operatorname{diag}(\sigma)V^\top$ with any orthogonal $U$ and $V$ .

In context: Lesson 9

Lesson 10 · Factorizations and LU

Eigenvectors are not singular vectors. The dashed circle is the unit circle and the aqua ellipse is its image under $A$ ; drag the grey dots (the columns of $A$ ) to change it. The blue arrows are the SVD: orthonormal $v_1, v_2$ (faint) go to the ellipse axes $\sigma_1 u_1, \sigma_2 u_2$ . The violet lines are the eigenvector directions, where $A$ only stretches. Try symmetric: the two pictures agree. Try triangular, nearly defective and shear: the eigenvectors close up and then collapse to one line, while the singular vectors stay orthogonal. With rotation there is no real eigenvector at all. In every case $\sigma_1\sigma_2 = |\det A|$ .

In context: Lesson 10

Step through elimination. Each Next step subtracts a multiple $\ell$ of the pivot row from a lower row. The multiplier goes into $L$ at the same position, and a zero appears in $U$ . The readout $\max|PA - LU| = 0$ shows that the equation holds after every step, not only at the end. Try zero pivot: without row exchanges elimination stops although $A$ is invertible; with partial pivoting it finishes, and every $|\ell_{ij}| \le 1$ . The 4×4 case shows how a swap also moves the multipliers already stored in $L$ .

In context: Lesson 10

Shrink the pivot. The system $\begin{pmatrix}\varepsilon & 1\\ 1 & 1\end{pmatrix}x = \begin{pmatrix}1\\2\end{pmatrix}$ has solution $x \approx (1, 1)$ for every small $\varepsilon$ , and $\kappa(A) \approx 4$ : the problem is easy. Move the slider. Without a row exchange the multiplier is $1/\varepsilon$ , $U$ has entries of size $1/\varepsilon$ , and the error follows the dashed line $u/\varepsilon$ until no correct digit is left. With partial pivoting the multiplier is $\varepsilon$ and the answer is correct to the last bit. The failure is in the algorithm, not in the problem.

In context: Lesson 10

Lesson 11 · Norms of vectors and matrices

Change $p$ and watch the ball. The green region is the unit ball of $\|\cdot\|_p$ ; the thin outlines are the $\ell_1$ , $\ell_2$ and $\ell_\infty$ balls. As $p$ grows, the ball grows from the diamond to the square. Move $p$ below 1: the ball becomes a star, and the midpoint of $e_1$ and $e_2$ falls outside it. A set that does not contain the midpoints of its points is not convex, so for $p < 1$ the formula is not a norm. Drag $x$ to see $\|x\|_\infty \le \|x\|_2 \le \|x\|_1$ .

In context: Lesson 11

Move $z$ and $\lambda$ . Top: the lasso objective $g$ with its minimizer, the ridge objective with its minimizer, and the two slopes of $g$ at the kink. Bottom: the output $x^\star$ against the input $z$ . Move $z$ into the band $-\lambda \le z \le \lambda$ : the left slope becomes $\le 0$ and the right slope $\ge 0$ , so the lowest point of $g$ sits exactly at the kink and the lasso output is exactly 0. The ridge output $z/(1 + \lambda)$ only gets smaller.

In context: Lesson 11

Where the ellipse meets the ball. The grey ellipses are level sets of the least-squares loss around $\hat w$ . The solution under a constraint is the point where the smallest ellipse reaches the diamond (lasso) or the disk (ridge). Drag $\hat w$ around: for a large range of positions the lasso answer sits on a corner, with one weight exactly 0, while the ridge answer only shrinks. Make $t$ large enough and both answers equal $\hat w$ . The tilt slider turns the ellipses (correlated features).

In context: Lesson 11

The largest stretch. The grey region is the unit ball of $\|\cdot\|_p$ and the green region is its image under $A$ . The dashed ball has radius $\|A\|_p$ : it is the smallest $p$ -ball around the image, and it touches the image at $Ax^\star$ . For $p = 1$ the maximizer $x^\star$ is a corner $e_j$ (largest column sum); for $p = \infty$ it is a corner $(\pm1, \pm1)$ (the signs of the largest row); for $p = 2$ it is $v_1$ . The matrix is half of the one in section 4, so every norm is half as large.

In context: Lesson 11

Lesson 12 · QR and least squares

Least squares is a projection. The green plane is $C(A)$ , spanned by the columns $a_1$ and $a_2$ . The vector $b$ is not in the plane, so $Ax = b$ has no solution. The closest point is $p = A\hat x$ , and the residual $r = b - p$ meets the plane at a right angle: $A^\top r = 0$ . Rotate the figure and move $b$ with the sliders. With line through 3 points, the same picture fits a line $c_0 + c_1 t$ to three data points: the three values $b_i$ are one vector in $\mathbb{R}^3$ .

In context: Lesson 12

Step through Gram–Schmidt. Press Next step. The grey arrows are the columns of $A$ : the two vectors above and $a_3 = (0, 1, 1)$ . At each step the magenta vector $v$ is what is left of a column after its parts along the earlier $q$ 's are removed (dashed); then it is normalized to $q_1$ , $q_2$ or $q_3$ . Watch $R$ fill in column by column. Then choose nearly dependent, set $\delta = 10^{-8}$ , go to the last step and switch between CGS and MGS: CGS gives a $q_3$ that is $45°$ away from orthogonal to $q_2$ .

In context: Lesson 12

Reflect $x$ onto the axis. The mirror is the line orthogonal to $v$ . Reflecting $x$ across it gives $Hx$ , a point on the $e_1$ axis at distance $\|x\|$ from 0. Drag $x$ close to the positive $e_1$ axis and switch to other sign: $v$ becomes very short, which is where cancellation hits in floating point (the readout repeats the $x = (1, 10^{-9}, 0)$ example in real double precision). Givens rotation reaches the axis by turning instead of reflecting.

In context: Lesson 12

Four QR algorithms, one test. Each dot is a random $60 \times 15$ matrix with condition number $\kappa$ , factored in double precision by classical Gram–Schmidt, modified Gram–Schmidt, Cholesky QR ( $R$ from the Cholesky factor of $A^\top A$ , then $Q = AR^{-1}$ ) and Householder. The dashed lines are $\varepsilon\kappa$ and $\varepsilon\kappa^2$ . CGS and Cholesky QR follow $\varepsilon\kappa^2$ and reach order 1 near $\kappa = 10^8$ (Cholesky QR then breaks down: $A^\top A$ is no longer numerically positive definite). MGS follows $\varepsilon\kappa$ . Householder stays at $10^{-15}$ . Move the $\kappa$ slider: the table shows that all four have a tiny residual $\|A - QR\|$ .

In context: Lesson 12

Fit a line, then try to beat it. The model is $y \approx c_0 + c_1 t$ , so $A$ has the columns $(1, \dots, 1)$ and $(t_1, \dots, t_m)$ . Drag the data points. The least-squares line keeps $A^\top r = 0$ : the residuals add up to 0, and so do the $t_i r_i$ . Switch to your own line and drag its two grey points: $A^\top r$ is no longer 0, and by the Pythagoras step of the proof its $\|r\|^2$ is always at least the least-squares value, with equality only on the least-squares line.

In context: Lesson 12

Lesson 13 · Floating point and conditioning

In context: Lesson 13

Drag $\delta b$ in the small disk. The two equations of $Ax = b$ are the orange and blue lines, and $x$ is where they meet. Changing $b$ shifts both lines in parallel (dashed), and the intersection moves to $x + \delta x$ . The violet ellipse holds every possible $x + \delta x$ for $\|\delta b\| \le r$ . Make $\delta$ small: the lines become nearly parallel, the ellipse stretches like $1/\sigma_2$ , and the amplification approaches $\kappa_2(A)$ when you press Worst δb.

In context: Lesson 13

Lesson 14 · Eigenvalue algorithms and Krylov methods

Move the shift $\sigma$ . All three methods start from the same vector of a $5 \times 5$ symmetric matrix. Power iteration falls at the slow rate $|\lambda_2/\lambda_1| = 0.8$ (0.96 for the second spectrum). Inverse iteration finds the eigenvalue closest to $\sigma$ (the blue dot on the strip) at the rate $|\lambda_J - \sigma|/|\lambda_K - \sigma|$ : put $\sigma$ near an eigenvalue and the line gets steep; put it halfway between two and it crawls. Rayleigh quotient iteration updates its shift every step and needs only a few.

In context: Lesson 14

Press Run, then change the shift. The chart follows the subdiagonal entries of $A_k$ on a log scale, and the heat map shows $A_k$ (stronger colour = larger $|a_{ij}|$ ). Without a shift each entry decays at its own rate $|\lambda_{j+1}/\lambda_j|$ (dashed) and the whole run takes about 100 steps. With the Rayleigh or Wilkinson shift the corner entry drops below $10^{-14}$ in a few steps, the block deflates (dashed frame), and the next corner starts: all four eigenvalues take 13 steps with the Rayleigh shift and 7 with the Wilkinson shift. Try the zero-diagonal matrix: the Rayleigh shift is 0, so the corner does not move until rounding errors break the symmetry (the first eigenvalue deflates after about 37 steps); the Wilkinson shift finishes in 8.

In context: Lesson 14

Move $k$ , then switch to the Hessenberg view. Top: the condition number of the plain basis $[b, Ab, \dots, A^{k-1}b]$ (each vector scaled to length 1) grows by about 6 times per step and comes close to $1/u$ (numerically singular) by $k = 20$ ; the Arnoldi basis $Q_k$ spans the same space and keeps $\kappa(Q_k) = 1$ . Bottom: the angle between $A^{k-1}b$ and the top eigenvector falls toward 0. In the Hessenberg view, press Next step: each Arnoldi step adds one column to $\tilde H_k$ . Switch to a symmetric matrix and the entries above the first superdiagonal become 0.

In context: Lesson 14

Move $k$ or press Play. Each column holds the $k$ Ritz values of a $200 \times 200$ symmetric matrix after $k$ Lanczos steps; the ticks on the right are its eigenvalues. The isolated eigenvalues 2.0 and 1.5 are found first, then the edges of the dense block $[0, 1]$ ; the interior never converges in 60 steps. Switch off reorthogonalization and go past $k \approx 30$ : a stray Ritz value climbs up and becomes a second copy of 2.0 (a ghost).

In context: Lesson 14

Change the spectrum and move $k$ . The blue curve is the CG polynomial $p_k$ : it starts at $p_k(0) = 1$ and its roots (blue dots) are the Ritz values. CG makes $p_k$ small at the eigenvalues (the ticks). The orange scaled Chebyshev polynomial is the best one can do knowing only $[\lambda_{\min}, \lambda_{\max}]$ ; the dashed lines at $\pm 1/T_k(\cdot)$ give the bound. With 3 distinct values CG is exact after 3 steps; with outliers three roots go to the outliers first, and then CG converges as if $\kappa$ were 2.

In context: Lesson 14

Drag $x_0$ and raise $\kappa$ . On the contours of $\varphi(x) = \tfrac12 x^\top Ax - b^\top x$ , steepest descent zigzags: each step stops where its line touches a contour, so the next gradient is perpendicular to it. CG takes the same first step, then a direction that is $A$ -conjugate to it, and reaches $x^*$ after 2 steps ( $n = 2$ ). The default start is the worst case for $\kappa = 10$ : steepest descent meets its bound $((\kappa - 1)/(\kappa + 1))^k$ .

In context: Lesson 14

Lesson 15 · Autodiff, LoRA, tensors, attention

In context: Lesson 15

One head, six tokens, $d_h = 2$ . The readout shows $\operatorname{rank}(\beta S) = 2$ but $\operatorname{rank} A = 6$ : the low rank is in the logits, not in $A$ . Move the scale $\beta$ (an inverse temperature) to 0: every row of $A$ becomes uniform, $A$ has rank 1, and the output is the mean of the values. Move it up: the rows become nearly one-hot and the output jumps to one value vector. Turn on the causal mask and pick $t_1$ : the first token can only attend to itself.

In context: Lesson 15

Lesson 16 · Matrix calculus and matrix functions

Choose a function and press New direction. The blue curve uses the correct derivative and falls with slope 2. The orange curve uses a common mistake ( $X^{-1}$ instead of $X^{-\top}$ , $2Ax$ instead of $(A + A^\top)x$ , $-X^{-2}dX$ instead of $-X^{-1}dX\,X^{-1}$ ) and falls only with slope 1. For $\operatorname{tr}(AX)$ the function is linear, so the correct prediction is exact and only rounding error is left.

In context: Lesson 16

The gradient is perpendicular to the level sets. Drag the violet point $x$ . The blue arrow is the true gradient $\tfrac12(A + A^\top)x - b$ ; it crosses the green contour lines at a right angle. The orange dashed arrow is $Ax - b$ : for a non-symmetric $A$ it points in another direction. Choose I + skew part: the contours are circles, yet $Ax - b$ is turned by a fixed angle. Switch the arrow field to $Ax - b$ to see the error everywhere.

In context: Lesson 16

Move the logits and the temperature. The bars are the probabilities $s$ ; the map is $J = \operatorname{diag}(s) - ss^\top$ (orange positive, blue negative). The row sums are always 0. The preset $z = (0, \ln 2, \ln 3)$ is worked example (b). Lower the temperature $T$ : $s$ becomes one-hot and every entry of $J$ goes to 0, which is why a saturated softmax passes almost no gradient.

In context: Lesson 16

Let the plane flow. Press Play or move $t$ . Every point moves along $x(t) = e^{tA}x(0)$ ; the green ring is the unit circle carried by $e^{tA}$ , and it stays an ellipse because $e^{tA}$ is a linear map. Try the presets: real eigenvalues give straight eigen-directions (dashed), complex ones give spirals (closed ellipses for the center, where $\operatorname{Re}\lambda = 0$ ), a saddle stretches along one eigen-direction and squeezes along the other. The readout shows that $\det e^{tA}$ and $e^{t\operatorname{tr}A}$ are the same number.

In context: Lesson 16

Lesson 17 · Perturbation theory

Shrink the gap. Here $A = \operatorname{diag}(1 + \delta, 1)$ and $E = \varepsilon\begin{pmatrix}0&1\\1&0\end{pmatrix}$ . On the left, the blue dashed ellipse is the unit circle mapped by $A$ , the orange one is mapped by $A + E$ ; the arrows are their long axes $\lambda_1 q_1$ and $\hat\lambda_1 \hat q_1$ . Move $\delta$ below $\varepsilon$ : the ellipse becomes a circle and $\hat q_1$ turns by almost $45°$ , while the eigenvalues move by less than $\varepsilon$ . On the right, the true $\sin\theta$ (orange) is about $\varepsilon/\delta$ for large gaps, where it almost touches the classical bound $\|Eq_1\|/\hat\delta$ (aqua), and it levels off at $\sin 45° \approx 0.71$ for small gaps. Both Davis–Kahan bounds of section 3 stay above it; the bound $2\varepsilon/\delta$ (violet) is twice as large for large gaps.

In context: Lesson 17

Each eigenvalue stays in its own interval. Row $k$ shows $\lambda_k$ of an $8\times8$ symmetric $A$ (blue), the interval $[\lambda_k - \|E\|_2,\ \lambda_k + \|E\|_2]$ (aqua) and $\hat\lambda_k$ of $A + E$ (orange). Increase $\|E\|_2$ and press New noise: the orange dot never leaves its interval, even when neighbouring intervals overlap. Choose $\varepsilon I$ to see that the bound is reached, and a rank-one $E$ to see every eigenvalue move up.

In context: Lesson 17

A non-normal matrix amplifies noise. The orange dots are the eigenvalues of $J + E$ in the complex plane. They sit on the violet circle of radius $\varepsilon^{1/n}$ , far outside the aqua disc $|\mu| \le \varepsilon$ that Weyl and Bauer–Fike give for a symmetric or normal matrix (for small $\varepsilon$ this disc is so small that it hides under the blue dot at 0). Make $n$ larger or $\varepsilon$ smaller and read the amplification. Switch to a random $E$ with the same norm: the eigenvalues still spread around a circle of about the same size (usually a little smaller, because only part of the norm of $E$ sits in the corner entry).

In context: Lesson 17

Turn and tilt $\hat V$ . Blue is $V$ (the $x$ -axis, or the $xy$ -plane), orange is $\hat V$ . The readout computes the singular values of $\hat V^\top V$ and the principal angles. With two planes, the magenta line is the shared line, and $\theta_1 = 0$ always; the arc shows $\theta_2$ . With line and plane, the dashed segment drops the line onto its shadow. For equal dimensions the readout also checks $\|VV^\top - \hat V\hat V^\top\|_2 = \sin\theta_{\max}$ .

In context: Lesson 17

Lesson 18 · Sketching, JL and random matrices

Project 50 points from $\mathbb{R}^{1000}$ to $\mathbb{R}^k$ . The histogram shows, for all 1225 pairs, the squared distance after the projection divided by the squared distance before. The violet curve is its exact law $\chi^2_k/k$ : it does not depend on the points or on $D$ . Move $k$ : the spread shrinks like $\sqrt{2/k}$ . Below $k \approx 94$ , Theorem 3 (without the $\delta$ term) gives no $\varepsilon < 1$ ; above it the aqua lines show $1 \pm \varepsilon$ , and the whole histogram almost always stays inside them (press New points a few times: the union bound is pessimistic).

In context: Lesson 18

Pure noise is not flat. Each data set has $n$ samples of $p = 100$ independent $N(0,1)$ features, so every true eigenvalue is 1. The histogram pools the eigenvalues of $X^\top X/n$ from four data sets; the violet curve is the Marchenko–Pastur density for $\gamma = p/n$ . Move $\gamma$ : the bulk widens to $[(1-\sqrt\gamma)^2, (1+\sqrt\gamma)^2]$ , and for $\gamma > 1$ part of the spectrum is exactly 0. Then add a spike $\theta$ (Theorem 5 below): its orange top eigenvalues leave the bulk only when $\theta > \sqrt\gamma$ .

In context: Lesson 18

Move $k$ and watch the ellipse. The dashed circle is the set of unit vectors $y$ of the plane. The orange ellipse shows the same vectors at the length $\|Sy\|$ ; its half-axes are the singular values $\sigma_1, \sigma_2$ of $SU$ . The aqua ring is the band $\sqrt{1-\varepsilon} \le \|Sy\| \le \sqrt{1+\varepsilon}$ . Drag the blue vector: its image (violet) always lies on the ellipse. For $\varepsilon = 0.5$ , the ellipse leaves the ring in about one draw out of three at $k = 20$ , and almost never at $k = 100$ . Press New S a few times, or Run 1000 draws to count.

In context: Lesson 18

Raise the dimension. Left: the norms of 300 random Gaussian vectors $g \sim N(0, I_d)$ , divided by $\sqrt d$ . Right: the angles between the 4950 pairs among 100 of them. The violet curves are the exact densities. As $d$ grows, $\|g\|$ stays at $\sqrt d$ with a standard deviation of about $0.71$ , whatever $d$ is, so $\|g\|/\sqrt d$ piles up at 1. The angles pile up at $90°$ with a spread of about $57.3°/\sqrt d$ . In $d = 2$ every angle is equally likely; in $d = 2000$ no pair is more than a few degrees from orthogonal.

In context: Lesson 18

Lesson 19 · Structured matrices in ML

Pick a kernel, then move the mode $k$ . The heatmap is the circulant $C$ built from the first column $c$ ; hover over a cell to see that $C$ is constant along each wrapped diagonal. The blue dots are a Fourier mode $f_k$ (real part), the violet squares are $Cf_k$ , computed as a matrix–vector product. They always form the same wave, only scaled by $|\hat c_k|$ and shifted in phase: $Cf_k = \hat c_k f_k$ , with $\hat c = \mathrm{fft}(c)$ . Try "shift P": all eigenvalues lie on the unit circle. Try "difference": $c$ is not symmetric, so most eigenvalues are complex.

In context: Lesson 19

Colour the graph by an eigenvector of $L = D - A$ . Nodes are orange where $u_k > 0$ and blue where $u_k < 0$ . For $k = 2$ (the Fiedler vector) the signs cut the two clusters apart along the single bridge. Click the bridge to remove it: a second eigenvalue drops to 0, one per connected component. Raise $k$ and the vector changes sign more often. "Signal kept on $u_1 \dots u_k$ " projects a noisy signal onto the $k$ smoothest eigenvectors: a low-pass filter on the graph. Try the karate club: the Fiedler vector almost predicts how the club split in 1977.

In context: Lesson 19

Follow both paths around the square. Right, then down: $P f(x)$ (dashed outlines). Down, then right: $f(Px)$ (filled bars). The layer is equivariant when they match, and the relative error $e$ is then 0 or at round-off level. Compare circular and zero-padded convolution, a convolution that also sees the position, and a dense layer. Then switch to "relabel graph nodes": a GCN layer passes the test only if the adjacency matrix is relabeled together with the features.

In context: Lesson 19

One Fourier layer with one channel. Move $k_{\max}$ : the layer keeps the blue modes, drops the grey ones and multiplies mode $k$ by $R_k$ . Then change the grid size $n$ : the violet dots (the output on your grid) stay on the violet curve (the same layer on a 1024-point grid), because $R$ belongs to frequencies, not to grid points. Choose "smooth + high modes" with $n = 32$ to see the limit: modes above $n/2$ alias onto the kept modes and the output changes. With "step", the truncation gives the Gibbs ripples.

In context: Lesson 19

Choose a propagation matrix, then add layers. The blue curve is the response $g(\lambda)^L$ after $L$ layers; the orange dots are the eigenvalues of the chosen graph (the violet dot has the largest $|g|$ ). With "2I − L_sym" (step 4) the eigenvalue 2 at $\lambda = 0$ makes the response grow like $2^L$ . With "I − L_sym" a bipartite graph (path, ring, star) has $g = -1$ , which never decays. With the renormalized $\hat A$ (step 5) every other eigenvalue shrinks to 0 as $L$ grows: only the top eigenvector survives. This is over-smoothing.

In context: Lesson 19

Take steps on the sphere. At the black point $X$ the grey arrow is the gradient $G = -SX$ of $f(x) = -\tfrac12 x^\top Sx$ , $S = \operatorname{diag}(1.6, 1, 0.4)$ . The blue arrow $P_X(G)$ lies in the aqua tangent plane. The orange step $V = -\eta P_X(G)$ ends at $X + V$ , slightly outside the sphere; the violet point is the retraction $(X+V)/\|X+V\|$ . Press Take one step a few times: $X$ moves to the minimizer $\pm e_1$ , the top eigenvector of $S$ . Drag to rotate the view.

In context: Lesson 19