20 lessons · about 26 hours · 94 figures
Linear Algebra
A matrix is a map that moves space. Start from that picture and climb to the spectral theorem, the SVD and PCA, stable numerical methods, and the linear algebra inside modern machine learning.
For students, engineers and researchers who know what a vector is and want to understand linear algebra well enough to use it in machine learning, from a first course to research level.
Quick recap
The key takeaways of all lessons on one page
All interactive figures
Every figure of the track on one page
Videos and books
The best free courses
Glossary
Every technical term of the track with a short definition
Concept map
How the lessons of this track connect to each other and to other tracks.
Keeps every lesson page and figure of this track in this browser, so you can read it without internet.
Part 1
Start here: matrices and subspaces
A plain-language tour of the core objects first. Then what a matrix does to space, the four fundamental subspaces, inverses, projections and changes of basis.
-
00
Linear algebra in plain words: a tour of the core objectsA matrix moves space. A plain tour of span, rank, dot products, eigenvectors, the determinant, the SVD, PCA and positive definite matrices, with one picture each.
50 min
7 figures
Beginner -
01
Matrices as maps, span, basis and the views of matrix multiplicationA matrix is where it sends the basis vectors. Span, independence, basis and dimension, and four ways to read a matrix product.
50 min
3 figures
Beginner -
02
Rank, rank–nullity and the four fundamental subspacesRank counts the independent directions that survive a matrix. Elimination computes it, rank–nullity counts the rest, and the four fundamental subspaces split input and output space into perpendicular pieces.
70 min
3 figures
Beginner -
03
Inverse, determinant, dot product and orthogonal projectionWhen a map can be undone, how much it scales volume, and how to find the closest point in a subspace. The invertible matrix theorem, the determinant as signed volume, and the projection matrix P = A(AᵀA)⁻¹Aᵀ.
70 min
5 figures
Beginner -
04
Orthogonal and unitary matrices, change of basis, similarityThe same vector has different coordinates in different bases, and the same map has different matrices. Similar matrices share eigenvalues, trace and determinant; orthogonal matrices keep lengths, angles and singular values.
60 min
4 figures
Intermediate
Part 2
Eigenvalues and the spectral theorem
Directions a matrix only stretches, symmetric matrices, the Rayleigh quotient and positive-definite matrices.
-
05
Eigenvalues, eigenvectors, power iteration, diagonalization and Jordan formDirections that a matrix only stretches. The characteristic polynomial, complex eigenvalues, diagonalization and powers, power iteration and its rate, defective matrices and the Jordan form.
80 min
4 figures
Intermediate -
06
Spectral theorem, Rayleigh quotient, Courant–Fischer and WeylSymmetric matrices have real eigenvalues and orthonormal eigenvectors. The Rayleigh quotient turns eigenvalues into optimization, Courant–Fischer does it for every eigenvalue, and Weyl bounds how far they move.
90 min
5 figures
Intermediate -
07
Positive (semi-)definite matrices, the five tests, Cholesky, Schur complements and the Loewner orderThe matrix version of a positive number. Five equivalent tests for positive definiteness, the Cholesky factorization and how to sample with it, the Schur complement and Gaussian conditioning, and the Loewner order for comparing covariances.
90 min
4 figures
Intermediate
Part 3
The SVD and PCA
Every matrix rotates, stretches and rotates. Best low-rank approximation and principal components.
-
08
The SVD, Eckart–Young–Mirsky, polar decomposition and the pseudoinverseEvery matrix maps an orthonormal basis to orthogonal vectors. The singular value decomposition, the best low-rank approximation, the polar decomposition and the minimum-norm least-squares solution.
95 min
5 figures
Intermediate -
09
PCA as SVD of centered data, and randomized SVDPrincipal components are the directions of largest variance, found from the SVD of the centered data. The randomized SVD finds the top singular directions of a huge matrix with a few passes over it.
70 min
4 figures
Intermediate
Part 4
Factorizations, norms and least squares
The factorization map, how to measure vectors and matrices, and how to fit data stably.
-
10
The factorization big picture: LU, QR, , , Factor once, then do cheap work with the factors. LU as the record of Gaussian elimination, why partial pivoting is needed, which factorization fits which problem, and the Woodbury formula for low-rank updates.
75 min
4 figures
Intermediate -
11
The norm zoo: , spectral, Frobenius, nuclear and Schatten normsHow to measure the size of vectors and matrices. Why the ℓ1 norm gives sparse solutions, and why the spectral, Frobenius and nuclear norms are three summaries of the singular values.
70 min
5 figures
Intermediate -
12
QR (Gram–Schmidt, Householder, Givens) and least squares done rightLeast squares is projection onto the column space. QR makes the columns orthonormal; Householder does it stably, Gram–Schmidt does not, and the normal equations square the condition number.
90 min
5 figures
Intermediate
Part 5
Numerical linear algebra
What floating point does to your algorithms, and how eigenvalues and large systems are really computed.
-
13
Floating point, cancellation, mixed precision, conditioning and backward errorHow floats are stored and why subtraction can destroy digits; why bf16 training keeps fp32 sums; the condition number κ(A) = σmax/σmin, the error bound for Ax = b, and backward error.
85 min
6 figures
Intermediate -
14
Eigenvalue and SVD algorithms (QR algorithm, Golub–Kahan) and Krylov methodsHow eigenvalues and SVDs are really computed (shifted QR, Golub–Kahan), and how Krylov methods (Arnoldi, Lanczos, CG, GMRES) solve huge problems with matrix-vector products only.
90 min
6 figures
Advanced
Part 6
Linear algebra for modern ML
Autodiff, LoRA and attention, matrix calculus, perturbation theory, random matrices and structured operators.
-
15
Linear algebra of modern ML: autodiff, LoRA/GaLore, tensor decompositions, attentionGradients are chains of transposed Jacobians, LoRA and GaLore use low rank, CP and Tucker extend the SVD to tensors, and an attention head is two low-rank matrices around a softmax.
85 min
5 figures
Advanced -
16
Matrix differentials, matrix functions, Kronecker product and vecGradients of functions of matrices with the differential method, matrix functions such as the exponential and the square root through the spectrum, and the Kronecker product with the vec rule.
85 min
5 figures
Advanced -
17
Perturbation theory: Weyl and Davis–KahanNoise moves the eigenvalues of a symmetric matrix by at most its norm (Weyl), but eigenvectors are only as stable as the eigengap (Davis–Kahan).
70 min
4 figures
Advanced -
18
Sketching, Johnson–Lindenstrauss and random matrix theoryRandom projections keep distances and solve huge least-squares problems, and random matrix theory tells you what pure noise looks like, so you can separate signal from noise.
90 min
4 figures
Advanced -
19
Structured matrices in ML for science: circulants and FNO, spectral graphs and GNNs, equivariance, matrix manifoldsMatrices that commute share eigenvectors. Circulants and the DFT give convolutions and Fourier neural operators, the graph Laplacian gives spectral clustering and GCNs, symmetry gives equivariant layers, and orthogonality constraints live on the Stiefel and Grassmann manifolds.
90 min
6 figures
Advanced