Videos and books

Every resource on this page is free to watch or read, from the author's or the publisher's own page. Where a classic book is not free, we say so and point to its official page. Each lesson of the track also ends with a Watch and read section that names the exact lecture or chapter.

How to use it. Watch for the picture, read for the proofs, and write code to check both. A good default: one 3Blue1Brown or Strang video before a lesson, the lesson and its notebook, then one book section from the table at the end of this page.

Video courses #

For the picture #

Vectors | Chapter 1, Essence of linear algebra3Blue1Brown · Essence of linear algebra, chapter 1 · 10 min

• 3Blue1Brown, Essence of linear algebra (course page, YouTube playlist). Short animated chapters on linear maps, determinants, null space, change of basis, eigenvectors. No proofs, but the geometric picture you keep for life. Watch all of it once before or during lessons 00–05.
• Gilbert Strang, A 2020 Vision of Linear Algebra (MIT OCW). Six short videos that retell the whole subject as a chain of factorizations: $A = CR$, $LU$, $QR$, $Q\Lambda Q^\top$, $U\Sigma V^\top$. A preview of the whole track with lesson 00, and a good review after lesson 10.

University courses #

Course Introduction of 18.065 by Professor StrangGilbert Strang · MIT 18.065, Spring 2018 · 7 min

• MIT 18.06, Linear Algebra (Strang, 2010) (course, video lectures). The classic first course: elimination, the four subspaces, projections, determinants, eigenvalues, positive definite matrices, the SVD. Pairs with lessons 01–07. The OCW Scholar version, 18.06SC, adds short recitation videos, lecture summaries and problem sets with solutions.
• MIT 18.065, Matrix Methods in Data Analysis, Signal Processing, and Machine Learning (Strang, 2018) (course, video lectures). The closest course to this track: the SVD and Eckart–Young, positive definite matrices, norms, least squares, randomized linear algebra, circulants and convolution, graphs and clusters, deep learning. The notes and problem sets of the newer 2023 offering are on GitHub (mitmath/18065).
• MIT 18.335, Introduction to Numerical Methods (Steven G. Johnson, 2019) (course). Graduate numerical linear algebra: floating point, backward stability, QR, eigenvalue algorithms, Arnoldi, Lanczos, GMRES and conjugate gradients. Pairs with lessons 12–14.
• MIT 18.S096, Matrix Calculus for Machine Learning and Beyond (Edelman and Johnson, 2023) (course). Derivatives as linear maps, Kronecker products, derivatives of the determinant, the inverse and eigenproblems, forward and reverse mode. Pairs with lessons 15–17.

For machine learning and data #

Singular Value Decomposition (SVD): OverviewSteve Brunton · Data-Driven Science and Engineering · 7 min

• Steve Brunton, Singular Value Decomposition (YouTube playlist). About 7 hours of short videos that follow chapter 1 of his book (below): the SVD, optimal truncation, the pseudoinverse and least squares, PCA, eigenfaces, randomized SVD, with Python and MATLAB code. Pairs with lessons 08–09.
• Rachel Thomas, Computational Linear Algebra for Coders (fast.ai; YouTube playlist, notebooks). A code-first course on matrix computations for speed and accuracy: randomized SVD, QR, NMF, floating point, PageRank. A practical companion to lessons 09 and 12–14.
• Geometric Deep Learning (Bronstein, Bruna, Cohen, Veličković) (lectures). The AMMI course on symmetry, groups, equivariance, graphs and manifolds in deep learning. Pairs with lesson 19.

Free books #

Main textbooks #

• Boyd and Vandenberghe, Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares (VMLS) (free PDF and slides). Applied and gentle: vectors, matrices, least squares, data fitting, classification, with Python and Julia companions. Chapters 1–3 (vectors, linear functions, norms and angles) are the background for lesson 00. Best for lessons 01–03 and 12.
• Sheldon Axler, Linear Algebra Done Right, 4th edition (free PDF and videos). Proof-based and coordinate-free: vector spaces, linear maps, eigenvalues, inner products, the spectral theorem, the SVD, Jordan form. Read it when you want the clean proofs behind lessons 01–08. The author's short videos for each section are on the same page.
• Margalit and Rabinoff, Interactive Linear Algebra (free online). A first-course textbook with interactive 2-D and 3-D figures. Good for lessons 01–05.
• Deisenroth, Faisal and Ong, Mathematics for Machine Learning (free PDF). Chapters 2–4 cover linear algebra, geometry and matrix decompositions, then the book uses them for regression, PCA and SVMs. A short route from linear algebra to ML.
• Gilbert Strang, Linear Algebra and Learning from Data (book site). The book of 18.065. It is not free, but the site has free sample sections (I.1–I.2 on matrix multiplication, VII.1–VII.2 on deep and convolutional networks) and notes. Strang's Introduction to Linear Algebra is also not free; the free 18.06 and 18.06SC material above covers it.

Numerical linear algebra #

• Driscoll and Braun, Fundamentals of Numerical Computation (free online). Floating point, conditioning, LU and pivoting, norms, QR and least squares, power iteration, Krylov methods, with code in every section. The free choice for lessons 10–14.
• Trefethen and Bau, Numerical Linear Algebra (SIAM 1997; official page). Not free, but the standard text: 40 short lectures that start from the SVD and QR and go to stability, eigenvalue algorithms and Krylov methods. Find it in a library for lessons 12–14.
• Halko, Martinsson and Tropp, "Finding structure with randomness" (SIAM Review 2011; arXiv). The reference for the randomized SVD: range finders, power iterations, error bounds. Lesson 09.
• Martinsson and Tropp, "Randomized numerical linear algebra: foundations and algorithms" (Acta Numerica 2020; arXiv). A longer survey: sketching, randomized least squares, low-rank approximation, trace estimation. Lesson 18.
• Jonathan Shewchuk, "An Introduction to the Conjugate Gradient Method Without the Agonizing Pain" (PDF). Conjugate gradients built up from pictures. Lesson 14.

Data, probability and modern ML #

• Brunton and Kutz, Data-Driven Science and Engineering, 2nd edition (book site, free PDF from the author). Chapter 1 is the SVD, PCA and randomized SVD with code; later chapters use them for dynamics and control. Lessons 08–09.
• Roman Vershynin, High-Dimensional Probability, 2nd edition (free PDF). Concentration of measure, random vectors and matrices, the Johnson–Lindenstrauss lemma. Advanced; read single sections for lesson 18.
• Chen, Chi, Fan and Ma, Spectral Methods for Data Science: A Statistical Perspective (arXiv). Davis–Kahan and Wedin with proofs, and how they give guarantees for PCA, clustering and matrix completion. Lesson 17.
• Bright, Edelman and Johnson, Matrix Calculus (for Machine Learning and Beyond) (arXiv). The notes of MIT 18.S096. Lessons 15–17.
• Petersen and Pedersen, The Matrix Cookbook (PDF). A list of identities for derivatives, traces, inverses and Kronecker products. For lookup, not for learning.
• Nicolas Boumal, An Introduction to Optimization on Smooth Manifolds (free PDF). Stiefel and Grassmann manifolds, Riemannian gradients and retractions. Lesson 19.
• Bronstein, Bruna, Cohen and Veličković, Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges (arXiv). Symmetry and equivariance as the common principle of CNNs, GNNs and transformers. Lesson 19.

Short reads and tools #

• Nick Higham, "What is" series (index). Two-page explanations by a leading numerical analyst: the SVD, Cholesky, backward error, condition numbers, and many more.
• David Goldberg, "What every computer scientist should know about floating-point arithmetic" (online). Rounding, ulps and cancellation. Lesson 13.
• Elhage et al., "A Mathematical Framework for Transformer Circuits" (online). Attention as low-rank linear maps: the residual stream and the QK and OV circuits. Lesson 15.
• Sanchez-Lengeling et al., "A Gentle Introduction to Graph Neural Networks" (Distill). Message passing with interactive figures. Lesson 19.
• matrixcalculus.org (online). Type a matrix expression and get its derivative. Use it to check your hand computations in lesson 16.

Which lesson pairs with what #

The first choice for each lesson. The Watch and read section at the end of each lesson gives more, with exact section numbers.

Lesson	Watch	Read
00 Linear algebra in plain words	3Blue1Brown ch. 1, 14; Visual Kernel, SEE Matrix ch. 3 (SVD); StatQuest, PCA in 5 minutes	VMLS ch. 1–3
01 Matrices, span and basis	3Blue1Brown ch. 2–4; 18.06 lecture 9	VMLS §5, §10.1; Axler §2A–2C
02 Rank and the four subspaces	3Blue1Brown ch. 7; 18.06 lecture 10	Axler §3B; Interactive LA ch. 2 (subspaces to the rank theorem)
03 Inverse, determinant, projections	3Blue1Brown ch. 6, 9; 18.06 lectures 15, 18	VMLS §11
04 Orthogonal matrices, change of basis	3Blue1Brown ch. 13; 18.06 lectures 17, 31	Axler §3D, §7D
05 Eigenvalues and diagonalization	3Blue1Brown ch. 14; 18.06 lectures 21–22	Axler §5D, §8A; Driscoll–Braun §8.2
06 Spectral theorem, Rayleigh quotient	18.065 lecture 18	Axler §7B
07 Positive definite matrices	18.065 lecture 5	Higham, "What is a Cholesky factorization?"
08 The SVD	18.065 lectures 6–7; Brunton SVD series	Brunton–Kutz ch. 1; Higham, "What is the SVD?"
09 PCA and randomized SVD	Brunton SVD series (PCA, randomized SVD)	Halko–Martinsson–Tropp §1, §4; Brunton–Kutz §1.5–1.8
10 The factorization map, LU	18.065 lecture 2; A 2020 Vision	Driscoll–Braun §2.4–2.6
11 Norms	18.065 lecture 8	Driscoll–Braun §2.7
12 QR and least squares	18.065 lectures 9–11; fast.ai	Driscoll–Braun ch. 3; Trefethen–Bau lectures 7–11
13 Floating point and conditioning	18.335	Driscoll–Braun §1.1–1.4; Goldberg
14 Eigenvalue algorithms and Krylov methods	18.335	Driscoll–Braun ch. 8; Shewchuk; Trefethen–Bau lectures 25–38
15 Linear algebra of modern ML	18.S096	Bright–Edelman–Johnson ch. 8; Transformer Circuits framework
16 Matrix calculus and matrix functions	18.S096	Bright–Edelman–Johnson ch. 2–3, 7; Matrix Cookbook
17 Perturbation theory	18.065 lecture 15	Chen–Chi–Fan–Ma §2.2–2.4; Bright–Edelman–Johnson ch. 13
18 Sketching, JL and random matrices	18.065 lecture 13 (randomized matrix multiplication)	Vershynin §5.3; Martinsson–Tropp
19 Structured matrices in ML	18.065 lectures 31, 35; Geometric Deep Learning	GDL book §3–5; Boumal §7.3