Linear algebra in plain words

Before this lesson: nothing. You need vectors as lists of numbers, such as $(2, 1)$ , and the idea of adding two vectors and multiplying a vector by a number. This lesson is an optional tour: every object gets one picture and one formula, and a link to the lesson that treats it in full. Time: about 50 minutes, plus the notebook.

Why this matters #

Almost every model in machine learning is built from matrices. A linear layer computes $y = Wx$ . Attention compares tokens with dot products, $q^\top k$ . LoRA fine-tunes a model with a low-rank update $BA$ . PCA compresses data with the SVD. A covariance matrix is positive definite, and the Hessian of a loss is positive definite near a strict minimum. This lesson gives you the picture behind each of these words, so that the formulas of the next lessons have something to attach to.

1. Predict first #

2. The picture #

A matrix moves space #

Think of a matrix as an action on space, not only as a table of numbers. A $2 \times 2$ matrix $A$ takes every point $x$ of the plane to a new point $Ax$ . Write $e_1 = (1, 0)$ and $e_2 = (0, 1)$ . Then the first column of $A$ is $Ae_1$ and the second column is $Ae_2$ : the columns are where the basic arrows land. Every other point follows. With $a_1, a_2$ the columns of $A$ ,

Ax = x_1\, a_1 + x_2\, a_2 .

So the grid of the plane goes to a grid of parallelograms, and the origin stays where it is.

In ML: a linear layer $y = Wx$ is this move; its output is a combination of the columns of $W$ . Full lesson: lesson 01.

Span, basis and dimension #

The span of some vectors is the set of all points you can reach by combining them: $c_1 v_1 + c_2 v_2 + \dots$ for all numbers $c_i$ . One arrow spans a line, two non-parallel arrows span a plane. Vectors are linearly independent when none of them is a combination of the others, so each one adds a new direction. A basis is a set of independent vectors that spans the whole space: a choice of axes. The dimension is the number of vectors in a basis.

In ML: the columns of a weight matrix span the set of all outputs of the layer. Full lesson: lesson 01.

Rank: the directions that survive #

The rank of a matrix is the number of independent directions that are still there after the move. It is the dimension of the set of outputs. If a $3 \times 3$ matrix flattens space onto a plane, its rank is 2: one direction was squashed to 0. The inputs that go to 0 form the null space. Rank and null space share the input dimensions: $\operatorname{rank} A + \dim N(A) = n$ (lesson 02 proves it).

In ML: low rank means redundancy. LoRA writes a weight update as $BA$ with a small inner size $r$ , so its rank is at most $r$ . Full lesson: lesson 02.

Quick check

A $3 \times 3$ matrix $A$ flattens all of $\mathbb{R}^3$ onto a plane through 0. What are its rank and its determinant?

3. Definitions and main results #

This section is a tour. For each object: one plain sentence, a picture or a small example, one formula, one use in ML, and the lesson that treats it in full. There are no proofs here.

Dot product, length and orthogonality #

The dot product measures how much two vectors point the same way. For $u, v \in \mathbb{R}^n$ ,

u^\top v = u_1 v_1 + \dots + u_n v_n = \|u\|\,\|v\|\cos\theta ,

where $\|u\| = \sqrt{u^\top u}$ is the norm (the length) and $\theta$ is the angle between the vectors. The sign tells the angle: positive means below $90°$ , negative means above $90°$ , and zero means the vectors are orthogonal (perpendicular).

In ML: cosine similarity of embeddings is $\cos\theta$ ; attention scores are dot products $q^\top k$ . Full lesson: lesson 03.

Quick check

$u = (1, 2)$ and $v = (2, -1)$ . What does $u^\top v$ tell you about the two vectors?

Eigenvalues and eigenvectors #

Most directions are turned by a matrix. An eigenvector of a square matrix $A$ is a nonzero vector $v$ whose direction is not turned: $Av$ lies on the line through $v$ . The factor is the eigenvalue $\lambda$ :

Av = \lambda v, \qquad v \ne 0 .

The factor can be larger than 1 (stretch), between 0 and 1 (shrink), negative (the arrow flips to the other side of the line), or 0 (the direction is flattened). A rotation of the plane by $90°$ turns every direction, so it has no real eigenvector; its eigenvalues are the complex numbers $\pm i$ .

Drag $x$ around the circle and watch $Ax$ . Usually $Ax$ points off the line through $x$ . For $A = \begin{pmatrix} 1 & 2 \\ 1 & 0 \end{pmatrix}$ there are two directions where it does not: $(2, 1)$ is stretched by $\lambda = 2$ , and $(1, -1)$ is flipped, $A(1, -1) = (-1, 1)$ , so $\lambda = -1$ . Press Show eigenvectors to check. Then pick rotation 90°: no direction stays on its line.

In ML: the eigenvectors of a covariance matrix are the principal directions of PCA; the eigenvalues of a Hessian give the curvature of a loss. Full lesson: lesson 05 (and lesson 06 for symmetric matrices).

Quick check

$A = \begin{pmatrix} 1 & 2 \\ 1 & 0 \end{pmatrix}$ sends $v = (1, -1)$ to $(-1, 1)$ . What does this say?

Determinant #

The determinant of a square matrix is the factor by which it multiplies areas (in 2D) or volumes (in 3D), with a sign for orientation. For a $2 \times 2$ matrix,

\det \begin{pmatrix} a & b \\ c & d \end{pmatrix} = ad - bc .

Small examples: $\det \begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix} = 6$ , so every area is multiplied by 6. The reflection $\begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}$ has determinant $-1$ : areas are kept, but the plane is mirrored. The matrix $\begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix}$ has parallel columns and $\det = 4 - 4 = 0$ : the plane is flattened onto a line. A determinant of 0 means the move loses a direction and cannot be undone. The determinant is also the product of the eigenvalues.

In ML: the density of a normalizing flow needs $\log|\det J|$ ; the Gaussian likelihood needs $\log\det\Sigma$ . Full lesson: lesson 03.

The SVD: rotate or reflect, stretch, rotate or reflect #

The singular value decomposition (SVD) says that every matrix, square or not, does three simple moves in a row:

A = U\,\Sigma\,V^\top .

First $V^\top$ turns space (a rotation, possibly with a reflection). Then $\Sigma$ stretches along the axes by the singular values $\sigma_1 \ge \sigma_2 \ge \dots \ge 0$ . Then $U$ turns again (again a rotation, possibly with a reflection). So the unit circle always becomes an ellipse, and the half-axes of the ellipse are the singular values.

Singular values are not eigenvalues. Eigenvalues ask "which directions keep their line?"; singular values ask "how long are the axes of the ellipse?". They agree only for special matrices, such as symmetric matrices with no negative eigenvalues. Keeping the largest singular values gives the best low-rank approximation of a matrix (Eckart–Young, lesson 08).

In ML: compression of weight matrices, LoRA initializations, the pseudoinverse in least squares. Full lesson: lesson 08.

Quick check

The shear $A = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$ has eigenvalues $1$ and $1$ . What are its singular values?

PCA: the directions where the data spreads most #

You have $n$ data points in $\mathbb{R}^d$ , the rows of a matrix $X$ . Principal component analysis (PCA) finds the direction in which the cloud of points is most spread out (the first principal component), then the most spread-out direction perpendicular to it, and so on. Describing each point by its first few components compresses the data. The recipe:

Center the data: subtract the mean of each column, so the cloud sits around 0.
Take the SVD of the centered matrix: $X_c = U\Sigma V^\top$ .
The columns of $V$ are the principal directions. The variance along direction $i$ is $\sigma_i^2 / (n - 1)$ .

In ML: visualizing embeddings, whitening inputs, compressing features; in science, empirical orthogonal functions (EOFs) in climate data. Full lesson: lesson 09.

Positive definite matrices #

A symmetric matrix $S$ is positive definite (PD) if $x^\top S x > 0$ for every $x \ne 0$ . Equivalently, all its eigenvalues are positive. Picture the graph of $f(x) = x^\top S x$ : for a PD matrix it is a bowl with a single lowest point at 0. Small examples:

$S = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$ : $x^\top S x = 2x_1^2 + 2x_1x_2 + 2x_2^2 = x_1^2 + x_2^2 + (x_1 + x_2)^2 > 0$ for $x \ne 0$ . PD, a bowl.
$M = \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}$ : $x^\top M x = x_1^2 - x_2^2$ , which is $-1$ at $x = (0, 1)$ . Not PD: a saddle.

In ML: every covariance matrix is positive semidefinite ( $\ge 0$ instead of $> 0$ ); a loss is locally a bowl when its Hessian is PD. Full lesson: lesson 07.

Factorization: the idea behind all of them #

Many results above say the same thing: write a matrix as a product of simple matrices to see what it does. "Factorization" and "decomposition" mean the same.

Factorization	Works for	Simple pieces	Main use	Lesson
$PA = LU$	square ( $P$ swaps rows)	lower and upper triangular	solve $Ax = b$	10
$A = QR$	any, columns independent	orthonormal columns, upper triangular	least squares	12
$S = LL^\top$ (Cholesky)	symmetric PD	lower triangular	Gaussians, fast solves	07
$A = X\Lambda X^{-1}$	square with $n$ independent eigenvectors	eigenvectors, eigenvalues	powers $A^k$ , dynamics	05
$S = Q\Lambda Q^\top$	symmetric	orthonormal eigenvectors, real eigenvalues	quadratic forms, PCA	06
$A = U\Sigma V^\top$	every matrix	orthogonal, diagonal $\ge 0$ , orthogonal	low rank, PCA, pseudoinverse	08

Answers to section 1. (1) Yes. $S(1, 1) = (3, 3) = 3 \cdot (1, 1)$ : the direction $(1, 1)$ is stretched by 3. There is a second one, $(1, -1)$ , which $S$ keeps unchanged: $S(1, -1) = (1, -1)$ , eigenvalue 1. (2) No. Flattening loses a direction: many points land on the same point of the plane, and no matrix can send one point back to many. The rank is 2 and the determinant is 0.

4. Worked example #

One matrix for everything: $S = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$ .

(a) Columns. $Se_1 = (2, 1)$ and $Se_2 = (1, 2)$ . The unit square goes to the parallelogram with corners $(0, 0)$ , $(2, 1)$ , $(3, 3)$ , $(1, 2)$ .

(b) Determinant and rank. $\det S = 2 \cdot 2 - 1 \cdot 1 = 3$ . Every area is multiplied by 3, and the orientation is kept (the sign is $+$ ). The columns are not parallel, so the rank is 2 and $S$ can be undone (it is invertible).

(c) Eigenvectors, by testing directions.

$S(1, 0) = (2, 1)$ : not on the line through $(1, 0)$ . Not an eigenvector.
$S(1, 1) = (2 + 1, 1 + 2) = (3, 3) = 3 \cdot (1, 1)$ . Eigenvector, $\lambda_1 = 3$ .
$S(1, -1) = (2 - 1, 1 - 2) = (1, -1) = 1 \cdot (1, -1)$ . Eigenvector, $\lambda_2 = 1$ .

Two checks: the product $3 \cdot 1 = 3 = \det S$ , and the sum $3 + 1 = 4 = 2 + 2$ , the sum of the diagonal entries (the trace). The two eigenvectors are perpendicular: $(1, 1)^\top (1, -1) = 1 - 1 = 0$ . This always happens for symmetric matrices (lesson 06).

(d) Positive definite. Both eigenvalues are positive, so $S$ is PD. Directly: $x^\top S x = x_1^2 + x_2^2 + (x_1 + x_2)^2 > 0$ for $x \ne 0$ .

(e) The SVD equals the eigendecomposition here. Put the unit eigenvectors in $Q = \tfrac{1}{\sqrt2}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}$ and the eigenvalues in $\Lambda = \operatorname{diag}(3, 1)$ . Then

Q\Lambda Q^\top = \frac12 \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\begin{pmatrix} 3 & 0 \\ 0 & 1 \end{pmatrix}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} = \frac12 \begin{pmatrix} 3 & 1 \\ 3 & -1 \end{pmatrix}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} = \frac12 \begin{pmatrix} 4 & 2 \\ 2 & 4 \end{pmatrix} = S .

$Q$ has orthonormal columns and the eigenvalues are $\ge 0$ , so this is also an SVD with $U = V = Q$ and $\Sigma = \Lambda$ : the singular values are 3 and 1. Note that $\det Q = \tfrac12(-1 - 1) = -1$ : this $Q$ is a reflection, not a rotation. That is why the SVD slogan says "rotate or reflect".

(f) PCA. If $S$ is the covariance matrix of a data cloud, the cloud is most spread out along $(1, 1)/\sqrt2$ (variance 3) and least along $(1, -1)/\sqrt2$ (variance 1). These are the two principal directions.

Quick check

For $S = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$ , compute $S\,(2, -2)$ .

5. In code #

NumPy checks the worked example. The notebook for this lesson has these cells and the exercises.

import numpy as np

S = np.array([[2., 1.], [1., 2.]])
print(S @ [1., 0.], S @ [0., 1.])       # [2. 1.] [1. 2.]: the columns
print(np.linalg.matrix_rank(S))         # 2
print(round(np.linalg.det(S), 12))      # 3.0

lam, Q = np.linalg.eigh(S)    # symmetric S; eigenvalues in increasing order
print(lam)                    # [1. 3.]
print(np.allclose(S @ Q, Q * lam))      # True: S q_i = lam_i q_i
print(np.linalg.svd(S, compute_uv=False))   # [3. 1.]: same as the eigenvalues

A = np.array([[1., 1.], [0., 1.]])      # the shear
print(np.linalg.eigvals(A).real)        # [1. 1.]
print(np.linalg.svd(A, compute_uv=False))   # [1.61803399 0.61803399]

# PCA = SVD of the centered data
rng = np.random.default_rng(0)
X = rng.multivariate_normal([5., -2.], S, size=500)  # covariance S
Xc = X - X.mean(axis=0)                              # step 1: center
U, s, Vt = np.linalg.svd(Xc, full_matrices=False)    # step 2: SVD
print(np.round(Vt, 2))       # rows ±(0.71, 0.7) and ±(-0.7, 0.71)
print(np.round(s**2 / (len(X) - 1), 2))    # [2.91 0.94]: close to 3 and 1

The signs of eigenvectors and singular vectors are arbitrary: $-v$ is as good as $v$ . The sample variances 2.91 and 0.94 are close to, but not equal to, 3 and 1, because 500 points are a finite sample.

6. Where it shows up #

Neural networks. Every linear layer is a matrix acting on a vector; a network is a chain of such moves with nonlinear functions in between.
Attention. The score of a query against a key is the dot product $q^\top k$ ; the score matrix $QK^\top$ has rank at most the head dimension.
Fine-tuning. LoRA learns a low-rank update $\Delta W = BA$ with a small rank $r$ .
Optimization. Near a strict minimum the Hessian is positive definite (a bowl); its eigenvalues set how fast gradient descent converges in each direction.
Data analysis. PCA (via the SVD) for visualization and compression; in climate science the same method is called EOF analysis, and in fluid dynamics proper orthogonal decomposition (POD).
Probabilistic models. Covariance matrices are positive semidefinite; Gaussian likelihoods and normalizing flows need log-determinants.

7. Common mistakes #

"An eigenvector keeps its direction exactly, so $\lambda > 0$ ." It only has to stay on its line. $\lambda$ can be negative (the arrow flips), zero, or complex (a rotation has no real eigenvector).
"Singular values are the eigenvalues." Only for symmetric matrices with no negative eigenvalues. The shear has eigenvalues $1, 1$ and singular values $1.618, 0.618$ .
"PCA on raw data." Without centering, the first direction points towards the mean of the data, not along the spread. Subtract the column means first.
"The SVD is rotate, stretch, rotate." $U$ and $V$ are orthogonal matrices, which can be reflections ( $\det = -1$ ), as in the worked example.
"Independent means pairwise not parallel." $(1, 0)$ , $(0, 1)$ and $(1, 1)$ are pairwise not parallel, but they are dependent: $(1, 1) = (1, 0) + (0, 1)$ .

8. Check yourself #

Which $2 \times 2$ matrix rotates the plane by $90°$ counter-clockwise? Use the images of $e_1$ and $e_2$ .
Answer
$e_1 \mapsto (0, 1)$ and $e_2 \mapsto (-1, 0)$ , so $A = \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}$ . It has no real eigenvector: every direction is turned.
What is the rank of $\begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix}$ , and what does it do to the plane?
Answer
Rank 1: the second column is twice the first. The plane is flattened onto the line through $(1, 2)$ ; the direction $(2, -1)$ goes to 0. The determinant is $4 - 4 = 0$ .
$u = (3, 4)$ , $v = (4, 3)$ . Compute $u^\top v$ and $\cos\theta$ . Is the angle acute?
Answer
$u^\top v = 12 + 12 = 24$ , $\|u\| = \|v\| = 5$ , so $\cos\theta = 24/25 = 0.96$ . Positive, so the angle is acute (about $16.3°$ ).
A $2 \times 2$ matrix has eigenvalues $4$ and $-0.5$ . What is its determinant? Does it keep the orientation of the plane?
Answer
$\det = 4 \cdot (-0.5) = -2$ . Areas are doubled, and the negative sign means the orientation is reversed (a mirror image).
Why must you center the data before you take the SVD in PCA?
Answer
PCA measures spread around the mean. The SVD of the raw data measures spread around 0, so if the cloud is far from 0, the first singular vector points towards the cloud's mean, not along its long axis.

Key takeaways

A matrix moves space. Its column $j$ is $Ae_j$ , and $Ax = x_1 a_1 + \dots + x_n a_n$ .
Span = everything reachable by combinations; basis = independent vectors that span; dimension = the size of a basis.
Rank = the number of independent directions that survive; $\operatorname{rank} A + \dim N(A) = n$ .
$u^\top v = \|u\|\,\|v\|\cos\theta$ : positive for acute angles, zero for perpendicular vectors, negative for obtuse angles.
An eigenvector stays on its own line: $Av = \lambda v$ . The eigenvalue $\lambda$ can be negative, zero or complex.
$\det A$ is the signed volume factor; $\det A = 0$ means a direction is lost and $A$ cannot be undone.
Every matrix has an SVD $A = U\Sigma V^\top$ : rotate or reflect, stretch by the singular values, rotate or reflect. Singular values are not eigenvalues in general.
PCA is the SVD of the centered data matrix; a symmetric matrix is positive definite when $x^\top S x > 0$ for all $x \ne 0$ , that is, when all its eigenvalues are positive.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

What is column $j$ of a matrix $A$ , seen as a map?
What does the rank of a matrix count?
A $3 \times 3$ matrix flattens space onto a plane. Give its rank and its determinant.
What does the sign of $u^\top v$ tell you?
Define eigenvector and eigenvalue.
Why does a $90°$ rotation of the plane have no real eigenvector?
What does $\det A$ measure?
What three moves make up the SVD $A = U\Sigma V^\top$ ?
How do you compute PCA with the SVD?
When is a symmetric matrix positive definite?

Open the review →

9. Interview angle #

"Explain eigenvectors in one sentence." — The directions that a matrix keeps on their own line; the eigenvalue is the stretch factor along that line, and it can be negative.
"What is the difference between the eigendecomposition and the SVD?" — The eigendecomposition needs a square matrix with enough eigenvectors, and its eigenvectors need not be orthogonal. The SVD exists for every matrix and uses two orthonormal bases.
"How is PCA related to the SVD?" — PCA is the SVD of the centered data matrix: right singular vectors are the principal directions, $\sigma_i^2/(n-1)$ are the variances.
"What does a determinant of 0 mean?" — The matrix flattens space: rank below $n$ , a nonzero null space, no inverse.

10. Practice #

The notebook for this lesson runs in Colab. It has three short exercises, each with a check cell and a folded solution:

Name the map: look at what four matrices do to a grid and name each one (stretch, rotation, shear, reflection, flatten).
Find the eigenpairs of a symmetric matrix with np.linalg.eigh, and check $Av = \lambda v$ .
PCA of a 2D cloud with the SVD of the centered data, compared with sklearn.decomposition.PCA.

11. Watch and read #

Three views of a vector (an arrow, a list of numbers, an element you can add and scale).

Open on YouTube · the player loads from youtube-nocookie.com when you press play

Watch for the directions that stay on their own span, and for the rotation with no real eigenvector.

Open on YouTube · the player loads from youtube-nocookie.com when you press play

The circle becomes an ellipse; the three factors are shown one at a time.

Open on YouTube · the player loads from youtube-nocookie.com when you press play

The main idea of PCA on a small data set, without formulas.

Open on YouTube · the player loads from youtube-nocookie.com when you press play

3Blue1Brown, Essence of linear algebra: the whole series, with text versions of each chapter. Watch it once for the pictures.
Gilbert Strang, A 2020 Vision of Linear Algebra (MIT OpenCourseWare): six short lectures on the same objects, from $A = CR$ to the SVD.
Boyd and Vandenberghe, Introduction to Applied Linear Algebra (VMLS), free PDF: chapters 1–3 (vectors, linear functions, norms and angles) for the beginner background.