Inverse, determinant, dot product and orthogonal projection

Before this lesson: lesson 01 (matrices as maps, span, basis, independence) and lesson 02 (rank, rank–nullity, the four subspaces). Eigenvalues appear once, for $\det A = \lambda_1 \cdots \lambda_n$; here you only need the definition $Av = \lambda v$ with $v \ne 0$ (lesson 05 treats eigenvalues in full). Time: about 70 minutes, plus the notebook.

---

Why this matters #

Three questions come up in almost every numerical method. Can I undo this map (inverse)? How much does it change volumes (determinant)? What is the closest point in a subspace (projection)? Least squares is a projection. Gaussian likelihoods and normalizing flows need $\log\det$. Attention scores and cosine similarity are dot products. And in practice you almost never form $A^{-1}$.

1. Predict first #

2. The picture #

Inverse. $A^{-1}$ undoes $A$: $A^{-1}(Ax) = x$. This is possible exactly when $A$ loses nothing. No nonzero input may go to 0 (otherwise two inputs give the same output), and every output must be reached. For a square matrix these two conditions are the same, by rank–nullity:

Interactive figure: loading…

Determinant in 2D. The columns $(a, c)$ and $(b, d)$ of $A = \begin{pmatrix} a & b \\ c & d \end{pmatrix}$ span a parallelogram of area $|ad - bc|$. The sign of $\det A = ad - bc$ is $+$ if $A$ keeps the orientation (the turn from the first column to the second is counter-clockwise, as from $e_1$ to $e_2$) and $-$ if it makes a mirror image. Every area is multiplied by $|\det A|$. If $\det A = 0$, the plane is squeezed onto a line or a point.

Interactive figure: loading…

Dot product and projection. $u^\top v = \|u\| \|v\| \cos\theta$, where $\theta$ is the angle between $u$ and $v$. The projection $p$ of $b$ onto a line or a subspace is the foot of the perpendicular from $b$: the error $b - p$ is perpendicular to every vector of the subspace. Section 3 shows both in figures.

3. Definitions and main results #

Inverse #

A square $n \times n$ matrix $A$ is invertible if there is a matrix $A^{-1}$ with $A^{-1}A = AA^{-1} = I$. The inverse is unique: if $BA = I$ and $AC = I$, then $B = B(AC) = (BA)C = C$.

Theorem (invertible matrix theorem). For a square $n \times n$ matrix $A$, these statements are equivalent:

• (a) $A$ is invertible;
• (b) $N(A) = \{0\}$: $Ax = 0$ only for $x = 0$;
• (c) $\operatorname{rank} A = n$: elimination (with row exchanges) gives $n$ nonzero pivots;
• (d) the columns of $A$ are independent, so they form a basis of $\mathbb{R}^n$;
• (e) $C(A) = \mathbb{R}^n$: $Ax = b$ has a solution for every $b$;
• (f) $Ax = b$ has exactly one solution for every $b$;
• (g) $\det A \ne 0$;
• (h) 0 is not an eigenvalue of $A$;
• (i) $A^\top$ is invertible (so the rows are independent too);
• (j) there is a $B$ with $BA = I$, or there is a $C$ with $AC = I$ (one side is enough).

Lesson 08 adds one more: all singular values are positive. The short version: a square matrix is invertible exactly when it loses no direction.

Proof of the main links

1. (a) ⇒ (b): if $Ax = 0$, then $x = A^{-1}Ax = 0$.
2. (b) ⇔ (d) is the null-space test of lesson 01. (b) ⇔ (c) ⇔ (e) is rank–nullity: $\dim N(A) = n - \operatorname{rank} A$ and $\dim C(A) = \operatorname{rank} A$.
3. (e) and (b) give (f): a solution exists, and two solutions differ by a vector of $N(A) = \{0\}$.
4. (e) ⇒ (a): choose $c_j$ with $Ac_j = e_j$ and set $C = [c_1 \; \cdots \; c_n]$, so $AC = I$. Then $A(CA - I) = ACA - A = 0$, so every column of $CA - I$ lies in $N(A)$, which is $\{0\}$ by step 2. Hence $CA = I$ too.
5. (b) ⇔ (h): a nonzero $x$ with $Ax = 0 = 0 \cdot x$ is exactly an eigenvector for $\lambda = 0$.
6. (c) ⇔ (g): $\det A = \pm$ (product of the pivots), shown below.
7. (i): $\operatorname{rank} A^\top = \operatorname{rank} A$ (row rank = column rank, lesson 02).
8. (j): $BA = I$ gives (b) as in step 1; $AC = I$ gives (e), because $b = A(Cb)$ for every $b$. $\blacksquare$

Theorem (one side is enough). If $A$ and $B$ are $n \times n$ and $BA = I$, then $AB = I$.

Proof.

1. $BA = I$ gives $N(A) = \{0\}$: if $Ax = 0$, then $x = BAx = B0 = 0$.
2. $A$ is square, so rank–nullity gives $\operatorname{rank} A = n$ and $C(A) = \mathbb{R}^n$. So each $e_j = Ac_j$ for some $c_j$; with $C = [c_1 \; \cdots \; c_n]$ we get $AC = I$.
3. $B = B(AC) = (BA)C = C$, so $AB = AC = I$. $\blacksquare$

This needs a square matrix. For $A = \begin{pmatrix} 1 \\ 0 \end{pmatrix}$ ($2 \times 1$) and $B = \begin{pmatrix} 1 & 0 \end{pmatrix}$, $BA = [1] = I_1$, but $AB = \begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix} \ne I_2$. A tall matrix with independent columns has left inverses, for example $(A^\top A)^{-1}A^\top$, but no right inverse: $\operatorname{rank}(AB) \le \operatorname{rank} A < m$.

Rules. $(AB)^{-1} = B^{-1}A^{-1}$ (undo the last step first), $(A^\top)^{-1} = (A^{-1})^\top$, and

In practice: solve, do not invert. To solve $Ax = b$, factor $A = LU$ once (about $\tfrac23 n^3$ flops, lesson 10) and do two triangular solves ($2n^2$ flops). Forming $A^{-1}$ costs about $2n^3$ flops, three times more. And $x = A^{-1}b$ is less accurate: its residual $\|b - A\hat x\|$ can be larger by a factor up to about $\kappa(A)$, the condition number (section 5 shows it).

Determinant #

The determinant is the unique function of square matrices with three rules:

1. $\det I = 1$;
2. exchanging two rows flips the sign;
3. $\det$ is linear in each row separately (the other rows fixed).

Everything else follows from these rules:

• two equal rows give $\det = 0$ (exchanging them changes nothing, but must flip the sign);
• subtracting a multiple of one row from another does not change $\det$ (rule 3, then the previous point);
• a triangular matrix has $\det$ = product of its diagonal entries;
• so elimination gives $\det A = \pm$ (product of the pivots), with $-$ for an odd number of row exchanges; hence $\det A = 0 \iff$ a pivot is missing $\iff A$ is singular;
• $\det A^\top = \det A$.

For $2 \times 2$ this gives $ad - bc$.

Determinant by elimination, step by step. Take

1. The first pivot position holds 0, so exchange rows 1 and 2. One exchange: the sign flips. $\begin{pmatrix} 1 & 1 & 0 \\ 0 & 2 & 1 \\ 2 & 4 & 3 \end{pmatrix}$, $\det = -\det A$.
2. Row 3 − 2·row 1: $(2, 4, 3) - (2, 2, 0) = (0, 2, 3)$. The determinant does not change.
3. Row 3 − row 2: $(0, 2, 3) - (0, 2, 1) = (0, 0, 2)$. The determinant does not change.
4. Now $U = \begin{pmatrix} 1 & 1 & 0 \\ 0 & 2 & 1 \\ 0 & 0 & 2 \end{pmatrix}$ is triangular: $\det U = 1 \cdot 2 \cdot 2 = 4$.
5. One exchange in total, so $\det A = (-1)^1 \cdot 4 = -4$.

Check by cofactor expansion along row 1: $0 \cdot (1 \cdot 3 - 0 \cdot 4) - 2 \cdot (1 \cdot 3 - 0 \cdot 2) + 1 \cdot (1 \cdot 4 - 1 \cdot 2) = 0 - 6 + 2 = -4$ ✓. Cofactor expansion costs about $n!$ operations; libraries use the pivots of LU, about $\tfrac23 n^3$ flops.

Geometric meaning. $|\det A|$ is the volume of the box (parallelepiped) spanned by the columns: the image of the unit cube. Since $\det A^\top = \det A$, the rules also hold for columns, and then they are the rules of volume: the unit cube has volume 1, stretching one edge by $c$ scales the volume by $c$, and a shear keeps base and height. The sign records orientation. Every small cube is scaled by the same factor, so every region's volume is multiplied by $|\det A|$.

Theorem (product rule). $\det(AB) = \det A \cdot \det B$.

Proof.

1. If $B$ is singular, so is $AB$ ($Bx = 0$ with $x \ne 0$ gives $ABx = 0$). Both sides are 0.
2. If $\det B \ne 0$, define $D(A) = \det(AB) / \det B$ as a function of $A$.
3. $D(I) = \det B / \det B = 1$: rule 1.
4. Row $i$ of $AB$ is (row $i$ of $A$)$\,B$. So exchanging two rows of $A$ exchanges the same rows of $AB$, and $D$ flips sign: rule 2.
5. Row $i$ of $AB$ depends linearly on row $i$ of $A$, so $D$ is linear in each row of $A$: rule 3.
6. The three rules fix the determinant, so $D(A) = \det A$. $\blacksquare$

The picture says the same: $AB$ means "apply $B$, then $A$". Volumes are scaled by $\det B$, then by $\det A$, and the signs multiply. Consequences: $\det(A^{-1}) = 1/\det A$; $\det(M^{-1}AM) = \det A$; $\det(cA) = c^n \det A$ (each of the $n$ rows is scaled by $c$).

Theorem (determinant = product of the eigenvalues). Let $A$ be $n \times n$ with eigenvalues $\lambda_1, \dots, \lambda_n$, complex ones included, each repeated as often as its algebraic multiplicity. Then $\det A = \lambda_1 \lambda_2 \cdots \lambda_n$ and $\operatorname{tr} A = \lambda_1 + \dots + \lambda_n$.

The idea: $\lambda$ is an eigenvalue exactly when $A - \lambda I$ loses a direction, that is, when $\det(A - \lambda I) = 0$. This determinant is a polynomial in $\lambda$, the characteristic polynomial, and its roots are the eigenvalues. Lesson 05 develops it; the proof below needs only these two facts. Check on a small case: $\begin{pmatrix} 4 & 1 \\ 2 & 3 \end{pmatrix}$ has eigenvalues 5 and 2 (section 4), and $5 \cdot 2 = 10 = \det A$.

Proof of $\det A = \lambda_1 \cdots \lambda_n$

1. $\lambda$ is an eigenvalue $\iff A - \lambda I$ is singular $\iff p(\lambda) = \det(A - \lambda I) = 0$.
2. $p$ is a polynomial of degree $n$ in $\lambda$ with leading term $(-\lambda)^n$ (it comes from the product of the diagonal entries).
3. Over $\mathbb{C}$, $p$ factors completely into its roots, the eigenvalues: $p(\lambda) = (\lambda_1 - \lambda)(\lambda_2 - \lambda) \cdots (\lambda_n - \lambda)$. The leading term $(-\lambda)^n$ matches.
4. Put $\lambda = 0$: $\det A = p(0) = \lambda_1 \lambda_2 \cdots \lambda_n$. $\blacksquare$

The coefficient of $\lambda^{n-1}$ gives $\operatorname{tr} A = \lambda_1 + \dots + \lambda_n$ in the same way: in the product of the diagonal entries $(A_{11} - \lambda)\cdots(A_{nn} - \lambda)$ it is $(-1)^{n-1}(A_{11} + \dots + A_{nn})$, the other terms of $p$ have degree at most $n - 2$, and in the factored form it is $(-1)^{n-1}(\lambda_1 + \dots + \lambda_n)$.

Numerics. The determinant is a bad test of near-singularity. $\det(0.1\, I_{100}) = 10^{-100}$, but $0.1\, I$ is perfectly conditioned ($\kappa = 1$). $\operatorname{diag}(10^5, 10^{-5})$ has $\det = 1$ but $\kappa = 10^{10}$. Use singular values to judge near-singularity (lesson 13), and compute $\log|\det A|$ from a factorization (slogdet, or Cholesky) when you need it.

Dot product, norm, orthogonality #

For $u, v \in \mathbb{R}^n$: the dot product is $u^\top v = \sum_i u_i v_i$, the norm is $\|v\| = \sqrt{v^\top v}$, and for nonzero $u, v$ the angle $\theta \in [0, \pi]$ satisfies $\cos\theta = u^\top v / (\|u\| \|v\|)$. The vectors are orthogonal ($u \perp v$) if $u^\top v = 0$.

The picture: a shadow. Drop a perpendicular from the tip of $v$ onto the line through $u$. The foot of the perpendicular is the shadow of $v$. Its signed length is $\|v\|\cos\theta$, and

Interactive figure: loading…

• Cauchy–Schwarz: $|u^\top v| \le \|u\| \|v\|$, so the cosine lies in $[-1, 1]$ and the angle is well defined. In the picture: a shadow is never longer than the vector that casts it. Proof for $v \ne 0$: $0 \le \|u - tv\|^2 = \|u\|^2 - 2t\,u^\top v + t^2 \|v\|^2$ for every number $t$; put $t = u^\top v / \|v\|^2$ to get $0 \le \|u\|^2 - (u^\top v)^2/\|v\|^2$.
• Pythagoras: $\|u + v\|^2 = \|u\|^2 + 2u^\top v + \|v\|^2$, so $\|u + v\|^2 = \|u\|^2 + \|v\|^2 \iff u \perp v$.
• Norm rules: $\|v\| \ge 0$ with $\|v\| = 0$ only for $v = 0$; $\|cv\| = |c| \|v\|$; $\|u + v\| \le \|u\| + \|v\|$ (from Cauchy–Schwarz).
• Nonzero, mutually orthogonal vectors are independent: take the dot product of $\sum_j c_j q_j = 0$ with $q_i$ to get $c_i \|q_i\|^2 = 0$.

Projection onto a line #

Let $a \ne 0$ and $b$ be vectors in $\mathbb{R}^m$. The projection $p = \hat x a$ is the point of the line through $a$ closest to $b$: the shadow of $b$, now as a vector.

1. The error $e = b - \hat x a$ is orthogonal to $a$: $a^\top(b - \hat x a) = 0$.
2. So $\hat x = a^\top b / a^\top a$ and

$P$ is $m \times m$ with rank 1, $P^\top = P$ and $P^2 = P$.

Interactive figure: loading…

Projection onto a subspace #

Theorem (projection onto $C(A)$). Let $A$ be $m \times n$ with independent columns, and $b \in \mathbb{R}^m$. The point of $C(A)$ closest to $b$ is $p = A\hat x$, where $\hat x$ solves the normal equations $A^\top A \hat x = A^\top b$. So $p = Pb$ with

Derivation in four steps, then the proof that it is closest.

1. Write $p$ with unknown coordinates. $p$ lies in $C(A)$, so $p = A\hat x$ for some $\hat x \in \mathbb{R}^n$.
2. Ask for a perpendicular error. The error $e = b - A\hat x$ must be orthogonal to every column $a_j$ of $A$: $a_j^\top(b - A\hat x) = 0$ for $j = 1, \dots, n$.
3. Stack the $n$ equations. The rows of $A^\top$ are the $a_j^\top$, so step 2 is $A^\top(b - A\hat x) = 0$, that is, $A^\top A \hat x = A^\top b$. (So $e \in N(A^\top)$, the left null space of lesson 02.)
4. Solve. $A^\top A$ ($n \times n$) is invertible, because $\operatorname{rank}(A^\top A) = \operatorname{rank}(A) = n$ (lesson 02: $A^\top A x = 0$ gives $\|Ax\|^2 = 0$, so $Ax = 0$ and $x = 0$). Hence $\hat x = (A^\top A)^{-1}A^\top b$ and $p = A(A^\top A)^{-1}A^\top b = Pb$.
5. $p$ is the closest point. For any $x$, $b - Ax = (b - p) + (p - Ax)$ with $b - p \perp C(A)$ and $p - Ax \in C(A)$. Pythagoras gives $\|b - Ax\|^2 = \|b - p\|^2 + \|p - Ax\|^2 \ge \|b - p\|^2$, with equality only when $Ax = p$. $\blacksquare$

For one column ($n = 1$, $A = a$) these steps give back the line formula: $A^\top A = a^\top a$ is a number.

Interactive figure: loading…

Properties of $P$.

• $P^2 = P$: $A(A^\top A)^{-1}(A^\top A)(A^\top A)^{-1}A^\top = P$. Projecting twice changes nothing.
• $P^\top = P$, because $A^\top A$ is symmetric. A matrix with $P^2 = P$ and $P^\top = P$ is an orthogonal projector.
• $I - P$ projects onto $C(A)^\perp = N(A^\top)$: $(I - P)b = e$.
• The eigenvalues are 1 (on $C(A)$) and 0 (on $N(A^\top)$). So $\operatorname{tr} P = \operatorname{rank} P = n$, and $\det P = 0$ unless $n = m$ (then $P = I$).
• Orthonormal columns. If $Q$ has orthonormal columns ($Q^\top Q = I$), then $P = QQ^\top$ and $p = \sum_i (q_i^\top b)\, q_i$. The projection onto the span is the sum of the projections onto the lines through the basis vectors (for every $b$) only when these vectors are mutually orthogonal.
• Dependent columns. Then $A^\top A$ is singular and the formula fails. The projection onto $C(A)$ still exists: use a basis of $C(A)$, or $P = AA^+$ with the pseudoinverse (lesson 08).

In practice you do not form the $m \times m$ matrix $P$, and you do not invert $A^\top A$: lesson 12 solves least squares with QR, because $\kappa(A^\top A) = \kappa(A)^2$.

Answers to section 1. (1) Area 3. By the product rule $|\det(AB)| = |\det A|\,|\det B| = 6 \cdot \tfrac12$. (2) Not in general: it holds for every $b$ only if $a_1 \perp a_2$. Section 4(c) shows a counterexample.

4. Worked example #

(a) Inverse and determinant. $A = \begin{pmatrix} 4 & 1 \\ 2 & 3 \end{pmatrix}$: $\det A = 4 \cdot 3 - 1 \cdot 2 = 10$, so

The unit square goes to the parallelogram spanned by $(4, 2)$ and $(1, 3)$, with area 10 and the orientation kept. The eigenvalues solve $\lambda^2 - (\operatorname{tr} A)\lambda + \det A = \lambda^2 - 7\lambda + 10 = 0$, so $\lambda = 5, 2$: product $10 = \det A$ ✓, sum $7 = \operatorname{tr} A$ ✓. With the swap $S = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}$ ($\det S = -1$), $AS = \begin{pmatrix} 1 & 4 \\ 3 & 2 \end{pmatrix}$ (the columns of $A$ swapped) has $\det(AS) = 2 - 12 = -10 = \det A \cdot \det S$: same area, reversed orientation. (Pick columns swapped in the signed-area figure.)

(b) Projection onto a line. $a = (1, 2, 2)$, $b = (3, 0, 3)$. Then $a^\top b = 3 + 0 + 6 = 9$ and $a^\top a = 9$, so $\hat x = 1$, $p = (1, 2, 2)$ and $e = b - p = (2, -2, 1)$. Check: $a^\top e = 2 - 4 + 2 = 0$ ✓.

(c) Projection onto a plane. $A = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{pmatrix}$, $b = (1, 2, 6)$. Follow the four steps.

Step 1. $p = A\hat x$ with unknown $\hat x = (\hat x_1, \hat x_2)$.

Steps 2 and 3. The error must be perpendicular to both columns, which gives the normal equations $A^\top A\hat x = A^\top b$. They need $A^\top A$ and $A^\top b$:

Step 4. Solve with the $2 \times 2$ inverse formula ($\det A^\top A = 3$):

So $p = A\hat x = (2, 3, 5)$ and $e = b - p = (-1, -1, 1)$. Check: $A^\top e = (-1 + 1, -1 + 1) = (0, 0)$ ✓. The plane is $x_3 = x_1 + x_2$, with normal $(1, 1, -1)$, and $e$ is parallel to that normal. Pythagoras: $\|b\|^2 = 41 = \|p\|^2 + \|e\|^2 = 38 + 3$ ✓. The projection matrix is

Check $P^\top = P$ and $P^2 = P$. $P$ is symmetric: the $(1, 2)$ and $(2, 1)$ entries are both $-\tfrac13$, and so on. For $P^2$, row 1 of $P$ times column 1 of $P$ is $\tfrac19\,(2 \cdot 2 + (-1)(-1) + 1 \cdot 1) = \tfrac69 = \tfrac23$, the $(1,1)$ entry of $P$; row 1 times column 2 is $\tfrac19\,(2 \cdot (-1) + (-1) \cdot 2 + 1 \cdot 1) = -\tfrac39 = -\tfrac13$, the $(1,2)$ entry. The other entries work the same way (the code in section 5 checks all of them). And $Pb = \tfrac13(2 - 2 + 6,\ -1 + 4 + 6,\ 1 + 2 + 12) = (2, 3, 5) = p$ ✓.

The wrong way. Project $b$ onto each column separately and add: $\tfrac{7}{2}(1, 0, 1) + \tfrac{8}{2}(0, 1, 1) = (3.5, 4, 7.5) \ne (2, 3, 5)$. It fails because $a_1^\top a_2 = 1 \ne 0$: the columns are not orthogonal.

5. In code #

The libraries check section 4. The notebook for this lesson has these cells and the exercises.

import numpy as np

# (a) inverse, determinant, eigenvalues
A = np.array([[4., 1.], [2., 3.]])
print(round(np.linalg.det(A), 12))                   # 10.0
print(np.round(10 * np.linalg.inv(A), 12))           # [[ 3. -1.] [-2.  4.]]
print(np.sort(np.linalg.eigvals(A).real))            # [2. 5.]
S = np.array([[0., 1.], [1., 0.]])                   # swap: det S = -1
print(round(np.linalg.det(A @ S), 12))               # -10.0

# determinant by elimination (section 3): one row exchange, pivots 1, 2, 2
M = np.array([[0., 2., 1.], [1., 1., 0.], [2., 4., 3.]])
print(round(np.linalg.det(M), 12))                   # -4.0

# (c) projection onto C(A); lstsq returns the x-hat of the normal equations
A = np.array([[1., 0.], [0., 1.], [1., 1.]])
b = np.array([1., 2., 6.])
x_hat = np.linalg.lstsq(A, b, rcond=None)[0]
p = A @ x_hat
print(np.round(x_hat, 12), np.round(p, 12))          # [2. 3.] [2. 3. 5.]
print(np.abs(A.T @ (b - p)).max() < 1e-12)           # True: e is orthogonal to C(A)
P = A @ np.linalg.pinv(A)                            # equals A (AᵀA)⁻¹ Aᵀ here
print(np.allclose(P @ P, P), np.allclose(P, P.T), round(np.trace(P), 12))   # True True 2.0

# a tiny determinant does not mean "nearly singular"
B = 0.1 * np.eye(100)
print(np.linalg.det(B), np.linalg.cond(B))           # about 1e-100, and 1.0
print(np.linalg.slogdet(B))                          # sign 1.0, log|det| = -230.258...

And the reason to solve instead of invert, on the Hilbert matrix $H_{ij} = 1/(i + j - 1)$ of size 8 ($\kappa \approx 1.5 \cdot 10^{10}$):

import numpy as np
from scipy.linalg import hilbert

H = hilbert(8)
b = H @ np.random.default_rng(0).standard_normal(8)
for name, x in [("solve", np.linalg.solve(H, b)), ("inv @ b", np.linalg.inv(H) @ b)]:
    rel_res = np.linalg.norm(b - H @ x) / (np.linalg.norm(H, 2) * np.linalg.norm(x))
    print(f"{name:8s} relative residual {rel_res:.0e}")
# solve    relative residual 5e-17   (backward stable)
# inv @ b  relative residual 1e-08   (about 10^8 times worse)

6. Where it shows up #

• Linear regression. The fitted values are $\hat y = P_X y$ with the "hat matrix" $P_X = X(X^\top X)^{-1}X^\top$, the projector onto the span of the features. The residuals are orthogonal to every feature, and $\operatorname{tr} P_X$ = the number of parameters.
• Gaussian models. $\log \mathcal{N}(x; \mu, \Sigma)$ contains $-\tfrac12 \log\det\Sigma$. Compute it as $2\sum_i \log L_{ii}$ from the Cholesky factor $\Sigma = LL^\top$ (lesson 07), never as $\log(\det \Sigma)$, which can underflow or overflow. Gaussian-process regression needs it at every hyperparameter step.
• Normalizing flows. For an invertible map $z = f(x)$, $\log p_X(x) = \log p_Z(f(x)) + \log|\det J_f(x)|$. A general determinant costs $O(n^3)$, so coupling layers (RealNVP, Glow) make $J_f$ triangular: $\log|\det J_f| = \sum_i \log|J_{ii}|$ costs $O(n)$.
• Attention and embeddings. Attention scores are dot products $q^\top k$ (scaled by $1/\sqrt{d_k}$); cosine similarity of embeddings is $\cos\theta = u^\top v/(\|u\|\|v\|)$.
• ML for science: hard constraints. If a surrogate model must satisfy a linear conservation law $Cx = 0$ (for example a discrete mass balance, $C$ with independent rows), you can enforce it exactly by projecting the prediction onto $N(C)$: $x \mapsto x - C^\top(CC^\top)^{-1}Cx$. This is $I - P$ with $P$ the projector onto $C(C^\top)$ (take $A = C^\top$ in the theorem). The same idea in function space is the Leray (Helmholtz) projection onto divergence-free velocity fields.

7. Common mistakes #

• $\det(cA) = c \det A$ and $\det(A + B) = \det A + \det B$. Both wrong: $\det(cA) = c^n \det A$ for $n \times n$ matrices, and $\det$ is not additive.
• Forgetting the sign of row exchanges. $\det A = (-1)^{\#\text{exchanges}} \times$ (product of the pivots).
• $(AB)^{-1} = A^{-1}B^{-1}$. Wrong order: $(AB)^{-1} = B^{-1}A^{-1}$.
• x = np.linalg.inv(A) @ b. Use np.linalg.solve(A, b) or a factorization.
• Adding projections onto non-orthogonal basis vectors. $p = \sum_i (q_i^\top b) q_i$ needs orthonormal $q_i$.
• $A(A^\top A)^{-1}A^\top$ with dependent columns. $A^\top A$ is singular; use a basis of $C(A)$.

8. Check yourself #

1. $A$ is $3 \times 3$ with $\det A = 2$. Find $\det(2A)$, $\det(A^{-1})$ and $\det(A^\top A)$.
   Answer

   $\det(2A) = 2^3 \cdot 2 = 16$; $\det(A^{-1}) = 1/2$; $\det(A^\top A) = \det A^\top \det A = 4$.
2. A $3 \times 3$ matrix has eigenvalues $2, -1, 3$. Is it invertible? Give $\det A$ and $\operatorname{tr} A$.
   Answer

   Yes: 0 is not an eigenvalue. $\det A = 2 \cdot (-1) \cdot 3 = -6$, $\operatorname{tr} A = 2 - 1 + 3 = 4$.
3. Project $b = (2, 1, 0)$ onto the line through $a = (1, 1, 1)$. Give $p$ and $e$, and check.
   Answer

   $a^\top b = 3$, $a^\top a = 3$, so $p = (1, 1, 1)$ and $e = (1, 0, -1)$; $a^\top e = 0$ ✓.
4. Why does a $3 \times 2$ matrix $A$ with independent columns have a left inverse but no right inverse?
   Answer

   Left: $(A^\top A)^{-1}A^\top A = I_2$, and $A^\top A$ is invertible because the columns are independent. Right: $AB = I_3$ would need rank 3, but $\operatorname{rank}(AB) \le \operatorname{rank} A = 2$.
5. $P$ is the orthogonal projector onto a $k$-dimensional subspace of $\mathbb{R}^m$. Give its eigenvalues, trace and determinant.
   Answer

   Eigenvalue 1 ($k$ times, on the subspace) and 0 ($m - k$ times, on its orthogonal complement). $\operatorname{tr} P = k$. $\det P = 0$ unless $k = m$, where $P = I$ and $\det P = 1$.
6. Compute $\det \begin{pmatrix} 2 & 4 & 2 \\ 1 & 2 & 3 \\ 0 & 1 & 1 \end{pmatrix}$ by elimination.
   Answer

   Row 2 − ½·row 1 gives $(0, 0, 2)$. The second pivot position is now 0, so exchange rows 2 and 3 (sign flips): rows $(2, 4, 2)$, $(0, 1, 1)$, $(0, 0, 2)$. Pivots $2, 1, 2$, one exchange: $\det = -(2 \cdot 1 \cdot 2) = -4$.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Give three conditions equivalent to "the square matrix $A$ is invertible".

   Show answer

2. What does $\det A$ mean geometrically, and what does its sign mean?

   Show answer

3. How do you get $\det A$ from elimination?

   Show answer

4. State the product rule for determinants, and one consequence.

   Show answer

5. How are $\det A$ and $\operatorname{tr} A$ related to the eigenvalues?

   Show answer

6. Write the angle between nonzero $u$ and $v$ with the dot product.

   Show answer

7. Give the projection of $b$ onto the line through $a$.

   Show answer

8. What equation defines $\hat x$ in the projection onto $C(A)$, and why?

   Show answer

9. What do $P^2 = P$ and $P^\top = P$ say about a projection matrix?

   Show answer

10. For square $A, B$ with $BA = I$, why is $AB = I$?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Why not compute $A^{-1}$ to solve $Ax = b$?" — Three times the flops of LU, and not backward stable: the residual can be up to about $\kappa(A)$ times worse.
• "What does $\det A = 0$ mean? Is a tiny $\det$ a problem?" — The columns span a flat box: $A$ is singular. A tiny $\det$ alone means little, because it depends on scale; look at $\kappa(A)$.
• "Derive the projection matrix." — The error is orthogonal to $C(A)$: $A^\top(b - A\hat x) = 0$; $A^\top A$ is invertible for independent columns.
• "How do you compute $\log\det$ of a covariance? In a normalizing flow?" — Cholesky, $2\sum_i \log L_{ii}$; flows use triangular Jacobians.

10. Practice #

The notebook for this lesson runs in Colab. It has five exercises, each with a check cell and a folded solution:

1. Compute the signed area of a polygon (shoelace formula) and check that a matrix multiplies it by $\det A$, sign included; then test the product rule on random matrices.
2. Write project_onto_line: return $\hat x$, $p$ and the rank-1 matrix $P = aa^\top / a^\top a$.
3. Write projector(A) with np.linalg.solve (no inverse), and check $P^2 = P$, $P^\top = P$ and $\operatorname{tr} P = \operatorname{rank} A$ on section 4(c).
4. Solve or invert? Measure the relative residuals of solve and inv(H) @ b on Hilbert matrices of growing size.
5. Enforce a conservation law $Cx = 0$ by projecting a prediction onto $N(C)$, and check that the result is the closest point that satisfies the law.

11. Watch and read #

The determinant3Blue1Brown · Essence of linear algebra, chapter 6 · 10 min

Dot products and duality3Blue1Brown · Essence of linear algebra, chapter 9 · 14 min

Projections onto SubspacesGilbert Strang · MIT 18.06, lecture 15 · 49 min

Properties of DeterminantsGilbert Strang · MIT 18.06, lecture 18 · 49 min

• Gilbert Strang, MIT 18.06 (2010), lecture 15: Projections onto subspaces, lecture 18: Properties of determinants (the three rules) and lecture 3: Multiplication and inverse matrices.
• 3Blue1Brown, The determinant and Dot products and duality, the text versions of chapters 6 and 9.
• Boyd and Vandenberghe, Introduction to Applied Linear Algebra (VMLS), free PDF: §3.1 and §3.4 (norm, angle), §11.1–11.3 (left and right inverses, inverse, solving linear equations).