Orthogonal and unitary matrices, change of basis, similarity

Before this lesson: lesson 01 (basis, coordinates) and lesson 03 (inverse, product rule for $\det$, projection). One theorem uses singular values; here "singular values of $A$" means the square roots of the eigenvalues of $A^\top A$ (lesson 08 builds the SVD). Time: about 60 minutes, plus the notebook.

---

Why this matters #

Much of linear algebra is the search for a good basis. In a basis of eigenvectors a matrix is diagonal (lesson 05). The SVD uses two orthonormal bases (lesson 08). The Fourier basis makes circular convolution diagonal. Orthogonal matrices are the safest changes of basis: they keep lengths, angles and the condition number, so stable algorithms are built from them (Householder QR, the QR algorithm). In ML they appear as orthogonal initialization and rotary position embeddings (RoPE). In ML for science they are the rotations under which physical laws do not change (equivariant networks).

1. Predict first #

2. The picture #

One vector, many names. An arrow in the plane does not change when you change the basis; only its coordinates change. The coordinates of $x$ in the basis $b_1, b_2$ are the weights $c_1, c_2$ in $x = c_1 b_1 + c_2 b_2$.

Which way does $M$ go? Put the new basis vectors, written in standard coordinates, as the columns of $M = [b_1 \; b_2]$. Then $Mc = c_1 b_1 + c_2 b_2$. So $M$ turns new coordinates into standard coordinates, and $M^{-1}$ goes the other way:

Interactive figure: loading…

The same map in another basis. A map $A$ is given in standard coordinates. To apply it to a vector given by new coordinates $c$, you need three steps: translate to standard coordinates ($Mc$), apply $A$, translate back ($M^{-1}$). So the map in new coordinates is $M^{-1}AM$. Section 3 goes through these steps one by one.

Interactive figure: loading…

Orthogonal matrices. A rotation or a reflection: the grid stays square, and lengths and angles do not change. In an orthonormal basis (a rotated, maybe mirrored, copy of the standard axes) the coordinates are plain dot products, $c_i = q_i^\top x$: no system has to be solved.

3. Definitions and main results #

Coordinates in a basis #

Let $b_1, \dots, b_n$ be a basis of $\mathbb{R}^n$ and $M = [b_1 \; \cdots \; b_n]$. $M$ is invertible (its $n$ columns are independent, lesson 03). The coordinates $c$ of $x$ in this basis satisfy

For two bases, $x = M_{\text{from}}\, c_{\text{from}} = M_{\text{to}}\, c_{\text{to}}$, so $c_{\text{to}} = M_{\text{to}}^{-1} M_{\text{from}}\, c_{\text{from}}$. The matrix $M_{\text{to}}^{-1} M_{\text{from}}$ is the change-of-basis matrix. In code, compute $c$ with a solve of $Mc = x$, not with an explicit inverse (lesson 13 explains why).

The same map in another basis #

The main difficulty of this lesson is the direction of $M$ and $M^{-1}$. Draw the two coordinate systems as the two rows of a square:

The top row lives in standard coordinates, where we know the map $A$. The bottom row lives in new coordinates, where we want the matrix $B$. There are two ways from $c$ (bottom left) to the bottom right corner:

1. Up: $M$ turns the new coordinates $c$ into the standard vector $x = Mc$.
2. Across: $A$ acts on $x$ and gives $Ax = AMc$.
3. Down: $M^{-1}$ turns $Ax$ back into new coordinates: $M^{-1}AMc$.

The direct way is $Bc$. Both ways must give the same coordinates for every $c$, so $B = M^{-1}AM$. Read the product from right to left: the matrix next to $c$ acts first. The same square read the other way gives $A = MBM^{-1}$.

Small numbers. Take $A = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix}$, $b_1 = (2, 1)$, $b_2 = (1, 1)$ and $c = (1, 1)$.

1. Up: $x = Mc = 1 \cdot (2, 1) + 1 \cdot (1, 1) = (3, 2)$.
2. Across: $Ax = (4 \cdot 3 - 2 \cdot 2,\ 3 + 2) = (8, 5)$.
3. Down: $M^{-1}(8, 5) = (8 - 5,\ -8 + 10) = (3, 2)$. Check: $3b_1 + 2b_2 = (6, 3) + (2, 2) = (8, 5)$ ✓.

Section 4(b) finds $B = \operatorname{diag}(3, 2)$ for this basis, and indeed $Bc = (3, 2)$.

Interactive figure: loading…

Theorem. Let $T(x) = Ax$ in standard coordinates. In the basis given by the columns of $M$, $T$ acts on coordinates by $B = M^{-1}AM$.

Proof.

1. The vector with new coordinates $c$ is $x = Mc$.
2. Its image, in standard coordinates, is $Ax = AMc$.
3. The new coordinates of the image are $M^{-1}(AMc)$.
4. So $T$ in new coordinates is $c \mapsto M^{-1}AMc$, that is, $B = M^{-1}AM$. $\blacksquare$

Column $j$ of $B$ is $M^{-1}Ab_j$: the new coordinates of $T(b_j)$. This is lesson 01's "columns are the images of the basis vectors", now in the new basis.

Similar matrices #

Two square matrices $A$ and $B$ are similar if $B = M^{-1}AM$ for some invertible $M$: they are the same map, written in two bases.

Theorem (similarity invariants). Similar matrices have the same characteristic polynomial. Hence they have the same eigenvalues with the same algebraic multiplicities, the same trace and the same determinant. They also have the same rank.

Proof.

1. $B - \lambda I = M^{-1}AM - \lambda M^{-1}M$, and this is $M^{-1}(A - \lambda I)M$.
2. Product rule (lesson 03): $\det(B - \lambda I) = \det(M^{-1})\det(A - \lambda I)\det(M)$. Since $\det(M^{-1})\det(M) = 1$, this is $\det(A - \lambda I)$.
3. The eigenvalues are the roots of this polynomial; the trace and the determinant are two of its coefficients (lesson 03). So they agree.
4. Rank: $\operatorname{rank}(XY) \le \min(\operatorname{rank} X, \operatorname{rank} Y)$ (lesson 01) gives $\operatorname{rank} B \le \operatorname{rank} A$, and $A = MBM^{-1}$ gives $\operatorname{rank} A \le \operatorname{rank} B$. $\blacksquare$

What does change. Eigenvectors: if $Av = \lambda v$, then $B(M^{-1}v) = \lambda (M^{-1}v)$. It is the same arrow under a new name: its new coordinates $M^{-1}v$. Singular values, norms and symmetry also change in general (section 4(b) shows it with numbers).

The converse is false: the same eigenvalues do not make two matrices similar. The shear $S = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$ and the identity $I$ both have the eigenvalues $1, 1$, trace 2 and determinant 1. But for every invertible $M$,

So no basis turns $I$ into $S$. In words: $I$ keeps every vector, and this does not depend on the basis; the shear moves most vectors. The same argument shows that $N = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}$ is not similar to the zero matrix. The Jordan form of lesson 05 gives the full answer to the question "when are two matrices similar?".

The aim of the next lessons is to choose $M$ so that $B$ is simple. In a basis of eigenvectors, $B$ is diagonal (lesson 05).

Orthogonal matrices #

A real $n \times n$ matrix $Q$ is orthogonal if $Q^\top Q = I$. Entry $(i, j)$ of $Q^\top Q$ is $q_i^\top q_j$, so this says: the columns are orthonormal (perpendicular, with length 1).

• $Q^{-1} = Q^\top$, and also $QQ^\top = I$ (for square matrices one side is enough, lesson 03). So the rows are orthonormal as well.
• Coordinates in the orthonormal basis $q_1, \dots, q_n$ are $c = Q^\top x$, that is, $c_i = q_i^\top x$.
• Products and inverses of orthogonal matrices are orthogonal: $(Q_1Q_2)^\top(Q_1Q_2) = Q_2^\top Q_1^\top Q_1 Q_2 = I$.
• A tall $Q$ ($m \times n$, $m > n$) with orthonormal columns has $Q^\top Q = I_n$ but $QQ^\top \ne I_m$: $QQ^\top$ is the projector onto $C(Q)$ (lesson 03). Such a $Q$ is not called orthogonal.

Theorem (orthogonal maps keep the geometry). If $Q^\top Q = I$, then for all $x, y$: $(Qx)^\top(Qy) = x^\top y$, $\|Qx\| = \|x\|$, and the angle between $Qx$ and $Qy$ equals the angle between $x$ and $y$.

Proof.

1. $(Qx)^\top(Qy) = x^\top Q^\top Q y = x^\top y$.
2. Take $y = x$: $\|Qx\|^2 = \|x\|^2$.
3. $\cos\theta = x^\top y / (\|x\|\|y\|)$ is built from dot products only, so it does not change. $\blacksquare$

The proof uses only $Q^\top Q = I$, so it also holds for a tall $Q$ with orthonormal columns.

Keeping lengths is enough

The converse also holds: if $\|Qx\| = \|x\|$ for all $x$, then $Q^\top Q = I$. The reason is that lengths fix dot products, through the polarization identity

$$x^\top y = \tfrac14\big(\|x + y\|^2 - \|x - y\|^2\big).$$

Apply it to $Qx, Qy$ and to $x, y$: the right-hand sides are equal, so $(Qx)^\top(Qy) = x^\top y$ for all $x, y$. That is, $x^\top(Q^\top Q - I)y = 0$ for all $x, y$. With $x = e_i$ and $y = e_j$ this says that entry $(i, j)$ of $Q^\top Q - I$ is 0. So a linear map that keeps every length is orthogonal, and it keeps every angle as well.

Theorem (determinant and eigenvalues). If $Q$ is orthogonal, then $\det Q = \pm 1$, and every eigenvalue $\lambda$ (real or complex) has $|\lambda| = 1$.

Proof.

1. $1 = \det I = \det(Q^\top Q)$, and by the product rule this is $\det(Q^\top)\det(Q) = (\det Q)^2$.
2. Let $Qx = \lambda x$ with $x \ne 0$, possibly complex. With the complex norm $\|x\|^2 = x^* x$ ($x^*$ is the conjugate transpose), $\|Qx\|^2 = x^* Q^\top Q x = \|x\|^2$ because $Q$ is real. Also $\|Qx\| = |\lambda|\,\|x\|$. So $|\lambda| = 1$. $\blacksquare$

Rotations and reflections. $\det Q = +1$: a rotation (orientation kept). $\det Q = -1$: the map contains a reflection. In 2D every orthogonal matrix is one of

Why only these two? The first column is a unit vector, so it is $(\cos\theta, \sin\theta)$ for some $\theta$. The second column is a unit vector orthogonal to it, so it is $\pm(-\sin\theta, \cos\theta)$. The sign $+$ gives $R_\theta$ and the sign $-$ gives $F_\theta$. $F_\theta$ reflects across the line at angle $\theta/2$. Other examples: permutation matrices, and Householder reflections $H = I - 2uu^\top$ with $\|u\| = 1$ (lesson 12).

Interactive figure: loading…

Unitary matrices #

For complex vectors the inner product is $\langle x, y \rangle = x^* y = \sum_i \bar x_i y_i$, and $\|x\|^2 = x^* x = \sum_i |x_i|^2$. Here $x^*$ is the conjugate transpose: transpose, and replace every entry by its complex conjugate ($\overline{a + bi} = a - bi$). The conjugate is needed so that $\|x\|^2$ is a positive real number: for $x = (1, i)$, $x^* x = 1 + (-i)(i) = 2$, while $x^\top x = 1 + i^2 = 0$.

A complex $n \times n$ matrix $U$ is unitary if $U^*U = I$. The same proofs as for orthogonal matrices give $\langle Ux, Uy \rangle = x^* U^* U y = \langle x, y \rangle$, $\|Ux\| = \|x\|$, $U^{-1} = U^*$, $|\det U| = 1$ and $|\lambda| = 1$ for every eigenvalue. A real orthogonal matrix is a unitary matrix with real entries.

Example, entry by entry. Take

($U$ is symmetric, so $U^*$ only conjugates the entries.) The product $U^*U$ is $\tfrac12$ times:

1. entry $(1, 1)$: $1 \cdot 1 + (-i)(i) = 1 + 1 = 2$;
2. entry $(1, 2)$: $1 \cdot i + (-i) \cdot 1 = 0$;
3. entry $(2, 1)$: $(-i) \cdot 1 + 1 \cdot i = 0$;
4. entry $(2, 2)$: $(-i)(i) + 1 \cdot 1 = 2$.

So $U^*U = \tfrac12 \begin{pmatrix} 2 & 0 \\ 0 & 2 \end{pmatrix} = I$: $U$ is unitary. Without the conjugate the test fails: $U^\top U = U^2 = \begin{pmatrix} 0 & i \\ i & 0 \end{pmatrix} \ne I$. Also $\det U = \tfrac12(1 - i^2) = 1$, and the eigenvalues are $(1 \pm i)/\sqrt2$, both of modulus 1.

The DFT matrix is unitary (Parseval)

The most important unitary matrix is the discrete Fourier transform. The DFT matrix $F_{jk} = e^{-2\pi i jk/n}$ ($j, k = 0, \dots, n-1$) satisfies $F^*F = nI$, so $F/\sqrt{n}$ is unitary. Entry $(j, k)$ of $F^*F$ is $\sum_{l} e^{2\pi i l (j - k)/n}$: a sum of $n$ equally spaced points on the unit circle when $j \ne k$ (it is 0), and $n$ when $j = k$. This is Parseval's theorem: the normalized Fourier transform keeps the energy $\sum_j |x_j|^2$. Lesson 19 uses it for circulant matrices and Fourier neural operators.

Orthogonal maps keep the singular values #

Theorem. Let $Q_1$ ($m \times m$) and $Q_2$ ($n \times n$) be orthogonal, and $A$ be $m \times n$. Then $Q_1AQ_2$ has the same singular values as $A$. Hence $\|Q_1AQ_2\|_2 = \|A\|_2$ and $\|Q_1AQ_2\|_F = \|A\|_F$.

Proof.

1. $(Q_1AQ_2)^\top(Q_1AQ_2) = Q_2^\top A^\top Q_1^\top Q_1 A Q_2$, and $Q_1^\top Q_1 = I$ turns this into $Q_2^\top (A^\top A) Q_2$.
2. $Q_2^\top = Q_2^{-1}$, so this matrix is $Q_2^{-1}(A^\top A)Q_2$: it is similar to $A^\top A$.
3. Similar matrices have the same eigenvalues. Their square roots are the singular values, so the singular values agree.
4. The two norms depend only on the singular values: $\|A\|_2 = \sigma_1$ (lesson 08) and $\|A\|_F^2 = \sum_i \sigma_i^2 = \operatorname{tr}(A^\top A)$. $\blacksquare$

Three consequences:

• Conditioning. All singular values of $Q$ are 1 ($Q^\top Q = I$), so $\kappa_2(Q) = 1$. A rounding error $\delta$ is not amplified: $\|Q\delta\| = \|\delta\|$. Algorithms built only from orthogonal transformations (Householder and Givens QR, the QR algorithm, Golub–Kahan) are backward stable.
• Symmetry. An orthogonal change of basis $B = Q^\top AQ$ keeps a symmetric $A$ symmetric: $B^\top = Q^\top A^\top Q = B$. A general $M^{-1}AM$ does not. This is why the spectral theorem (lesson 06) is written $S = Q\Lambda Q^\top$.
• Orthogonal similarity keeps both spectra. A general $M^{-1}AM$ keeps the eigenvalues but changes the singular values; $Q^\top AQ$ keeps both.

Answers to section 1. (1) $Mc = 1 \cdot (1, 0) + 1 \cdot (1, 1) = (2, 1)$. The vector $(1, 1)$ is $b_2$ itself, so its new coordinates are $c = M^{-1}(1, 1) = (0, 1)$. (2) Shared: eigenvalues, trace, determinant and rank (all from the characteristic polynomial and the rank argument). Not shared in general: eigenvectors ($v$ becomes $M^{-1}v$) and singular values (unless $M$ is orthogonal).

4. Worked example #

(a) Coordinates. Basis $b_1 = (2, 1)$, $b_2 = (1, 1)$:

For $x = (4, 3)$: $c = M^{-1}x = (4 - 3, -4 + 6) = (1, 2)$. Check: $1 \cdot (2, 1) + 2 \cdot (1, 1) = (4, 3)$ ✓.

(b) One map, three matrices. Take $A = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix}$.

• In the basis of (a), use the column rule (column $j$ of $B$ = new coordinates of $Ab_j$): $Ab_1 = (8 - 2, 2 + 1) = (6, 3) = 3b_1$, so column 1 of $B$ is $(3, 0)$. $Ab_2 = (4 - 2, 1 + 1) = (2, 2) = 2b_2$, so column 2 of $B$ is $(0, 2)$. Hence $B = M^{-1}AM = \begin{pmatrix} 3 & 0 \\ 0 & 2 \end{pmatrix}$. The basis vectors are eigenvectors, and in this basis the map is two independent stretches.
• In the basis $b_1' = (1, 0)$, $b_2' = (1, 1)$, with $M_2 = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$ and $M_2^{-1} = \begin{pmatrix} 1 & -1 \\ 0 & 1 \end{pmatrix}$, multiply from right to left:

	$A$	$B = \operatorname{diag}(3, 2)$	$B_2$
trace	5	5	5
determinant	6	6	6
eigenvalues	3, 2	3, 2	3, 2
singular values	4.497, 1.334	3, 2	3.257, 1.842

For $A$: $A^\top A = \begin{pmatrix} 17 & -7 \\ -7 & 5 \end{pmatrix}$ has eigenvalues $11 \pm \sqrt{85}$, so $\sigma = \sqrt{11 \pm \sqrt{85}} \approx 4.497, 1.334$. The singular values change from basis to basis, because $M$ and $M_2$ are not orthogonal. Their product stays $|\det| = 6$. The eigenvector $(2, 1)$ of $A$ has new coordinates $M^{-1}(2, 1) = (1, 0) = e_1$: the eigenvector of $B$.

(c) A rotation and a reflection.

Both have orthonormal columns ($9 + 16 = 25$, and $3 \cdot (-4) + 4 \cdot 3 = 0$). $\det Q = 1$: $Q$ is the rotation $R_\theta$ with $\cos\theta = 3/5$, $\sin\theta = 4/5$ ($\theta \approx 53.13°$). For $x = (1, 2)$ and $y = (3, 1)$: $Qx = (-1, 2)$ and $Qy = (1, 3)$. Lengths: $\|x\|^2 = \|Qx\|^2 = 5$. Dot products: $x^\top y = 3 + 2 = 5$ and $(Qx)^\top(Qy) = -1 + 6 = 5$ ✓. The eigenvalues of $Q$ are $(3 \pm 4i)/5$, both of modulus 1.

$\det H = -\tfrac{9}{25} - \tfrac{16}{25} = -1$: a reflection. $H(2, 1) = (2, 1)$ and $H(1, -2) = (-1, 2)$, so $H$ fixes the line through $(2, 1)$ and flips the perpendicular direction. Its eigenvalues are $1$ and $-1$.

5. In code #

NumPy and SciPy check the claims of sections 3 and 4. The notebook for this lesson has these cells and the exercises.

import numpy as np
from scipy.linalg import polar

# the three-step path c -> Mc -> AMc -> M^{-1}AMc equals Bc
A = np.array([[4., -2.], [1., 1.]])
M, B = np.array([[2., 1.], [1., 1.]]), np.diag([3., 2.])
c = np.array([1., 1.])
x = M @ c
print(x, A @ x, np.linalg.solve(M, A @ x), B @ c)   # [3. 2.] [8. 5.] [3. 2.] [3. 2.]

# (b) one map, three matrices: check AM = MB instead of recomputing B
M2, B2 = np.array([[1., 1.], [0., 1.]]), np.array([[3., 0.], [1., 2.]])
print(np.allclose(A @ M, M @ B), np.allclose(A @ M2, M2 @ B2))   # True True
for X in (A, B, B2):
    print(round(np.trace(X), 12), round(np.linalg.det(X), 12),
          np.sort(np.linalg.eigvals(X).real), np.round(np.linalg.svd(X, compute_uv=False), 3))
# 5.0 6.0 [2. 3.] [4.497 1.334]
# 5.0 6.0 [2. 3.] [3. 2.]
# 5.0 6.0 [2. 3.] [3.257 1.842]

# (c) a rotation keeps dot products; its eigenvalues have modulus 1
Q = np.array([[3., -4.], [4., 3.]]) / 5
x, y = np.array([1., 2.]), np.array([3., 1.])
print(np.round(Q @ x, 12), np.round(Q @ y, 12), x @ y, round((Q @ x) @ (Q @ y), 12))
# [-1.  2.] [1. 3.] 5.0 5.0
print(np.abs(np.linalg.eigvals(Q)))                  # [1. 1.]

# a unitary matrix: U* U = I, but U^T U is not I
U = np.array([[1, 1j], [1j, 1]]) / np.sqrt(2)
print(np.allclose(U.conj().T @ U, np.eye(2)), np.allclose(U.T @ U, np.eye(2)))   # True False

# orthogonal factors on both sides keep the singular values
rng = np.random.default_rng(0)
Q1, _ = np.linalg.qr(rng.standard_normal((4, 4)))
Q2, _ = np.linalg.qr(rng.standard_normal((3, 3)))
C = rng.standard_normal((4, 3))
print(np.allclose(np.linalg.svd(Q1 @ C @ Q2, compute_uv=False),
                  np.linalg.svd(C, compute_uv=False)))   # True

# the polar factor of a matrix with det < 0 is a reflection, not a rotation
U_polar, P_polar = polar(np.array([[1., 2.], [3., 4.]]))   # det = -2
print(round(np.linalg.det(U_polar), 12))                   # -1.0

# the normalized DFT is unitary (Parseval)
n = 8
F = np.fft.fft(np.eye(n), norm="ortho")    # F[j, k] = exp(-2πi jk/n) / sqrt(n)
print(np.allclose(F.conj().T @ F, np.eye(n)))                          # True
z = rng.standard_normal(n)
print(np.isclose(np.linalg.norm(np.fft.fft(z, norm="ortho")), np.linalg.norm(z)))   # True

np.linalg.solve(M, v) computes $M^{-1}v$ without forming $M^{-1}$. For complex arrays, U.T does not conjugate; use U.conj().T.

6. Where it shows up #

• PCA of an ensemble forecast is an orthogonal change of basis onto the principal axes. The total variance does not change, $\operatorname{tr}(Q^\top \Sigma Q) = \operatorname{tr}\Sigma$; it is only redistributed so that the first axes carry most of it.
• Rotary position embeddings (RoPE). A query at position $m$ is rotated by a block-diagonal orthogonal matrix $R_m$ (2×2 rotations by the angles $m\theta_i$), and a key at position $n$ by $R_n$. Since $R_m^\top R_n = R_{n-m}$, the score $(R_m q)^\top (R_n k) = q^\top R_{n-m} k$ depends only on the offset $n - m$, and the rotations do not change the norms of $q$ and $k$.
• Norm-preserving layers. An orthogonal weight matrix keeps $\|Wx\| = \|x\|$, with all singular values 1, so the linear part neither blows up nor shrinks signals and gradients. This motivates orthogonal initialization (Saxe et al., 2014) and unitary RNNs. The Muon optimizer (2024) replaces the momentum update $G = U\Sigma V^\top$ of each weight matrix by an approximation of $UV^\top$ (its polar factor: the same singular vectors, all singular values set to 1).
• Equivariant networks for science. If a molecule or a wind field is rotated by $Q$, a vector output (forces, velocities) should rotate too, $f(Qx) = Qf(x)$, and a scalar output (energy) should not change. Distances $\|x_i - x_j\|$ and dot products are invariant under $Q$, which is why E(n)-equivariant GNNs build their messages from them (lesson 19).

7. Common mistakes #

• Direction of the conversion. The coordinates are $c = M^{-1}x$, not $Mx$; $M$ goes from new coordinates to standard ones.
• Reading $M^{-1}AM$ from left to right. The matrix next to the vector acts first: $M$, then $A$, then $M^{-1}$.
• "Similar matrices have the same eigenvectors, singular values or norms." False in general. They share the characteristic polynomial and the rank (and the Jordan form, lesson 05).
• "Same eigenvalues ⇒ similar." False: the shear $\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$ and $I$.
• "Orthogonal = rotation." $\det Q = -1$ means a reflection is included (also for $U$, $V$ and $UV^\top$).
• Orthogonal but not normalized columns. Then $Q^\top Q$ is diagonal, not $I$, and $Q^{-1} \ne Q^\top$.
• Tall $Q$: $QQ^\top = I$. False; $QQ^\top$ is the projector onto $C(Q)$.
• Complex case. Use $U^*$ (conjugate transpose), not $U^\top$; NumPy's A.T does not conjugate.

8. Check yourself #

1. $B = M^{-1}AM$, where $A$ is $3 \times 3$ with eigenvalues $1, 2, 4$. Give $\operatorname{tr} B$ and $\det B$. Is $B$ invertible?
   Answer

   $\operatorname{tr} B = 7$, $\det B = 8$. Yes: 0 is not an eigenvalue (equivalently $\det B \ne 0$).
2. $q_1 = (3/5, 4/5)$ and $q_2 = (-4/5, 3/5)$. Find the coordinates of $x = (5, 0)$ in this basis without solving a linear system.
   Answer

   The basis is orthonormal, so $c_i = q_i^\top x$: $c = (3, -4)$. Check: $3(3/5, 4/5) - 4(-4/5, 3/5) = (5, 0)$ ✓.
3. Show that the product of two orthogonal matrices is orthogonal. Is the sum orthogonal?
   Answer

   $(Q_1Q_2)^\top(Q_1Q_2) = Q_2^\top Q_1^\top Q_1 Q_2 = Q_2^\top Q_2 = I$. The sum is not, in general: $I + I = 2I$ and $(2I)^\top(2I) = 4I$.
4. A $5 \times 3$ matrix $Q$ has orthonormal columns. What are $Q^\top Q$ and $QQ^\top$? Give the rank and the trace of $QQ^\top$.
   Answer

   $Q^\top Q = I_3$. $QQ^\top$ is the $5 \times 5$ orthogonal projector onto $C(Q)$, not $I_5$; its rank and its trace are both 3.
5. Does $M^{-1}AM$ keep symmetry? Try $A = \operatorname{diag}(1, 2)$ and $M = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$.
   Answer

   $M^{-1}AM = \begin{pmatrix} 1 & -1 \\ 0 & 2 \end{pmatrix}$: not symmetric. With an orthogonal $Q$, $(Q^\top AQ)^\top = Q^\top A^\top Q = Q^\top AQ$, so symmetry is kept.
6. Why does every $3 \times 3$ rotation (orthogonal, $\det = +1$) have the eigenvalue 1?
   Answer

   The characteristic polynomial is a real cubic, so it has a real root, and every eigenvalue has $|\lambda| = 1$. If the other two eigenvalues are a complex pair $\mu, \bar\mu$, then $\det = \lambda \cdot |\mu|^2 = \lambda = 1$. If all three are real ($\pm 1$), their product is 1, so the number of $-1$'s is even and at least one eigenvalue is $+1$. Its eigenvector is the rotation axis.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. The columns of $M$ are the new basis vectors. How do you get the new coordinates $c$ of a vector $x$?

   Show answer

2. Explain $B = M^{-1}AM$ in words.

   Show answer

3. What do similar matrices always share?

   Show answer

4. What does a general similarity $M^{-1}AM$ not keep?

   Show answer

5. Give two $2 \times 2$ matrices with the same eigenvalues that are not similar.

   Show answer

6. Why does $Q^\top Q = I$ keep dot products?

   Show answer

7. What values can $\det Q$ take for an orthogonal $Q$, and what do they mean?

   Show answer

8. What is $|\lambda|$ for an eigenvalue of an orthogonal matrix, and why?

   Show answer

9. Define a unitary matrix.

   Show answer

10. What are the singular values of $Q_1AQ_2$ when $Q_1$, $Q_2$ are orthogonal?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "What is the inverse of an orthogonal matrix, and why are orthogonal transformations numerically safe?" — $Q^{-1} = Q^\top$; $\kappa_2(Q) = 1$ and $\|Q\delta\| = \|\delta\|$, so errors are not amplified.
• "Two matrices have the same eigenvalues. Are they similar?" — Not always (a Jordan block versus a diagonal matrix). Yes if both are diagonalizable: both are similar to the same $\Lambda$.
• "How does RoPE encode relative position?" — $R_m^\top R_n = R_{n-m}$, so the score depends only on $n - m$.

10. Practice #

The notebook for this lesson runs in Colab. It has five exercises, each with a check cell and a folded solution:

1. Convert coordinates in both directions (to_basis_coords, from_basis_coords) with a solve, not an inverse, and check section 4(a).
2. Write matrix_in_basis(A, M) $= M^{-1}AM$, and check which quantities of section 4(b) stay the same.
3. Write has_orthonormal_columns(Q) with a tolerance that scales with the size of $Q$.
4. Build $R_\theta$ and $F_\theta$, and check their determinants, their eigenvalues and the mirror line of $F_\theta$.
5. Implement RoPE on a vector of even length, and check that the score depends only on the offset $n - m$ and that the norms do not change.

11. Watch and read #

Change of basis3Blue1Brown · Essence of linear algebra, chapter 13 · 13 min

Watch how a vector gets two names, and the "translate, apply, translate back" reading of $M^{-1}AM$.

Orthogonal Matrices and Gram-SchmidtGilbert Strang · MIT 18.06, lecture 17 · 49 min

Change of Basis; Image CompressionGilbert Strang · MIT 18.06, lecture 31 · 50 min

• 3Blue1Brown, Change of basis, the text version of chapter 13.
• Gilbert Strang, MIT 18.06 (2010), lecture 31: Change of basis; image compression and lecture 17: Orthogonal matrices and Gram–Schmidt.
• Sheldon Axler, Linear Algebra Done Right, 4th edition, free PDF: §3D (3.84 is the change-of-basis formula) and §7D (isometries, unitary operators and matrix factorization).
• Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding, §3.2 (equations 14–16 show $R_m^\top R_n = R_{n-m}$).