Linear Algebra · 196

Glossary

The technical terms of this track, with short definitions (and the Persian term where available). Each term links to the lesson where it is introduced.

$A$ -conjugate: Vectors with $p_i^\top A p_j = 0$ for $i \ne j$ : orthogonal in the inner product defined by an SPD matrix $A$ . Introduced in: Lesson 14
affine map نگاشت آفین: A linear map plus a fixed shift, $x \mapsto Wx + b$ . It is not linear when $b \ne 0$ . Introduced in: Lesson 1
algebraic multiplicity: The number of times $\lambda$ is a root of the characteristic polynomial. Introduced in: Lesson 5
Arnoldi iteration: Builds an orthonormal basis $Q_k$ of the Krylov subspace with Gram–Schmidt, and the upper Hessenberg matrix $H_k = Q_k^\top AQ_k$ . Introduced in: Lesson 14
backward error: The smallest relative change of the data that makes the computed answer exact. It can be computed from the residual. Introduced in: Lesson 13
backward stable: An algorithm whose backward error is $O(u)$ for every input. Its forward error is then at most about $\kappa \cdot u$ . Introduced in: Lesson 13
basis پایه: An independent list of vectors that spans a subspace. Every vector of the subspace has unique coordinates in it. Introduced in: Lesson 1
Bauer–Fike theorem: For diagonalizable $A = X\Lambda X^{-1}$ , every eigenvalue of $A + E$ is within $\kappa_2(X)\|E\|_2$ of an eigenvalue of $A$ . Introduced in: Lesson 17
BBP phase transition: In the spiked covariance model, the top sample eigenvalue separates from the noise bulk only when the spike strength $\theta$ is above $\sqrt\gamma$ ; below it the spike cannot be detected. Introduced in: Lesson 18
Bessel's correction: Dividing by $N - 1$ instead of $N$ in the sample covariance. It makes the estimate unbiased, because the mean was fitted to the same data. Introduced in: Lesson 9
catastrophic cancellation: The loss of correct digits when two nearly equal rounded numbers are subtracted. The fix is to rewrite the formula. Introduced in: Lesson 13
Cauchy interlacing: If $B = V^\top S V$ with $V \in \mathbb{R}^{n \times m}$ with orthonormal columns (for example a principal submatrix), then $\lambda_k(S) \ge \lambda_k(B) \ge \lambda_{k+n-m}(S)$ . Introduced in: Lesson 6
Cauchy–Schwarz inequality: $|u^\top v| \le \|u\|\,\|v\|$ , with equality only when one vector is a multiple of the other. It makes the angle between two vectors well defined. Introduced in: Lesson 3
centering: Subtracting the mean of each feature from the data: $X_c = X - \mathbf{1}\mu^\top$ . PCA needs centered data. Introduced in: Lesson 9
change-of-basis matrix: The matrix that turns coordinates in one basis into coordinates in another. If the new basis vectors are the columns of $M$ , then $x = Mc$ and $c = M^{-1}x$ . Introduced in: Lesson 4
characteristic polynomial: $p(\lambda) = \det(A - \lambda I)$ , a polynomial of degree $n$ for an $n \times n$ matrix. Its roots are the eigenvalues. Introduced in: Lesson 5
Cholesky factorization: $S = LL^\top$ with $L$ lower triangular with a positive diagonal. It exists, and is unique, exactly when $S$ is positive definite. Introduced in: Lesson 7
circulant matrix: A matrix $C$ with $C_{jl} = c_{j-l}$ (indices mod $n$ ): each column is the previous one shifted down by one. $Cx$ is the circular convolution $c \circledast x$ . Introduced in: Lesson 19
circular convolution: $(c \circledast x)_j = \sum_l c_{j-l}\,x_l$ with indices mod $n$ . The DFT turns it into an entrywise product: $c \circledast x = F^{-1}(\hat c \odot Fx)$ . Introduced in: Lesson 19
column space فضای ستونی: $C(A)$ , the span of the columns of $A$ . It is the set of all outputs $Ax$ . Introduced in: Lesson 1
composition ترکیب نگاشت‌ها: Applying one map after another. $AB$ is the matrix of "first $B$ , then $A$ ": $(AB)x = A(Bx)$ . Introduced in: Lesson 1
condition number: The worst amplification of a small relative input change. For an invertible matrix, $\kappa(A) = \|A\|\,\|A^{-1}\|$ , and $\kappa_2(A) = \sigma_{\max}/\sigma_{\min}$ . Introduced in: Lesson 13
conjugate gradient (CG): A Krylov method for $Ax = b$ with $A$ symmetric positive definite. Step $k$ minimizes the $A$ -norm of the error over $x_0 + \mathcal{K}_k(A, r_0)$ ; convergence depends on $\sqrt{\kappa}$ . Introduced in: Lesson 14
conjugate transpose: $A^*$ : the transpose of $A$ with every entry replaced by its complex conjugate. For a complex vector, $\|x\|^2 = x^* x$ . Introduced in: Lesson 4
coordinates مختصات: The numbers $c_j$ in $x = \sum_j c_j v_j$ for a basis $v_1, \dots, v_k$ . Introduced in: Lesson 1
Courant–Fischer theorem: The min–max form of the eigenvalues of a symmetric matrix, sorted downward: $\lambda_k = \max_{\dim U = k}\, \min_{x \in U,\, x \ne 0} R(x)$ . Introduced in: Lesson 6
CP decomposition: A tensor written as a sum of $R$ rank-1 tensors, $\mathcal{X} \approx \sum_r a_r \circ b_r \circ c_r$ . The smallest exact $R$ is the CP rank. Introduced in: Lesson 15
Davis–Kahan theorem: Bounds the rotation of an eigenvector or eigenspace of a symmetric matrix under noise $E$ : $\sin\theta \lesssim \|E\|_2 / \text{gap}$ , for example $\sin\theta(\hat q_k, q_k) \le 2\|E\|_2/\delta_k$ . Introduced in: Lesson 17
defective matrix: A square matrix with fewer independent eigenvectors than its size, such as the shear $\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$ . It is not diagonalizable. Introduced in: Lesson 5
deflation: Setting a tiny subdiagonal entry to 0 in the QR algorithm, which splits off a converged eigenvalue so the work continues on a smaller block. Introduced in: Lesson 14
determinant: The factor $\det A$ by which a square matrix changes volumes, with a minus sign when the map mirrors space. $\det A = 0$ exactly when $A$ is not invertible. Introduced in: Lesson 3
diagonalization: Writing $A = X\Lambda X^{-1}$ with eigenvectors as the columns of $X$ and eigenvalues on the diagonal of $\Lambda$ . It is possible exactly when $A$ has $n$ independent eigenvectors. Introduced in: Lesson 5
differential: The linear part $df$ of the change $f(X + dX) - f(X)$ . If $df = \operatorname{tr}(G^\top dX)$ for every $dX$ , then $\nabla_X f = G$ . Introduced in: Lesson 16
dimension بُعد: The number of vectors in a basis of a subspace. All bases of a subspace have the same size. Introduced in: Lesson 1
dot product: $u^\top v = u_1 v_1 + \dots + u_n v_n = \|u\|\,\|v\|\cos\theta$ . Its sign tells whether the angle between $u$ and $v$ is below, at or above $90°$ . Introduced in: Lesson 0
dual norm: $\|y\|_* = \max_{\|x\| \le 1} x^\top y$ . The dual of $\ell_p$ is $\ell_q$ with $1/p + 1/q = 1$ . Introduced in: Lesson 11
echelon form: The staircase form that elimination produces. The first nonzero entry (the pivot) of each nonzero row lies to the right of the pivot in the row above. Introduced in: Lesson 2
Eckart–Young–Mirsky theorem: The truncated SVD $A_k$ is a best approximation of $A$ of rank at most $k$ in the spectral and the Frobenius norm. The errors are $\sigma_{k+1}$ and $\sqrt{\sigma_{k+1}^2 + \dots + \sigma_r^2}$ . Introduced in: Lesson 8
eigengap: The distance from an eigenvalue (or a group of eigenvalues) to the rest of the spectrum. A small gap makes the eigenvector sensitive to noise. Introduced in: Lesson 17
eigenspace: $E_\lambda = N(A - \lambda I)$ : all eigenvectors for the eigenvalue $\lambda$ , together with $0$ . Introduced in: Lesson 5
eigenvalue: A number $\lambda$ with $Av = \lambda v$ for some nonzero vector $v$ . The eigenvalues are the roots of $\det(A - \lambda I)$ . Introduced in: Lesson 5
eigenvector: A nonzero vector $v$ with $Av = \lambda v$ . The matrix only stretches, shrinks or flips it; its line does not change. Introduced in: Lesson 5
elimination حذف: Subtracting multiples of rows from other rows to create zeros below each pivot (Gaussian elimination). Introduced in: Lesson 1
entry درایه: One number of a matrix; $A_{ij}$ is the entry in row $i$ and column $j$ . Introduced in: Lesson 1
equivariant map: A map $f$ with $f(\rho(g)x) = \rho'(g)f(x)$ for every group element $g$ : transforming the input transforms the output in a matching way. It is invariant if $\rho'(g) = I$ . Introduced in: Lesson 19
Euclidean norm: The length of a vector, $\|v\| = \|v\|_2 = \sqrt{v^\top v} = \sqrt{v_1^2 + \dots + v_n^2}$ . Introduced in: Lesson 0
explained variance ratio: The share of the total variance along principal direction $i$ : $\lambda_i / \sum_j \lambda_j = \sigma_i^2 / \|X_c\|_F^2$ . Introduced in: Lesson 9
Fiedler vector: The eigenvector of the second smallest Laplacian eigenvalue $\lambda_2$ . Its signs split the graph in two (spectral bisection). Introduced in: Lesson 19
forward error: $\|\hat x - x\| / \|x\|$ : how wrong the computed answer is. It is usually unknown, because $x$ is unknown. Introduced in: Lesson 13
forward mode: Automatic differentiation that multiplies the chain of Jacobians from the input side (JVPs). One pass per input direction. Introduced in: Lesson 15
four fundamental subspaces: The column space $C(A)$ , null space $N(A)$ , row space $C(A^\top)$ and left null space $N(A^\top)$ of a matrix $A$ . Introduced in: Lesson 2
Fourier neural operator (FNO): A network whose layers apply a learned filter to the lowest Fourier modes plus a pointwise linear map: a global convolution that can be used on grids of different size. Introduced in: Lesson 19
free variable: A variable of $Ax = 0$ whose column has no pivot. You choose it freely; back substitution then gives the pivot variables. Introduced in: Lesson 2
frequency response: The eigenvalues $\hat c = Fc$ of a circulant (the DFT of its first column). They tell how the filter scales each Fourier mode. Introduced in: Lesson 19
Frobenius norm: $\|A\|_F = \sqrt{\sum_{ij} a_{ij}^2} = \sqrt{\sum_i \sigma_i^2}$ : the Euclidean norm of $A$ seen as a long vector. Introduced in: Lesson 11
Gavish–Donoho threshold: A cut-off for singular values of a noisy matrix that minimizes the mean squared error; for a square matrix it is $(4/\sqrt3)\sqrt n\,\sigma$ , a little above the bulk edge $2\sqrt n\,\sigma$ . Introduced in: Lesson 18
generalized eigenvector: A nonzero vector $v$ with $(A - \lambda I)^k v = 0$ for some $k \ge 1$ . A defective matrix needs them to complete a basis. Introduced in: Lesson 5
generalized Rayleigh quotient: $x^\top S x / x^\top B x$ with $S$ symmetric and $B$ symmetric positive definite. Its critical values are the eigenvalues of $Sx = \lambda Bx$ . Introduced in: Lesson 6
geometric multiplicity: $\dim N(A - \lambda I)$ , the number of independent eigenvectors for $\lambda$ . It is at least 1 and at most the algebraic multiplicity. Introduced in: Lesson 5
Gershgorin disc theorem: Every eigenvalue of $A$ lies in at least one disc with center $a_{ii}$ and radius $\sum_{j \ne i} |a_{ij}|$ . Introduced in: Lesson 5
Givens rotation: A rotation that changes only two coordinates. It is chosen to set one entry to zero. Introduced in: Lesson 12
GMRES: A Krylov method for any nonsingular $A$ : it picks $x_k \in x_0 + \mathcal{K}_k(A, r_0)$ with the smallest residual $\|b - Ax_k\|_2$ . Introduced in: Lesson 14
Golub–Kahan bidiagonalization: $A = UBV^\top$ with $B$ bidiagonal, computed with Householder reflections from both sides. It is the first phase of computing the SVD. Introduced in: Lesson 14
gradient check: A test of a derivative: compare $f(x + h\,d) - f(x)$ with $h$ times the claimed directional derivative for decreasing $h$ . The error must fall like $h^2$ . Introduced in: Lesson 16
Gram–Schmidt: Builds orthonormal vectors from the columns of $A$ by subtracting projections onto the earlier ones. The modified version (MGS) is more stable than the classical one (CGS). Introduced in: Lesson 12
graph Fourier transform: $\hat x = U^\top x$ , the coordinates of a graph signal in the eigenvectors of the Laplacian $L = U\Lambda U^\top$ . Small eigenvalues belong to smooth signals. Introduced in: Lesson 19
graph Laplacian: $L = D - A$ for a graph with adjacency matrix $A$ and degree matrix $D$ . It is PSD, $x^\top Lx = \sum_{\text{edges}} A_{ij}(x_i - x_j)^2$ , and its number of zero eigenvalues is the number of connected components. Introduced in: Lesson 19
Grassmann manifold: $\operatorname{Gr}(n, p)$ , the set of $p$ -dimensional subspaces of $\mathbb{R}^n$ . One point can be stored as the projector $XX^\top$ for any $X$ in $\operatorname{St}(n, p)$ that spans it. Introduced in: Lesson 19
group representation: A rule $\rho$ that gives each element $g$ of a group an invertible matrix $\rho(g)$ , with $\rho(gh) = \rho(g)\rho(h)$ . Introduced in: Lesson 19
Hessenberg form: A matrix with zeros below the first subdiagonal. An orthogonal similarity $A = QHQ^\top$ (Householder reflections) brings any square matrix to this form, and a symmetric matrix to tridiagonal form. Introduced in: Lesson 14
Hoffman–Wielandt inequality: For symmetric $A$ and $A + E$ with sorted eigenvalues: $\sum_k (\lambda_k(A + E) - \lambda_k(A))^2 \le \|E\|_F^2$ . Introduced in: Lesson 17
HOSVD: Higher-order SVD: in each mode, take the top left singular vectors of the unfolding, then project the tensor onto them. The truncated HOSVD is quasi-optimal, not optimal. Introduced in: Lesson 15
Householder reflection: $H = I - 2vv^\top / (v^\top v)$ , the reflection across the hyperplane orthogonal to $v$ . It is symmetric and orthogonal and can map a vector to a multiple of $e_1$ . Introduced in: Lesson 12
image (of a vector) تصویر (یک بردار): The output $T(x)$ of a map for the input $x$ . The columns of $A$ are the images of the standard basis vectors. Introduced in: Lesson 1
indefinite matrix: A symmetric matrix with both positive and negative eigenvalues. The graph of its quadratic form is a saddle. Introduced in: Lesson 7
induced norm: The matrix norm $\|A\| = \max_{x \ne 0} \|Ax\|/\|x\|$ given by a vector norm. $\|A\|_1$ is the largest absolute column sum and $\|A\|_\infty$ the largest absolute row sum. Introduced in: Lesson 11
inverse iteration: Power iteration with $(A - \sigma I)^{-1}$ : solve $(A - \sigma I)y = x_k$ and normalize. It converges to the eigenvector whose eigenvalue is closest to the shift $\sigma$ . Introduced in: Lesson 14
invertible matrix: A square matrix $A$ with a matrix $A^{-1}$ such that $A^{-1}A = AA^{-1} = I$ . Equivalently, $N(A) = \{0\}$ , or $\det A \ne 0$ . Introduced in: Lesson 3
Jacobian: The $m \times n$ matrix $J_{ij} = \partial f_i / \partial x_j$ of a map $f: \mathbb{R}^n \to \mathbb{R}^m$ . Near $x$ , $f(x + \delta) \approx f(x) + J\delta$ . Introduced in: Lesson 15
Johnson–Lindenstrauss lemma: A random projection to $k = O(\varepsilon^{-2}\log n)$ dimensions keeps all pairwise distances of $n$ points within a factor $1 \pm \varepsilon$ , with high probability. $k$ does not depend on the original dimension. Introduced in: Lesson 18
Jordan form: A block-diagonal matrix of Jordan blocks $\lambda I + N$ , where $N$ has ones just above the diagonal. Every square matrix is similar to a Jordan form over $\mathbb{C}$ . Introduced in: Lesson 5
JVP (Jacobian-vector product): The product $Jv$ of the Jacobian with a tangent vector $v$ . Forward mode computes it in one pass. Introduced in: Lesson 15
Kronecker product: The block matrix $A \otimes B$ with blocks $a_{ij}B$ . Its eigenvalues are the products $\lambda_i(A)\mu_j(B)$ . Introduced in: Lesson 16
Kronecker sum: $A \otimes I_m + I_n \otimes B$ for $A$ of size $n \times n$ and $B$ of size $m \times m$ . Its eigenvalues are the sums $\lambda_i(A) + \mu_j(B)$ . Introduced in: Lesson 16
Krylov subspace: $\mathcal{K}_k(A, b) = \operatorname{span}\{b, Ab, \dots, A^{k-1}b\} = \{p(A)b : \deg p \le k - 1\}$ . It needs only matrix-vector products. Introduced in: Lesson 14
Lanczos iteration: Arnoldi for a symmetric matrix. $H_k$ becomes tridiagonal, and each new vector needs only the two previous ones. Introduced in: Lesson 14
least squares: Find $\hat x$ that minimizes $\|Ax - b\|_2$ . Then $A\hat x$ is the orthogonal projection of $b$ onto $C(A)$ . Introduced in: Lesson 12
left null space: $N(A^\top) = \{y : A^\top y = 0\}$ , a subspace of $\mathbb{R}^m$ for an $m \times n$ matrix $A$ . It is the orthogonal complement of $C(A)$ . Introduced in: Lesson 2
linear combination ترکیب خطی: A sum $c_1 v_1 + \dots + c_k v_k = Vc$ of vectors multiplied by numbers. Introduced in: Lesson 1
linear map نگاشت خطی: A map $T$ with $T(\alpha x + \beta y) = \alpha T(x) + \beta T(y)$ for all vectors $x, y$ and numbers $\alpha, \beta$ . With a fixed basis, it is the same as a matrix. Introduced in: Lesson 1
linearly dependent وابسته خطی: Not independent; some nonzero combination gives 0, so one vector is a combination of the others. Introduced in: Lesson 1
linearly independent مستقل خطی: Vectors $v_1, \dots, v_k$ with $c_1 v_1 + \dots + c_k v_k = 0$ only for $c = 0$ . Each one adds a new direction. Introduced in: Lesson 1
Lipschitz constant: The smallest $L$ with $\|f(x) - f(y)\| \le L\|x - y\|$ for all $x, y$ . For a linear map $x \mapsto Wx$ in the 2-norm it is $\sigma_1(W)$ . Introduced in: Lesson 11
Loewner order: $A \preceq B$ means that $B - A$ is positive semidefinite: $x^\top A x \le x^\top B x$ for every $x$ . It is a partial order; some pairs cannot be compared. Introduced in: Lesson 7
LoRA: Low-rank adaptation: fine-tune $W_0 + \tfrac{\alpha}{r}BA$ with $B \in \mathbb{R}^{d \times r}$ , $A \in \mathbb{R}^{r \times k}$ trainable and $W_0$ frozen, so only $r(d + k)$ numbers are trained. Introduced in: Lesson 15
LU factorization: $A = LU$ with $L$ unit lower triangular (it stores the elimination multipliers) and $U$ upper triangular (it holds the pivots). With row exchanges, $PA = LU$ . Introduced in: Lesson 10
machine epsilon: The gap $\varepsilon = 2^{1-p}$ between 1 and the next larger float, for $p$ bits of precision. In double precision $\varepsilon = 2^{-52} \approx 2.2 \times 10^{-16}$ . Introduced in: Lesson 13
Marchenko–Pastur law: The limit of the eigenvalue histogram of a sample covariance of pure noise when $p/n \to \gamma$ . For $\gamma \le 1$ it fills $[\sigma^2(1 - \sqrt\gamma)^2, \sigma^2(1 + \sqrt\gamma)^2]$ . Introduced in: Lesson 18
matrix exponential: $e^A = \sum_{k \ge 0} A^k / k!$ , defined for every square $A$ . $x(t) = e^{tA}x_0$ solves $x' = Ax$ . Introduced in: Lesson 16
matrix factorization: Writing a matrix as a product of simple matrices (triangular, orthogonal or diagonal), for example $LU$ , $QR$ , $LL^\top$ or $U\Sigma V^\top$ . Introduced in: Lesson 10
matrix function: For symmetric $S = Q\Lambda Q^\top$ , $f(S) = Qf(\Lambda)Q^\top$ , with $f$ applied to each eigenvalue (not to each entry). Examples are $e^S$ , $S^{1/2}$ and $\log S$ . Introduced in: Lesson 16
mode- $n$ unfolding: The matrix $X_{(n)}$ whose columns are the mode- $n$ fibers of a tensor (all indices fixed except the $n$ -th). For a third-order tensor, $X_{(1)}$ is $I \times JK$ . Introduced in: Lesson 15
multilinear rank: The tuple $(\operatorname{rank} X_{(1)}, \operatorname{rank} X_{(2)}, \operatorname{rank} X_{(3)})$ of a tensor. Its entries can differ. Introduced in: Lesson 15
norm: A function $\|\cdot\|$ with $\|x\| \ge 0$ , $\|x\| = 0$ only for $x = 0$ , $\|cx\| = |c|\,\|x\|$ and $\|x + y\| \le \|x\| + \|y\|$ . Introduced in: Lesson 11
normal equations: $A^\top A\hat x = A^\top b$ . They say that the error $b - A\hat x$ is orthogonal to every column of $A$ , and they give the least-squares solution. Introduced in: Lesson 3
normalized Laplacian: $L_{\text{sym}} = I - D^{-1/2}AD^{-1/2}$ . Its eigenvalues lie in $[0, 2]$ . Introduced in: Lesson 19
nuclear norm: $\|A\|_* = \sum_i \sigma_i$ , the sum of the singular values. It is the convex surrogate for rank used in low-rank models. Introduced in: Lesson 11
null space فضای پوچ: $N(A) = \{x : Ax = 0\}$ . The columns of $A$ are independent exactly when $N(A) = \{0\}$ . Introduced in: Lesson 1
numerical rank: The number of singular values above a cut-off, such as $\text{rcond} \cdot \sigma_1$ . Smaller singular values are treated as zero. Introduced in: Lesson 8
orthogonal complement: The set $V^\perp$ of all vectors that are orthogonal to every vector of a subspace $V$ . In $\mathbb{R}^n$ , $N(A)$ and $C(A^\top)$ are orthogonal complements. Introduced in: Lesson 2
orthogonal matrix: A real square matrix $Q$ with $Q^\top Q = I$ . Its columns are orthonormal, $Q^{-1} = Q^\top$ , and it keeps lengths and angles. It is a rotation ( $\det Q = 1$ ) or a rotation combined with a reflection ( $\det Q = -1$ ). Introduced in: Lesson 4
orthogonal projection: The point $p$ of a subspace that is closest to a vector $b$ . The error $b - p$ is orthogonal to the subspace. Introduced in: Lesson 3
orthogonal projector: A matrix $P$ with $P^2 = P$ and $P^\top = P$ . For $A$ with independent columns, $P = A(A^\top A)^{-1}A^\top$ projects onto $C(A)$ . Introduced in: Lesson 3
orthogonal vectors: Two vectors with dot product zero, $u^\top v = 0$ . In the plane and in space they are perpendicular. Introduced in: Lesson 0
orthonormal: Vectors that are pairwise orthogonal and have length 1: $q_i^\top q_j = 0$ for $i \ne j$ and $q_i^\top q_i = 1$ . Introduced in: Lesson 4
outer product ضرب بیرونی: The matrix $uv^\top$ for column vectors $u, v$ . Its column $j$ is $v_j u$ , so its rank is 1 when $u, v \ne 0$ . Introduced in: Lesson 1
over-smoothing: In a deep GCN without residual connections, repeated propagation acts like power iteration, so the node features become almost the same up to a degree-dependent factor. Introduced in: Lesson 19
oversampling: Taking a few more random vectors than the target rank in a randomized SVD, so that one bad draw does not matter. Introduced in: Lesson 9
partial pivoting: At each elimination step, swap in the row with the largest $|a_{ik}|$ in the current column. Then all multipliers satisfy $|\ell_{ij}| \le 1$ . Introduced in: Lesson 10
Perron–Frobenius theorem: A square matrix with all entries positive has a real, positive, simple eigenvalue that is larger in absolute value than all others. Its eigenvector can be chosen with all entries positive. Introduced in: Lesson 5
pivot محور: The first nonzero entry of a row after elimination. A column without a pivot is a free column. Introduced in: Lesson 1
polar decomposition: For square $A$ : $A = QP$ with $Q$ orthogonal and $P$ symmetric positive semidefinite. From the SVD, $Q = UV^\top$ and $P = V\Sigma V^\top$ . Introduced in: Lesson 8
positive definite: $S \succ 0$ : a symmetric matrix with $x^\top S x > 0$ for every $x \ne 0$ . Equivalently, all its eigenvalues are positive. Introduced in: Lesson 7
positive semidefinite: $S \succeq 0$ : a symmetric matrix with $x^\top S x \ge 0$ for every $x$ . Equivalently, all its eigenvalues are $\ge 0$ . Introduced in: Lesson 7
power iteration: Repeat $x_{k+1} = Ax_k / \|Ax_k\|$ . If $|\lambda_1| > |\lambda_2|$ , almost every start vector turns toward the top eigenvector, with error of order $|\lambda_2/\lambda_1|^k$ . Introduced in: Lesson 5
preconditioning: Solving an equivalent system with a cheap $M \approx A$ , so that the iteration sees $M^{-1}A$ , which has a smaller condition number or clustered eigenvalues. Introduced in: Lesson 14
principal angles: The angles $0 \le \theta_1 \le \dots \le \theta_d \le 90°$ between two $d$ -dimensional subspaces with orthonormal bases $V$ and $\hat V$ : $\cos\theta_i = \sigma_i(\hat V^\top V)$ . Introduced in: Lesson 17
principal component analysis (PCA): Finds the orthogonal directions of largest variance of a data set: the eigenvectors of the sample covariance matrix, or the right singular vectors of the centered data matrix. Introduced in: Lesson 9
principal directions: The eigenvectors $q_i$ of the sample covariance matrix, equal to the right singular vectors of $X_c$ . They are sorted by the variance $\lambda_i = \sigma_i^2/(N - 1)$ along them. Introduced in: Lesson 9
principal minor: The determinant of a submatrix with the same set of row and column indices. A leading principal minor uses the indices $1, \dots, k$ . Introduced in: Lesson 7
pseudoinverse: The Moore–Penrose pseudoinverse $A^+ = V\Sigma^+ U^\top$ , where $\Sigma^+$ inverts each nonzero $\sigma_i$ and keeps the zeros. $A^+ b$ is the minimum-norm least-squares solution. Introduced in: Lesson 8
push-through identity: $X^\top(XX^\top + \lambda I)^{-1} = (X^\top X + \lambda I)^{-1}X^\top$ . It lets ridge regression invert the smaller of the two matrices. Introduced in: Lesson 10
QK and OV circuits: The two low-rank $d \times d$ matrices of an attention head: $W_{QK}$ decides where a token looks, and $W_{OV}$ decides what is read and written. Both have rank at most $d_h$ . Introduced in: Lesson 15
QR algorithm: Repeat: factor $A_k = Q_kR_k$ , then set $A_{k+1} = R_kQ_k$ . With Hessenberg reduction, shifts and deflation it is the standard dense eigenvalue method. Introduced in: Lesson 14
QR factorization: $A = QR$ with $Q$ having orthonormal columns and $R$ upper triangular. The reduced (thin) QR has $Q$ of size $m \times n$ . Introduced in: Lesson 12
QR with column pivoting: $AP = QR$ , where at each step the remaining column of largest norm goes first. Small diagonal entries of $R$ reveal the numerical rank. Introduced in: Lesson 12
quadratic form: The function $x \mapsto x^\top S x$ of a symmetric matrix $S$ , also called the energy. Introduced in: Lesson 7
random matrix theory: The study of the eigenvalues and singular values of large random matrices. It shows how pure noise looks, so you can tell which eigenvalues carry signal. Introduced in: Lesson 18
randomized SVD: An approximate truncated SVD: multiply $A$ by a random matrix, orthonormalize the result to get $Q$ , then take the small SVD of $Q^\top A$ . Introduced in: Lesson 9
range finder: The first stage of the randomized SVD: $Q = \operatorname{orth}(A\Omega)$ for a random $\Omega$ , an orthonormal basis for (almost) the top part of the range of $A$ . Introduced in: Lesson 9
rank رتبه: The dimension of the column space of a matrix: the number of independent columns. Introduced in: Lesson 1
rank–nullity theorem: For $A \in \mathbb{R}^{m \times n}$ : $\operatorname{rank}(A) + \dim N(A) = n$ . Introduced in: Lesson 2
Rayleigh quotient: $R(x) = x^\top S x / x^\top x$ for $x \ne 0$ . For symmetric $S$ it lies between the smallest and the largest eigenvalue and reaches them at the eigenvectors. Introduced in: Lesson 6
Rayleigh quotient iteration: Inverse iteration with the shift updated to the Rayleigh quotient at every step. For symmetric matrices it converges cubically. Introduced in: Lesson 14
reduced row echelon form (RREF): An echelon form in which every pivot is 1 and every other entry in a pivot column is 0. Each matrix has exactly one RREF. Introduced in: Lesson 2
residual: $r = b - A\hat x$ for an approximate solution $\hat x$ . In least squares it is orthogonal to every column of $A$ . Introduced in: Lesson 12
retraction: A smooth map that brings a point $X + V$ back onto the manifold after a step, for example the Q factor of a QR factorization or the polar factor. Introduced in: Lesson 19
reverse mode: Automatic differentiation that multiplies the chain of Jacobians from the output side (VJPs). One pass gives the full gradient of a scalar loss; backpropagation is reverse mode. Introduced in: Lesson 15
Riemannian gradient: The Euclidean gradient projected onto the tangent space of the manifold. On the Stiefel manifold it is $G - X\operatorname{sym}(X^\top G)$ . Introduced in: Lesson 19
Ritz values: The eigenvalues of $H_k = Q_k^\top AQ_k$ , the matrix compressed onto a Krylov subspace. They approximate the extreme eigenvalues of $A$ first. Introduced in: Lesson 14
row space فضای سطری: The span of the rows of $A$ . Every row of $AB$ lies in the row space of $B$ . Introduced in: Lesson 1
sample covariance matrix: $S = X_c^\top X_c / (N - 1)$ for $N$ centered samples in the rows of $X_c$ . It is symmetric positive semidefinite. Introduced in: Lesson 9
Schatten $p$ -norm: The $\ell_p$ norm of the vector of singular values. $p = 1$ , $2$ and $\infty$ give the nuclear, Frobenius and spectral norms. Introduced in: Lesson 11
Schur complement: For $M = \begin{pmatrix} A & B \\ B^\top & C \end{pmatrix}$ with $A$ invertible: $M/A = C - B^\top A^{-1} B$ . Then $M \succ 0$ exactly when $A \succ 0$ and $M/A \succ 0$ . Introduced in: Lesson 7
scores: The coordinates of the samples on the principal directions: $X_c V = U\Sigma$ . Introduced in: Lesson 9
shear برش: A map that keeps one axis fixed and slides the points parallel to it, e.g. $(x, y) \mapsto (x + y, y)$ . Introduced in: Lesson 1
Sherman–Morrison formula: The rank-1 case of the Woodbury formula: $(A + uv^\top)^{-1} = A^{-1} - \frac{A^{-1}uv^\top A^{-1}}{1 + v^\top A^{-1}u}$ , if $1 + v^\top A^{-1}u \ne 0$ . Introduced in: Lesson 10
significand: The number $(1.f)_2$ that a float multiplies by a power of 2. Its leading 1 is not stored (the hidden bit). Introduced in: Lesson 13
similar matrices: $A$ and $B = M^{-1}AM$ for an invertible $M$ : one linear map written in two bases. They have the same eigenvalues, trace and determinant, but in general not the same singular values. Introduced in: Lesson 4
singular value: A diagonal entry $\sigma_i \ge 0$ of $\Sigma$ in the SVD. The $\sigma_i^2$ are the eigenvalues of $A^\top A$ , and $\sigma_1$ is the largest stretch of $A$ . Introduced in: Lesson 8
singular value decomposition (SVD): $A = U\Sigma V^\top$ with $U$ and $V$ orthogonal and $\Sigma$ diagonal with $\sigma_1 \ge \sigma_2 \ge \dots \ge 0$ . Every real matrix has one. Introduced in: Lesson 8
singular vectors: The columns $u_i$ of $U$ (left singular vectors) and $v_i$ of $V$ (right singular vectors) in the SVD, with $Av_i = \sigma_i u_i$ and $A^\top u_i = \sigma_i v_i$ . Introduced in: Lesson 8
sketching: Multiplying the data by a short random matrix $S$ and working with the small result $SA$ , for example to solve a huge least-squares problem. Introduced in: Lesson 18
soft-thresholding: $S_\lambda(z) = \operatorname{sign}(z)\max(|z| - \lambda, 0)$ , the minimizer of $\tfrac12(x - z)^2 + \lambda|x|$ . It sets small values exactly to zero, as in the lasso. Introduced in: Lesson 11
span فضای پوشش: The set of all linear combinations of some vectors; for the columns of $V$ it is the column space $C(V)$ . Introduced in: Lesson 1
spanning set مجموعه‌ی پوشا: A set of vectors whose span is the whole subspace. It may contain extra (dependent) vectors. Introduced in: Lesson 1
special solution: The solution of $Ax = 0$ with one free variable equal to 1 and the other free variables equal to 0. The special solutions form a basis of $N(A)$ . Introduced in: Lesson 2
spectral norm: $\|A\|_2 = \max_{x \ne 0} \|Ax\|/\|x\|$ , the largest stretch of $A$ . It equals the largest singular value $\sigma_1$ ; for symmetric $A$ it is $\max_i |\lambda_i|$ . Introduced in: Lesson 6
spectral normalization: Dividing each weight matrix by an estimate of its largest singular value, so that every layer is (about) 1-Lipschitz. Introduced in: Lesson 11
spectral theorem: Every real symmetric matrix is $S = Q\Lambda Q^\top$ with $Q$ orthogonal and $\Lambda$ real and diagonal: real eigenvalues and an orthonormal basis of eigenvectors. Introduced in: Lesson 6
standard basis vector بردار پایه‌ی استاندارد: The vector $e_j$ with a 1 in position $j$ and 0 elsewhere. $Ae_j$ is column $j$ of $A$ . Introduced in: Lesson 1
standardization: Dividing each centered feature by its standard deviation, so that all features have variance 1. PCA of standardized data is PCA of the correlation matrix. Introduced in: Lesson 9
Stiefel manifold: $\operatorname{St}(n, p) = \{X \in \mathbb{R}^{n \times p} : X^\top X = I_p\}$ , the set of matrices with orthonormal columns. Its dimension is $np - p(p+1)/2$ . Introduced in: Lesson 19
subspace زیرفضا: A set of vectors that contains 0 and is closed under addition and multiplication by numbers. Introduced in: Lesson 1
subspace embedding: A matrix $S$ with $(1 - \varepsilon)\|y\|^2 \le \|Sy\|^2 \le (1 + \varepsilon)\|y\|^2$ for all $y$ in a subspace. It is oblivious if it works for any fixed subspace without looking at the data. Introduced in: Lesson 18
subspace iteration: Power iteration on a block of vectors, with an orthonormalization after every product. In the randomized SVD it turns the weights $\sigma_j$ into $\sigma_j^{2q+1}$ . Introduced in: Lesson 9
superposition: A network represents more features than it has dimensions by giving each feature an almost-orthogonal direction. It works when features are sparse. Introduced in: Lesson 18
swamping: The loss of a small term $t$ added to a large sum $s$ , when $|t|$ is less than half the gap between floats near $s$ . Introduced in: Lesson 13
Sylvester equation: $AX + XB = C$ , linear in $X$ . It has a unique solution exactly when no $\lambda_i(A) + \mu_j(B)$ is zero. Introduced in: Lesson 16
symmetric matrix: A square matrix with $S^\top = S$ , that is, $s_{ij} = s_{ji}$ . Introduced in: Lesson 6
tensor: An array with three or more indices, such as $\mathcal{X} \in \mathbb{R}^{I \times J \times K}$ with entries $x_{ijk}$ . Introduced in: Lesson 15
thin SVD: The reduced SVD of an $m \times n$ matrix: it keeps only $p = \min(m, n)$ columns of $U$ and $V$ and the $p \times p$ block of $\Sigma$ . Introduced in: Lesson 8
truncated SVD: $A_k = \sum_{i \le k} \sigma_i u_i v_i^\top$ : the SVD with only the $k$ largest terms. Introduced in: Lesson 8
Tucker decomposition: $\mathcal{X} \approx \mathcal{G} \times_1 A \times_2 B \times_3 C$ : a small core tensor $\mathcal{G}$ multiplied by a factor matrix in each mode. Introduced in: Lesson 15
underflow: A result too small for the float format: it loses precision or becomes 0. Small fp16 gradients underflow without loss scaling. Introduced in: Lesson 13
unit ball: The set $\{x : \|x\| \le 1\}$ of a norm: a disk for $\ell_2$ , a diamond for $\ell_1$ and a square for $\ell_\infty$ in the plane. Introduced in: Lesson 11
unit lower triangular: A lower triangular matrix with ones on the diagonal. Introduced in: Lesson 10
unit roundoff: $u = \varepsilon/2$ , the largest relative error of rounding to nearest: $\operatorname{fl}(x) = x(1 + \delta)$ with $|\delta| \le u$ . Introduced in: Lesson 13
unitary matrix: A complex square matrix $U$ with $U^* U = I$ , the complex version of an orthogonal matrix. Every eigenvalue has $|\lambda| = 1$ . Introduced in: Lesson 4
vec: Stacks the columns of a matrix into one long vector. Then $\operatorname{vec}(AXB) = (B^\top \otimes A)\operatorname{vec}(X)$ . Introduced in: Lesson 16
VJP (vector-Jacobian product): The product $J^\top u$ (the row vector $u^\top J$ ). Reverse mode computes it in one pass. Introduced in: Lesson 15
Wedin's theorem: The Davis–Kahan bound for singular vectors of any matrix: the top- $k$ singular subspaces rotate by at most about $\|E\|_2$ divided by the gap $\sigma_k - \sigma_{k+1}$ . Introduced in: Lesson 17
Weyl's inequality: For symmetric $A$ and $E$ , each sorted eigenvalue moves by at most the spectral norm of the change: $|\lambda_k(A + E) - \lambda_k(A)| \le \|E\|_2$ . Introduced in: Lesson 6
Wilkinson shift: The eigenvalue of the trailing $2 \times 2$ block that is closer to the last diagonal entry. The QR algorithm uses it as the shift. Introduced in: Lesson 14
Woodbury formula: $(A + UCV)^{-1} = A^{-1} - A^{-1}U(C^{-1} + VA^{-1}U)^{-1}VA^{-1}$ . It updates an inverse after a rank- $k$ change at the cost of a $k \times k$ inverse. Introduced in: Lesson 10
$\ell_p$ norm: $\|x\|_p = (\sum_i |x_i|^p)^{1/p}$ for $p \ge 1$ , and $\|x\|_\infty = \max_i |x_i|$ . For $p < 1$ it is not a norm. Introduced in: Lesson 11