Deep Learning · 181

Glossary

The technical terms of this track, with short definitions (and the Persian term where available). Each term links to the lesson where it is introduced.

1-WL test: Weisfeiler–Leman colour refinement: repeatedly recolour each node by its colour and the multiset of its neighbours’ colours. Message-passing GNNs cannot separate graphs that 1-WL cannot separate. Introduced in: Lesson 13
1×1 convolution: The same linear map from $C_{\text{in}}$ to $C_{\text{out}}$ channels at every pixel. It mixes channels cheaply. Introduced in: Lesson 7
activation: The output $A_l = \sigma(Z_l)$ of layer $l$ after the elementwise nonlinearity $\sigma$ . The function $\sigma$ itself is the activation function. Introduced in: Lesson 1
activation checkpointing: Store the activations only at some layers and recompute the others during the backward pass. It trades extra compute for less memory. Also called rematerialization. Introduced in: Lesson 3
activation function: The elementwise nonlinearity $\sigma$ between affine layers, such as ReLU, tanh, GELU or SiLU. Without it a deep network collapses to one affine map. Introduced in: Lesson 4
active parameters: The parameters used for one token: the shared layers plus $k$ experts per MoE layer. Total parameters count all experts. Introduced in: Lesson 15
Adam: An optimizer that keeps running averages of the gradient ( $m$ ) and of the squared gradient ( $v$ ) and updates $\theta \leftarrow \theta - \eta\,\hat m/(\sqrt{\hat v} + \epsilon)$ with bias-corrected $\hat m, \hat v$ . Introduced in: Lesson 2
AdamW: Adam with decoupled weight decay: the moments use the loss gradient only, and the decay $\eta\lambda\theta$ is applied separately to the weights. Introduced in: Lesson 6
adjacency matrix: The $n \times n$ matrix $A$ with $A_{ij} = 1$ if there is an edge between nodes $i$ and $j$ and 0 otherwise. Introduced in: Lesson 13
adjoint: The derivative $\bar u = \partial\mathcal{L}/\partial u$ of the final scalar loss with respect to an intermediate variable $u$ . Introduced in: Lesson 3
ALiBi: Attention with linear biases: subtract $m_h(i - j)$ from the score of query $i$ and key $j$ , with a fixed slope $m_h$ per head. Introduced in: Lesson 10
all-to-all: A collective operation in which every GPU sends a different piece of data to every other GPU. An expert-parallel MoE layer needs two of them in the forward pass. Introduced in: Lesson 15
attention: $\operatorname{softmax}(QK^\top/\sqrt{d_k} + M)V$ : each query takes a weighted average of the values, with weights from its scores against all keys. Introduced in: Lesson 9
backpropagation: Reverse-mode automatic differentiation applied to a network: a backward pass computes the gradient of the loss with respect to all parameters for a small constant times the cost of the forward pass. Introduced in: Lesson 3
backpropagation through time (BPTT): Backpropagation on the RNN unrolled in time: a deep network with one layer per step and shared weights. Introduced in: Lesson 8
backward pass: The pass that visits the operations in reverse order and turns the adjoint of each output into adjoints of its inputs. Introduced in: Lesson 3
batch normalization (BatchNorm): Normalize each channel to mean 0 and variance 1 over the examples of the batch (and the positions), then apply a learned scale $\gamma$ and shift $\beta$ . In eval mode it uses running estimates of the statistics. Introduced in: Lesson 5
benign overfitting: Fitting noisy labels exactly with little harm to the test error, when many low-variance directions absorb the noise. Introduced in: Lesson 17
bf16: "Brain float": a 16-bit format with the exponent range of fp32 but only 8 significant bits. Introduced in: Lesson 12
bottleneck block: A ResNet block $1\times1 \to 3\times3 \to 1\times1$ that reduces the channels, convolves, and expands them again, plus the identity. Introduced in: Lesson 7
capacity factor: The factor cf in the expert capacity. $\mathrm{cf} = 1$ gives exactly enough slots for perfectly balanced routing. Introduced in: Lesson 15
causal mask: An additive mask with $-\infty$ above the diagonal, so query $i$ attends only to keys $j \le i$ . Introduced in: Lesson 9
cell state: The second state $c_t$ of an LSTM. It is updated by addition, which avoids long products of Jacobians. Introduced in: Lesson 8
cheap gradient principle: Reverse mode computes the gradient of any scalar program for a small constant times the cost of the program, whatever the number of inputs. Introduced in: Lesson 3
CLIP: Contrastive image–text pretraining: a symmetric InfoNCE loss over a batch of image–caption pairs. It allows zero-shot classification by comparing an image with text prompts. Introduced in: Lesson 14
collapse: The trivial solution of a self-supervised objective in which the encoder maps every input to the same vector. Introduced in: Lesson 14
computational graph: A graph with one node per operation and one edge per value. Backpropagation walks it in reverse; a value used in several places sums the contributions. Introduced in: Lesson 3
context length: The largest number of positions $T_{\max}$ that a model processes at once. Introduced in: Lesson 11
contrastive learning: Pull the embeddings of two views of the same item together and push the embeddings of different items apart. Introduced in: Lesson 14
convolution layer: A layer that slides small filters over the input and takes a dot product at each position. The same weights are used at every position. Introduced in: Lesson 7
cosine decay: $\eta(t) = \eta_{\min} + \frac12(\eta_{\max} - \eta_{\min})(1 + \cos(\pi r))$ , where $r$ goes from 0 to 1 after warmup. Introduced in: Lesson 12
critical batch size: The batch size beyond which a larger batch gives little gain per step: the gradient is already accurate, and only the number of steps matters. Introduced in: Lesson 12
cross-attention: Attention where the queries come from one sequence and the keys and values from another, for example from an encoder. Introduced in: Lesson 9
cross-correlation: $y_i = \sum_u w_u x_{i+u}$ : the operation that deep-learning libraries call convolution. A true convolution flips the kernel: $y_i = \sum_u w_u x_{i-u}$ . Introduced in: Lesson 7
cross-entropy: The classification loss $-\log p_y$ : minus the log of the probability that the model gives to the correct class. It is the negative log-likelihood of a categorical model. Introduced in: Lesson 2
curse of dimensionality: A grid of spacing $h$ in $d$ dimensions has about $h^{-d}$ cells, so constructions that cover the input space need a number of units that grows exponentially with $d$ . Introduced in: Lesson 1
data augmentation: Train on transformed copies of the inputs (flips, crops, colour changes) that keep the label. It should use only true symmetries of the task. Introduced in: Lesson 6
dead ReLU: A ReLU unit whose pre-activation is negative for all inputs, so its output and its gradient are always zero and it stops learning. Introduced in: Lesson 4
decoder-only language model: A stack of causal transformer blocks between a token embedding and an unembedding, trained to predict the next token. GPT and LLaMA are examples. Introduced in: Lesson 11
depth: The number $L$ of affine layers of a network. Introduced in: Lesson 1
depthwise convolution: A convolution with one filter per input channel ( $g = C_{\text{in}}$ groups), so the channels do not mix. Introduced in: Lesson 7
dilation: The distance $d$ between neighbouring taps of a kernel. $d = 1$ is an ordinary kernel; $d = 2$ skips every other input. Introduced in: Lesson 7
double descent: The test error first falls, then rises to a peak at the interpolation threshold, then falls again as the model size (or the training time) grows. Introduced in: Lesson 17
dropout: During training, set each unit to zero with probability $p$ and divide the survivors by $1 - p$ . At test time, use all units. Introduced in: Lesson 6
dynamical isometry: All singular values of the input-output Jacobian of a deep network are close to 1, so signals and gradients keep their size through depth. Introduced in: Lesson 4
early stopping: Stop training when the validation error stops falling. For least squares with gradient descent it acts like L2 (ridge) regularization. Introduced in: Lesson 6
effective learning rate: For weights followed by a normalization layer, the direction $W/\|W\|$ moves at a rate of about $\eta/\|W\|^2$ . Weight decay keeps $\|W\|$ small and so keeps this rate up. Introduced in: Lesson 5
effective model complexity: The largest number of samples on which a training procedure reaches a training error of at most $\epsilon$ (for example 0.1). Double descent peaks when it is close to the number of samples. Introduced in: Lesson 17
embedding: A learned vector for each token. The embedding matrix $E \in \mathbb{R}^{|\mathcal V| \times D}$ has one row per token of the vocabulary. Introduced in: Lesson 8
empirical risk: The average loss over the training set, $\frac1n\sum_{i=1}^n \ell(f_\theta(x_i), y_i)$ . Training minimizes it. Introduced in: Lesson 2
epoch: One pass over the whole training set: $n/B$ iterations with mini-batches of size $B$ . Introduced in: Lesson 2
expert capacity: The number of token slots of each expert, $C = \lceil kT\,\mathrm{cf}/E\rceil$ . Tokens beyond it are dropped. Introduced in: Lesson 15
fan-in: The number of inputs $n_{\text{in}}$ of a unit or a weight matrix. The fan-out is the number of outputs. Introduced in: Lesson 4
feature map: The output of one filter of a convolution layer: one value per position. Introduced in: Lesson 7
filter (kernel): The small grid of learned weights, for example $3\times3$ per input channel, that a convolution layer slides over its input. Introduced in: Lesson 7
filter factors: The factors that say how much of each eigen-direction of $S = X^\top X/n$ (eigenvalue $s_i$ ) a solution keeps: $s_i/(s_i + \lambda)$ for ridge and $1 - (1 - \eta s_i)^t$ for $t$ steps of gradient descent. Introduced in: Lesson 6
forget gate: The LSTM gate $f_t \in (0, 1)$ that decides how much of the old cell state to keep. Introduced in: Lesson 8
forward mode: Automatic differentiation that pushes one input direction through the graph per pass (a Jacobian-vector product). It needs $n$ passes for $n$ inputs. Introduced in: Lesson 3
forward pass: Running the network on the inputs to compute the loss, while storing the intermediate values needed by the backward pass. Introduced in: Lesson 3
gated linear unit (GLU): A layer that multiplies a linear branch entry by entry by a gate: $(xW + b) \odot s(xV + c)$ . SwiGLU uses SiLU as the gate and is the feed-forward network of LLaMA-style models. Introduced in: Lesson 4
GELU: Gaussian error linear unit, $x\,\Phi(x)$ , where $\Phi$ is the standard normal CDF. A smooth version of ReLU. Introduced in: Lesson 4
gradient check: Compare the gradient of a backward pass with a finite-difference estimate such as the central difference $(f(x + he_i) - f(x - he_i))/(2h)$ . Introduced in: Lesson 3
gradient clipping: Rescale a gradient whose norm exceeds a threshold $c$ : $g \leftarrow g \cdot \min(1, c/\|g\|)$ . Global-norm clipping uses the norm of all parameters together. Introduced in: Lesson 8
graph attention (GAT): Attention masked by the adjacency matrix: each node takes a learned weighted average over its neighbours. Introduced in: Lesson 13
graph convolutional network (GCN): The layer $H' = \sigma(\hat AHW)$ with $\hat A = \tilde D^{-1/2}(A + I)\tilde D^{-1/2}$ : a normalized average over each node and its neighbours. Introduced in: Lesson 13
grouped-query attention (GQA): Query heads share $H_{kv} < H$ key and value heads, which makes the KV cache $H/H_{kv}$ times smaller. Multi-query attention is the case $H_{kv} = 1$ . Introduced in: Lesson 11
GRU: Gated recurrent unit: an RNN with an update gate that mixes the old state and a candidate, $h_t = u_t \odot h_{t-1} + (1 - u_t)\odot\tilde h_t$ . No separate cell state. Introduced in: Lesson 8
He initialization: Draw weights with variance $2/n_{\text{in}}$ . For ReLU it keeps the second moment of the activations constant from layer to layer. Introduced in: Lesson 4
hidden layer: A layer between the input and the output layer (layers $1, \dots, L-1$ ). Its values are not given by the data. Introduced in: Lesson 1
hidden state: The vector $h_t$ that an RNN carries from step to step; a running summary of the inputs read so far. Introduced in: Lesson 8
HiPPO: A structured initialization of $A$ derived from online polynomial approximation of the input history. It lets an SSM keep a useful summary of a long past. Introduced in: Lesson 16
homophily: Neighbouring nodes usually have the same label. Smoothing then helps, which is why a 2-layer GCN works on citation graphs. Introduced in: Lesson 13
im2col: Copy every input patch into a column of a matrix, so that a convolution becomes one matrix product. Introduced in: Lesson 7
implicit regularization (implicit bias): The preference of a training algorithm for some solutions over others with the same training loss, without any penalty term in the loss. Introduced in: Lesson 17
impulse response: The output of a linear system for the input $(1, 0, 0, \dots)$ . For an LTI SSM it is the convolution kernel. Introduced in: Lesson 16
in-batch negatives: Using the other $N - 1$ items of the batch as the negatives of each positive pair. Introduced in: Lesson 14
inductive bias: An assumption built into the architecture, such as locality and translation equivariance in a CNN. Less inductive bias needs more data. Introduced in: Lesson 11
InfoNCE loss: The cross-entropy of identifying the positive pair among $N$ candidates, with logits $u_i^\top w_j/\tau$ . It is a lower bound on mutual information: $I \ge \log N - \mathcal{L}$ . Introduced in: Lesson 14
interpolation threshold: The model size at which the model can just fit every training point exactly. Introduced in: Lesson 17
KV cache: The keys and values of past tokens, stored during generation so that each new token computes only its own query, key and value. Introduced in: Lesson 9
L2 regularization: Add $\frac\lambda2\|\theta\|^2$ to the loss. Its gradient $\lambda\theta$ pulls the weights toward zero. Introduced in: Lesson 6
label smoothing: Train on the target $q = (1 - \varepsilon)\,\text{onehot}(y) + \frac{\varepsilon}{K}\mathbf 1$ instead of the one-hot vector. The optimal logits then stay finite. Introduced in: Lesson 6
layer normalization (LayerNorm): Normalize the features of each token (or example) to mean 0 and variance 1, then apply learned $\gamma, \beta$ . It never mixes the examples of a batch. Introduced in: Lesson 5
learning rate: The step size $\eta$ in the update $\theta \leftarrow \theta - \eta g$ . Introduced in: Lesson 2
learning-rate schedule: The learning rate as a function of the step, for example linear warmup followed by cosine decay or warmup–stable–decay (WSD). Introduced in: Lesson 12
learning-rate warmup: Start training with a very small learning rate and increase it over the first steps, so the first updates stay small. Introduced in: Lesson 5
linear probe: Freeze a pretrained encoder and train only a linear classifier on its features, to measure the quality of the representation. Introduced in: Lesson 14
linear scaling rule: When the batch size is multiplied by $k$ , multiply the SGD learning rate by $k$ . It holds up to the critical batch size. Introduced in: Lesson 12
linear time invariance (LTI): The parameters are the same at every step, so the layer is a convolution with a fixed kernel $K_j = C\bar A^j\bar B$ . Introduced in: Lesson 16
load-balancing loss: An auxiliary loss $\alpha E\sum_i f_iP_i$ , with $f_i$ the fraction of tokens sent to expert $i$ and $P_i$ its mean router probability. It is smallest when the load is uniform. Introduced in: Lesson 15
log-sum-exp trick: Compute $\log\sum_j e^{z_j}$ as $m + \log\sum_j e^{z_j - m}$ with $m = \max_j z_j$ , so no exponential overflows. Introduced in: Lesson 2
logits: The raw outputs $Z_L$ of the last layer of a classifier, before the softmax. They are unnormalized log-probabilities. Introduced in: Lesson 1
long short-term memory (LSTM): An RNN with a cell state $c_t = f_t \odot c_{t-1} + i_t \odot g_t$ controlled by forget, input and output gates. A forget gate near 1 lets information and gradients pass many steps. Introduced in: Lesson 8
loss function: A function $\ell(\hat y, y)$ that measures how wrong a prediction $\hat y$ is for the target $y$ . Introduced in: Lesson 2
loss scaling: Multiply the loss by a factor $s$ before the backward pass and divide the gradients by $s$ afterwards, so small fp16 gradients do not underflow to 0. Introduced in: Lesson 12
loss spike: A sudden jump of the training loss after a step that is too large for the current curvature. If the run does not recover, it diverges. Introduced in: Lesson 12
machine epsilon: The gap between 1 and the next larger number of a floating-point format: $2^{-23}$ for fp32, $2^{-10}$ for fp16, $2^{-7}$ for bf16. Introduced in: Lesson 12
masked autoencoder (MAE): Hide a random 75% of the image patches, encode only the visible ones, and train a light decoder to reconstruct the pixels of the masked patches. Introduced in: Lesson 14
masked language modelling: Hide about 15% of the input tokens and train an encoder to predict them from both sides, as in BERT. Introduced in: Lesson 11
maximal update parametrization (μP): A rule for how initialization scales and learning rates change with width so that every layer learns at a width-independent rate. It allows hyperparameters tuned on a small model to transfer to a wide one (μTransfer). Introduced in: Lesson 4
message-passing neural network (MPNN): A GNN layer in which each node sends messages along its edges, aggregates the incoming messages with a permutation-invariant function, and updates its state. Introduced in: Lesson 13
mini-batch stochastic gradient descent (SGD): Gradient descent that uses the average gradient over a random mini-batch of $B$ examples instead of the whole training set. Introduced in: Lesson 2
minimum-norm interpolant: Among all solutions of $Xw = y$ , the one with the smallest norm: $X^+y = X^\top(XX^\top)^{-1}y$ . Gradient descent from $w_0 = 0$ finds it. Introduced in: Lesson 17
mixed-precision training: Compute in 16 bits but keep fp32 master weights and accumulate sums in fp32, so small updates are not lost. Introduced in: Lesson 12
mixture of experts (MoE): A layer with many expert networks and a router that sends each token to only $k$ of them, so the parameter count grows without more compute per token. Introduced in: Lesson 15
momentum: A running average of past gradients that the optimizer uses instead of the current gradient alone. It smooths the steps and speeds up progress along consistent directions. Introduced in: Lesson 2
momentum encoder: An exponential moving average of an encoder’s weights, $\xi \leftarrow m\xi + (1 - m)\theta$ , used as a slowly changing target (MoCo, BYOL, DINO). Introduced in: Lesson 14
Monte Carlo dropout: Keep dropout on at test time and average several predictions. Their spread gives a cheap uncertainty estimate. Introduced in: Lesson 6
multi-head attention: $H$ attention heads in parallel, each with its own projections of size $d_k = D/H$ . Their outputs are concatenated and mixed by $W_O$ . Introduced in: Lesson 9
multilayer perceptron (MLP): A stack of affine layers with an elementwise nonlinearity between them: $Z_l = A_{l-1}W_l + \mathbf 1 b_l$ , $A_l = \sigma(Z_l)$ . The last layer has no activation. Introduced in: Lesson 1
negative log-likelihood: The loss $-\log p_\theta(y \mid x)$ . Minimizing its average is maximum likelihood estimation. Introduced in: Lesson 2
neural tangent kernel (NTK): The kernel $J(x)J(x')^\top$ of the linearized network at initialization. Very wide networks train like this linear model. Introduced in: Lesson 17
nucleus (top-p) sampling: Sample only from the smallest set of tokens whose probabilities sum to at least $p$ , after renormalization. Introduced in: Lesson 11
over-smoothing: After many propagation steps, all node states of a connected graph become nearly equal (up to degree), so the nodes cannot be told apart. Introduced in: Lesson 13
over-squashing: Information from many distant nodes must pass through a few edges, so it is compressed into fixed-size vectors and lost. Introduced in: Lesson 13
overfitting: The training loss keeps falling while the validation loss rises: the model fits details of the training set that do not carry over to new data. Introduced in: Lesson 2
parallel scan: An algorithm that computes all prefix results of an associative operation, such as a linear recurrence, in $O(\log L)$ parallel rounds. Introduced in: Lesson 16
permutation equivariance: Renumbering the nodes renumbers the outputs in the same way: $F(PH, PAP^\top) = PF(H, A)$ for every permutation matrix $P$ . Introduced in: Lesson 13
permutation invariance: Renumbering the nodes does not change the output: $R(PH, PAP^\top) = R(H, A)$ . Graph-level readouts need it. Introduced in: Lesson 13
pooling: Replace each window by its maximum (max pooling) or mean (average pooling). Global average pooling averages each channel over all positions. Introduced in: Lesson 7
population risk: The expected loss on new data from the same distribution, $R(\theta) = \mathbb{E}_{(x,y)}\,\ell(f_\theta(x), y)$ . It is what we want to be small. Introduced in: Lesson 2
position interpolation: To extend a RoPE model from length $L$ to $L'$ , use the positions $m\,L/L'$ , so every angle stays in the range seen in training. Introduced in: Lesson 10
positional encoding: Information about the position of each token, added to the input or to the attention scores, because attention by itself ignores order. Introduced in: Lesson 10
post-LN: A transformer block that normalizes after the addition: $\operatorname{LN}(x + F(x))$ . Used by the original Transformer and BERT; it usually needs learning-rate warmup. Introduced in: Lesson 5
pre-activation: The value $Z_l = A_{l-1}W_l + \mathbf 1 b_l$ of layer $l$ before the activation function is applied. Introduced in: Lesson 1
pre-LN: A transformer block that normalizes the input of each branch: $x + F(\operatorname{LN}(x))$ . Used by GPT-2, LLaMA and most modern models; it needs a final LayerNorm. Introduced in: Lesson 5
prefill and decode: The two phases of inference: prefill reads the prompt in one parallel pass and fills the KV cache; decode makes one token per step and is limited by memory bandwidth. Introduced in: Lesson 11
qk-layernorm: A LayerNorm applied to each query and each key before the dot product. It bounds the attention logits and prevents a common kind of divergence. Introduced in: Lesson 12
query, key, value: Three learned projections of the tokens. The query says what a token looks for, the key what it can be matched on, and the value what it gives when chosen. Introduced in: Lesson 9
Rademacher complexity: A measure of how well a function class can fit random $\pm1$ labels on $n$ inputs. Uniform convergence bounds use it. Introduced in: Lesson 17
readout: A permutation-invariant map, such as a sum over nodes followed by an MLP, that turns all node states into one vector for a graph-level task. Introduced in: Lesson 13
receptive field: The set of input pixels that can change the value of a unit. Its size grows with each layer, and faster after strided layers. Introduced in: Lesson 7
rectified linear unit (ReLU): The activation $\operatorname{relu}(u) = \max(0, u)$ . One ReLU unit $\operatorname{relu}(wx + b)$ of one input has a hinge-shaped graph with its joint at $x = -b/w$ . Introduced in: Lesson 1
recurrent neural network (RNN): A network that reads a sequence one item at a time and updates a hidden state with the same weights at every step: $h_t = \tanh(h_{t-1}W_{hh} + x_tW_{xh} + b_h)$ . Introduced in: Lesson 8
regularizer: Any change to the model, the loss, the data or the training procedure that aims to lower the test error rather than the training error. Introduced in: Lesson 6
residual connection: A block computes $x_{l+1} = x_l + F_l(x_l)$ instead of $F_l(x_l)$ . The identity path gives the gradient a direct route back to early layers. Introduced in: Lesson 5
residual stream: The sequence $x_0, x_1, \dots$ of vectors that the residual blocks of a network read from and add to. Introduced in: Lesson 5
reverse mode: Automatic differentiation that multiplies the chain of Jacobians from the output side. One pass gives the gradient of a scalar with respect to all inputs. Introduced in: Lesson 3
RMSNorm: Divide each token vector by its root mean square, $\sqrt{\frac1D\sum_i x_i^2 + \varepsilon}$ , and multiply by a learned $\gamma$ . No mean subtraction and no shift. Introduced in: Lesson 5
rotary position embedding (RoPE): Rotate each pair $i$ of every query and key at position $m$ by the angle $m\theta_i$ . Then $\langle R_mq, R_nk\rangle = q^\top R_{n-m}k$ depends only on the offset. Introduced in: Lesson 10
router: A small linear layer with a softmax that scores the experts for each token and picks the top $k$ . Introduced in: Lesson 15
router z-loss: $\frac1T\sum_t(\log\sum_j e^{z_{t,j}})^2$ : it keeps the router logits small and so stabilizes training. Introduced in: Lesson 15
row-vector convention: The convention of PyTorch and of this track: each example is a row, and a layer computes $XW + b$ with $W \in \mathbb{R}^{n_{\text{in}} \times n_{\text{out}}}$ . Introduced in: Lesson 1
selective SSM (S6): The SSM of Mamba: $\Delta_t$ , $B_t$ and $C_t$ depend on the current input, so the model chooses what to remember. It is no longer a convolution. Introduced in: Lesson 16
self-attention: Attention where the queries, keys and values all come from the same sequence. Without positions it is permutation-equivariant. Introduced in: Lesson 9
self-supervised learning (SSL): Learning representations with a training signal made from the data itself, without labels. Introduced in: Lesson 14
shared expert: An expert that every token uses, in addition to the routed experts (DeepSeekMoE). Introduced in: Lesson 15
sharpness: The largest eigenvalue $\lambda_{\max}$ of the Hessian of the loss. Gradient descent on a quadratic is stable only if $\eta < 2/\lambda_{\max}$ . Introduced in: Lesson 2
SiLU (Swish): Sigmoid linear unit, $x\,s(x)$ with $s$ the logistic sigmoid. Smooth, non-monotone, and close to ReLU for large $|x|$ . Introduced in: Lesson 4
sinusoidal encoding: Fixed position vectors with $PE(t)_{2i} = \sin(\theta_it)$ and $PE(t)_{2i+1} = \cos(\theta_it)$ , where $\theta_i = 10000^{-2i/d}$ . Introduced in: Lesson 10
softmax: The map from logits $z \in \mathbb{R}^K$ to probabilities, $\operatorname{softmax}(z)_c = e^{z_c}/\sum_j e^{z_j}$ . It does not change when the same number is added to all logits. Introduced in: Lesson 2
spectral bias: Networks trained by gradient descent fit low-frequency components of a function first; fine details are learned slowly. Introduced in: Lesson 1
spectral radius: $\rho(W) = \max_i|\lambda_i(W)|$ , the largest absolute eigenvalue. It sets the long-run growth rate of $W^n$ . Introduced in: Lesson 8
state-space model (SSM): A linear recurrent layer derived from the continuous system $x'(t) = Ax(t) + Bu(t)$ , $y = Cx$ , sampled with a step $\Delta$ . Introduced in: Lesson 16
stochastic depth: During training, randomly skip whole residual branches, so the block computes only $x$ . Introduced in: Lesson 6
stop-gradient: Treat a value as a constant in the backward pass. In BYOL and SimSiam it blocks the gradient into the target branch and prevents collapse. Introduced in: Lesson 14
stride: The step $s$ between neighbouring positions of the filter. The output size along one axis is $\lfloor (n + 2p - d(k-1) - 1)/s \rfloor + 1$ . Introduced in: Lesson 7
symmetry problem: Units with identical initial weights get identical gradients and stay identical, so a layer acts like one unit. Random initialization breaks the symmetry. Introduced in: Lesson 4
teacher forcing: During training, the input at each step is the true previous token, not the model’s own prediction. Introduced in: Lesson 8
temperature (contrastive): The scale $\tau$ that divides the cosine similarities before the softmax. A small $\tau$ focuses the loss on the hardest negatives. Introduced in: Lesson 14
temperature sampling: Sample the next token from $\operatorname{softmax}(z/\tau)$ . $\tau < 1$ sharpens the distribution, $\tau > 1$ flattens it. Introduced in: Lesson 11
tent map: The function $t(x) = 2\operatorname{relu}(x) - 4\operatorname{relu}(x - \tfrac12)$ on $[0, 1]$ . Composing it $k$ times gives $2^k$ linear pieces with only $2k$ units, which shows that depth can be exponentially more efficient than width. Introduced in: Lesson 1
test set: Held-out data used once, at the end, to estimate the population risk. Choices based on it make the estimate optimistic. Introduced in: Lesson 2
transductive and inductive: Transductive: the test nodes are in the graph during training. Inductive: the model must handle new nodes or new graphs at test time. Introduced in: Lesson 13
transformer block: Self-attention across tokens, then an MLP across features, each wrapped in a residual connection and a normalization layer. Introduced in: Lesson 11
translation equivariance: Shifting the input shifts the output by the same amount. A stride-1 convolution has it; striding and pooling break it (aliasing). Introduced in: Lesson 7
transposed convolution: The adjoint of a convolution as a linear map. It appears in the backward pass of a convolution and is used to upsample in decoders. Introduced in: Lesson 7
truncated BPTT: Cut the backward graph every $k$ steps. Memory falls to $O(k\,d_h)$ , but the gradient no longer sees dependencies longer than $k$ steps. Introduced in: Lesson 8
U-Net: An encoder–decoder CNN whose skip connections copy each encoder feature map to the decoder stage of the same resolution. Introduced in: Lesson 7
unembedding: The final matrix $W_U \in \mathbb{R}^{D \times |\mathcal V|}$ that maps each token vector to one logit per vocabulary token. With weight tying, $W_U = E^\top$ . Introduced in: Lesson 11
universal approximation theorem: A network with one hidden layer and a non-polynomial activation can approximate any continuous function on a compact set to any accuracy. It says that good weights exist, not how many units are needed or how to find them. Introduced in: Lesson 1
validation set: Held-out data used to choose hyperparameters and to stop training. It is never used to compute gradients. Introduced in: Lesson 2
vanishing and exploding gradients: The gradient from step $t$ back to step $k$ is a product of $t - k$ Jacobians, so it shrinks or grows exponentially with the distance. Introduced in: Lesson 8
VC dimension: The largest number of points that a class of binary classifiers can label in all possible ways. Introduced in: Lesson 17
vector-Jacobian product (VJP): The product $\bar u^\top\,\partial u/\partial x$ that maps the adjoint of an output to the adjoint of an input. The full Jacobian is never formed. Introduced in: Lesson 3
vision transformer (ViT): A transformer that cuts an image into $P\times P$ patches, embeds each patch as a token, and processes them with standard transformer blocks. Introduced in: Lesson 11
weight decay: Shrink every weight by a factor $1 - \eta\lambda$ at each step. With plain SGD it is the same as L2 regularization; with Adam it is not. Introduced in: Lesson 6
width: The number $n_l$ of units in layer $l$ . Introduced in: Lesson 1
Xavier (Glorot) initialization: Draw weights with variance $2/(n_{\text{in}} + n_{\text{out}})$ , a compromise between keeping the forward and the backward variance constant for tanh-like activations. Introduced in: Lesson 4
z-loss: An extra loss term $c_z(\log Z)^2$ , with $Z = \sum_j e^{s_j}$ , that keeps the log-normalizer of the output softmax near 0. Introduced in: Lesson 12
zero-order hold (ZOH): The discretization $\bar A = e^{\Delta A}$ , $\bar B = A^{-1}(e^{\Delta A} - I)B$ , exact when the input is constant on each step. Introduced in: Lesson 16