Generative Models · 171

Glossary

The technical terms of this track, with short definitions (and the Persian term where available). Each term links to the lesson where it is introduced.

1-Lipschitz: A function with $|f(x) - f(y)| \le \|x - y\|$ for all $x, y$ . For a differentiable $f$ this means $\|\nabla f\| \le 1$ everywhere. Introduced in: Lesson 5
active units: The latent dimensions $j$ with $\operatorname{Var}_x(\mathbb E_q[z_j \mid x]) > 0.01$ . Dimensions that are not active have collapsed. Introduced in: Lesson 2
adaLN-Zero: Conditioning by a shift, a scale and a gate per block, computed from a global vector (timestep and class). The gates start at zero, so every block is the identity at initialization. Introduced in: Lesson 13
affine coupling layer: A flow layer that keeps one half of the coordinates and uses them to scale and shift the other half, $y_B = x_B \odot e^{s(x_A)} + t(x_A)$ . Its Jacobian is triangular, so the log-determinant is $\sum s(x_A)$ . Introduced in: Lesson 4
amortization gap: The part of the ELBO gap caused by the encoder not outputting the best member of the family for a given $x$ , $\mathrm{ELBO}(q^*_x) - \mathrm{ELBO}(q_\phi)$ . Introduced in: Lesson 1
amortized inference: Using one encoder network to output the variational distribution for every data point, instead of optimizing a separate $q$ for each point. Introduced in: Lesson 1
ancestral sampling: Sampling a model in the order of its factors, for example first $z \sim p(z)$ and then $x \sim p_\theta(x \mid z)$ , or $x_1, x_2, \dots$ one after another in an autoregressive model. Introduced in: Lesson 1
approximation gap: The part of the ELBO gap that remains with the best member $q^*_x$ of the chosen family, $\mathrm{KL}(q^*_x \,\|\, p(z \mid x))$ . It is large when the family is too simple. Introduced in: Lesson 1
autoregressive model: A model that learns each conditional $p(x_i \mid x_{<i})$ of the chain rule with a network that sees only the earlier variables. It gives an exact likelihood in one pass and samples in $D$ sequential passes. Introduced in: Lesson 3
autoregressive property: The requirement that the parameters of factor $i$ depend only on $x_{<i}$ . It guarantees that the product of the conditionals is a normalized distribution. Introduced in: Lesson 3
backbone: The denoising network of a diffusion model, usually a U-Net or a transformer. Introduced in: Lesson 13
bits per dimension (bpd): The negative log-likelihood of an image in bits divided by the number of dimensions $D$ . For 8-bit data, a uniform model has 8 bpd. Introduced in: Lesson 14
blind spot: A wedge of earlier pixels to the upper right that a stack of masked $3 \times 3$ convolutions in PixelCNN never sees, although they come before the target in raster order. The gated PixelCNN removes it with two stacks. Introduced in: Lesson 3
calibration: Statistical consistency between forecasts and outcomes. A calibrated ensemble treats the outcome as one more exchangeable member. Introduced in: Lesson 14
causal mask: The attention mask that sets the scores from position $i$ to all later positions $j > i$ to $-\infty$ before the softmax, so position $i$ sees only its prefix. Introduced in: Lesson 3
chain rule of probability: The exact factorization $p(x) = \prod_{i=1}^D p(x_i \mid x_{<i})$ , valid for every distribution and every order of the variables. Introduced in: Lesson 3
classifier guidance: Adding $s\,\nabla_{x_t}\log p_t(y \mid x_t)$ from a separately trained noisy classifier to the unconditional score, with guidance scale $s$ . Introduced in: Lesson 11
classifier-free guidance (CFG): Combining the conditional and unconditional predictions of one network, $\tilde\varepsilon = \hat\varepsilon_u + w(\hat\varepsilon_c - \hat\varepsilon_u)$ . $w = 1$ is the conditional model and $w > 1$ is guidance; it costs 2 NFE per step. Introduced in: Lesson 11
codebook: The finite list of code vectors $e_1, \dots, e_K$ of a VQ-VAE. The codebook loss moves each code toward the encoder outputs assigned to it. Introduced in: Lesson 2
commitment loss: The VQ-VAE term $\beta\,\|z_e - \mathrm{sg}[e_{k^*}]\|^2$ that keeps the encoder output close to its chosen code. Introduced in: Lesson 2
condition dropout: Replacing the condition by a null token with probability $p_{\text{uncond}}$ (often 0.1–0.2) during training, so one network learns the conditional and the unconditional model. Introduced in: Lesson 11
conditional flow matching (CFM) loss: $\mathbb E\,\|v_\theta(x_t, t) - (\varepsilon - x_0)\|^2$ . It equals the flow matching loss plus a constant, so it has the same minimizer, but its minimum is not zero. Introduced in: Lesson 10
conditional path: The path of one pair, $x_t = (1 - t)\,x_0 + t\,\varepsilon$ for the linear path. Its velocity $\varepsilon - x_0$ is the conditional velocity. Introduced in: Lesson 10
consistency distillation (CD): Training a consistency model by comparing its outputs at two adjacent noise levels, where the lower point comes from one ODE solver step of a pretrained teacher. Introduced in: Lesson 12
consistency function: The map $f(x_\sigma, \sigma)$ from any point of a PF-ODE path to the end point of that path. It satisfies self-consistency and the boundary condition $f(x, \sigma_{\min}) = x$ . Introduced in: Lesson 12
consistency model: A network trained to be a consistency function, so it generates a sample in one step (or a few steps with re-noising). Introduced in: Lesson 12
consistency training (CT): Training a consistency model without a teacher, using $x_0 + \sigma_n z$ and $x_0 + \sigma_{n+1} z$ with the same noise $z$ as the pair of adjacent points. Introduced in: Lesson 12
continuity equation: $\partial_t p_t + \nabla \cdot (p_t f) = 0$ . It says that probability mass moves with the vector field $f$ and is conserved. Introduced in: Lesson 4
continuous normalizing flow (CNF): A flow defined by an ODE $dx/dt = f(x, t)$ . The log-density changes along a path by minus the divergence of $f$ , $\frac{d}{dt}\log p_t(x(t)) = -\nabla \cdot f$ . Introduced in: Lesson 4
continuous ranked probability score (CRPS): $\mathrm{CRPS}(F, y) = \int (F(x) - \mathbb 1\{x \ge y\})^2\,dx = \mathbb E|X - y| - \frac12\mathbb E|X - X'|$ . It is strictly proper, has the units of $y$ , and works with samples. Introduced in: Lesson 14
contrastive divergence (CD- $k$ ): An EBM training method that starts MCMC chains at the data batch and runs $k$ steps to get negative samples. It is cheap but biased. Introduced in: Lesson 6
convex conjugate: $f^*(t) = \sup_u\,[tu - f(u)]$ . It gives the variational lower bound $D_f(P \,\|\, Q) \ge \mathbb E_P[T] - \mathbb E_Q[f^*(T)]$ used by f-GAN. Introduced in: Lesson 5
coupling: A joint distribution of $(x, y)$ whose marginals are $P$ and $Q$ . It is a transport plan that says how much mass moves from each $x$ to each $y$ . Introduced in: Lesson 5
critic: The network $f_w$ of a Wasserstein GAN. It has no sigmoid and is kept (approximately) 1-Lipschitz, so its objective estimates $W_1$ . Introduced in: Lesson 5
cross-attention: Attention in which the queries come from the image (or latent) tokens and the keys and values come from the condition tokens, such as text. Introduced in: Lesson 13
data assimilation: Estimating the state of a dynamical system, such as the atmosphere, from a model and sparse noisy observations over time. Introduced in: Lesson 15
data distribution: The unknown distribution $p_{\text{data}}$ from which the training examples were drawn. Introduced in: Lesson 1
DDIM: A sampler that predicts $\hat x_0$ and puts it back at a lower noise level with the predicted noise. With no fresh noise ( $\eta = 0$ ) it is the Euler step of the PF-ODE in EDM coordinates; with $\eta = 1$ it is DDPM. Introduced in: Lesson 9
decoder: The conditional distribution $p_\theta(x \mid z)$ of a latent variable model, usually given by a neural network. Also called the likelihood. Introduced in: Lesson 1
denoiser: The function $D(x) = \mathbb E[x_0 \mid x]$ that returns the mean clean point given a noisy point. It is the optimal predictor of $x_0$ in squared error. Introduced in: Lesson 6
denoising score matching (DSM): Regressing $s_\theta(x_0 + \sigma\varepsilon)$ on the known target $-\varepsilon/\sigma$ . Its minimizer is the score of the noisy density $p_\sigma$ , and its loss differs from explicit score matching on $p_\sigma$ only by a constant. Introduced in: Lesson 6
diffusion coefficient: The factor $g(t)$ that scales the random part of an SDE. Introduced in: Lesson 9
diffusion model: A generative model that learns to reverse a fixed process that adds Gaussian noise to data step by step. Sampling starts from pure noise and removes noise step by step. Introduced in: Lesson 7
diffusion transformer (DiT): A transformer backbone for latent diffusion that cuts the latent into $p \times p$ patches, one token per patch, and uses adaLN-Zero blocks. Introduced in: Lesson 13
Dirac-GAN: A GAN with data at a point mass at 0, a generator at a point mass at $\theta$ and a linear discriminator. Plain gradient play circles around the equilibrium and does not converge. Introduced in: Lesson 5
distortion: The reconstruction term of the negative ELBO, $-\mathbb E_q[\log p_\theta(x \mid z)]$ . It measures how badly the decoder rebuilds $x$ . Introduced in: Lesson 2
distribution matching distillation (DMD): Training a one-step generator whose KL gradient is the difference of the teacher's score and a score model fitted to the generator's own samples, plus a regression loss. Introduced in: Lesson 12
divergence: The trace of the Jacobian of a vector field, $\nabla \cdot f = \sum_i \partial f_i/\partial x_i$ . It is the rate at which the flow expands volume. Introduced in: Lesson 4
downsampling factor: The integer $f$ with latent size $h = H/f$ , $w = W/f$ . Stable Diffusion uses $f = 8$ . Introduced in: Lesson 13
DPS (diffusion posterior sampling): Zero-shot guidance with the likelihood score $\nabla_{x_t}\log\mathcal N(y;\ \mathcal A(\hat x_0(x_t)),\ \sigma_y^2 I)$ . It works for any differentiable $\mathcal A$ but ignores the uncertainty of $\hat x_0$ . Introduced in: Lesson 15
drift: The deterministic part $f(x, t)$ of an SDE. Introduced in: Lesson 9
EDM coordinates: $\tilde x_t = x_t/a_t$ and $\tilde\sigma_t = \sigma_t/a_t$ , so that $\tilde x_t = x_0 + \tilde\sigma_t\varepsilon$ . In them the PF-ODE is $d\tilde x/d\tilde\sigma = (\tilde x - D)/\tilde\sigma$ . Introduced in: Lesson 9
EDM preconditioning: The form $D_\theta(x; \sigma) = c_{\text{skip}}\,x + c_{\text{out}}\,F_\theta(c_{\text{in}}\,x; c_{\text{noise}})$ of Karras et al. 2022, with coefficients chosen so that the network input and its effective target have unit variance. Introduced in: Lesson 8
ELBO (evidence lower bound): $\mathcal L = \mathbb E_{q(z \mid x)}[\log p(x, z) - \log q(z \mid x)]$ . It satisfies $\log p(x) = \mathcal L + \mathrm{KL}(q(z \mid x) \,\|\, p(z \mid x))$ , so it is a lower bound that is tight when $q$ is the posterior. Introduced in: Lesson 1
EM algorithm: Expectation–maximization. It alternates an E-step that sets $q$ to the exact posterior (gap 0) and an M-step that maximizes the ELBO over $\theta$ . Each iteration does not decrease the log-likelihood. Introduced in: Lesson 1
encoder: A network $q_\phi(z \mid x)$ that outputs the variational distribution for any input $x$ . Introduced in: Lesson 1
energy score: The multivariate version of the CRPS, $\mathbb E\|X - y\| - \frac12\mathbb E\|X - X'\|$ . It is strictly proper but weak at detecting wrong correlations. Introduced in: Lesson 14
energy-based model (EBM): A density $p_\theta(x) = e^{-E_\theta(x)}/Z_\theta$ given by an energy function $E_\theta$ . Low energy means high probability. Introduced in: Lesson 6
Euler–Maruyama: The simplest discretization of an SDE, $X_{k+1} = X_k + h\,b(X_k) + \sqrt{2h}\,\xi_k$ for $dX = b\,dt + \sqrt2\,dW$ . The noise scales with $\sqrt h$ . Introduced in: Lesson 6
evidence: The marginal density $p_\theta(x)$ of an observed point in a latent variable model. Its logarithm is the quantity that the ELBO bounds from below. Introduced in: Lesson 1
exposure bias: The mismatch between training on real prefixes and sampling from the model's own prefixes, which can let errors build up over long sequences. It does not bias the likelihood. Introduced in: Lesson 3
f-divergence: $D_f(P \,\|\, Q) = \mathbb E_Q[f(p/q)]$ for a convex $f$ with $f(1) = 0$ . KL ( $f(u) = u \log u$ ) and the JSD are examples. Introduced in: Lesson 5
FID: Fréchet Inception Distance. The Fréchet distance between Gaussians fitted to Inception-v3 features of real and generated images; lower is better. It is biased upward for a finite sample size. Introduced in: Lesson 14
Fisher divergence: $\frac12\,\mathbb E_{x \sim p}\|s_\theta(x) - \nabla \log p(x)\|^2$ , the mean squared distance between two scores. Introduced in: Lesson 6
flow matching: Learning the velocity field of an ODE that carries noise to data by regression on a probability path. The flow matching loss regresses on the unknown marginal velocity. Introduced in: Lesson 10
forward operator: The map $\mathcal A$ from the unknown $x$ to what the sensors see, for example a blur, a mask or a subsampled Fourier transform. Introduced in: Lesson 15
forward posterior: The Gaussian $q(x_{t-1} \mid x_t, x_0)$ , known in closed form. Each ELBO term of DDPM compares a learned reverse step with it. Introduced in: Lesson 7
forward process: The fixed noising chain $q(x_t \mid x_{t-1}) = \mathcal N(\sqrt{\alpha_t}\,x_{t-1}, \beta_t I)$ of DDPM. Its marginal is $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\varepsilon$ . Introduced in: Lesson 7
free bits: Replacing $\sum_j \mathrm{KL}_j$ by $\sum_j \max(\lambda, \mathrm{KL}_j)$ , so the first $\lambda$ nats per latent dimension cost nothing. It is a fix for posterior collapse. Introduced in: Lesson 2
Fréchet distance: The squared 2-Wasserstein distance between two Gaussians, $\|\mu_r - \mu_g\|^2 + \operatorname{tr}(\Sigma_r + \Sigma_g - 2(\Sigma_r^{1/2}\Sigma_g\Sigma_r^{1/2})^{1/2})$ . Introduced in: Lesson 14
generative adversarial network (GAN): A generator $G$ that maps noise $z$ to samples and a discriminator $D$ that classifies samples as real or fake, trained as a two-player game. Introduced in: Lesson 5
generative model: A model of a probability distribution $p_\theta(x)$ that should be close to the data distribution. It is used to draw new samples, to score inputs by their density, or to condition on part of the input. Introduced in: Lesson 1
Glow: A flow for images built from actnorm, invertible $1 \times 1$ convolutions and affine coupling layers. Introduced in: Lesson 4
gradient penalty: The WGAN-GP term $\lambda\,\mathbb E(\|\nabla_x f(\hat x)\|_2 - 1)^2$ at random interpolates $\hat x$ of real and fake points. It softly enforces the 1-Lipschitz condition. Introduced in: Lesson 5
guidance: Changing the score (or noise prediction) at sampling time to make samples more typical of a condition $y$ , at the cost of diversity. Introduced in: Lesson 11
Heun's method: A second-order ODE solver: an Euler predictor step, then a trapezoidal corrector that averages the slopes at both ends. It costs 2 NFE per step. Introduced in: Lesson 9
Hutchinson's estimator: An unbiased estimate of a trace, $\operatorname{tr}(J) = \mathbb E[v^\top J v]$ for random $v$ with $\mathbb E[vv^\top] = I$ . FFJORD uses it to estimate the divergence with one vector–Jacobian product per probe. Introduced in: Lesson 4
IAF (inverse autoregressive flow): An autoregressive flow in the sampling direction, $x_i = u_i e^{\alpha_i(u_{<i})} + \mu_i(u_{<i})$ . Sampling takes one pass, but the density of an outside point takes $D$ passes. Introduced in: Lesson 4
ill-posed: A problem in which many $x$ fit the data equally well, or the solution changes a lot with small changes in $y$ . The prior decides which solutions are plausible. Introduced in: Lesson 15
implicit classifier gradient: $\nabla \log p_t(x \mid y) - \nabla \log p_t(x) = \nabla \log p_t(y \mid x)$ , read off from the difference of the conditional and unconditional predictions. Introduced in: Lesson 11
importance weight: The ratio $w(z) = p(x, z)/q(z \mid x)$ . Its mean under $q$ is $p(x)$ . Introduced in: Lesson 1
inverse problem: Recovering an unknown $x$ from measurements $y = \mathcal A(x) + n$ . The answer is the posterior $p(x \mid y) \propto p(y \mid x)\,p(x)$ . Introduced in: Lesson 15
IWAE bound: The importance-weighted bound $\mathcal L_K = \mathbb E[\log \frac1K\sum_{k=1}^K w_k]$ with $K$ samples from $q$ . $\mathcal L_1$ is the ELBO, and $\mathcal L_K$ grows with $K$ toward $\log p(x)$ . Introduced in: Lesson 1
Jensen–Shannon divergence (JSD): $\mathrm{JSD}(p \,\|\, q) = \frac12\mathrm{KL}(p \,\|\, m) + \frac12\mathrm{KL}(q \,\|\, m)$ with $m = \frac12(p + q)$ . It is symmetric and lies between $0$ and $\log 2$ ; it is $\log 2$ whenever the supports do not overlap. Introduced in: Lesson 5
joint attention: Attention over the concatenation of image and text tokens, so both streams read each other. MM-DiT (Stable Diffusion 3) uses it with separate weights per stream. Introduced in: Lesson 13
Kantorovich–Rubinstein duality: $W_1(P, Q) = \sup_{\|f\|_L \le 1} \mathbb E_P[f] - \mathbb E_Q[f]$ . It turns the transport problem into a maximization over 1-Lipschitz functions. Introduced in: Lesson 5
KID: Kernel Inception Distance. An unbiased estimate of MMD$^2$ on Inception features with the cubic kernel $k(a, b) = (a^\top b/d + 1)^3$ . Introduced in: Lesson 14
KL annealing: Raising the weight of the KL term from 0 to 1 during training, to reduce posterior collapse. Introduced in: Lesson 2
KV cache: The stored keys and values of all earlier positions in a causal transformer. With it, each new sampling step computes attention only for the new position. Introduced in: Lesson 3
Langevin dynamics: The SDE $dX_t = \nabla \log p(X_t)\,dt + \sqrt2\,dW_t$ , which has $p$ as a stationary density. It is noisy gradient ascent on $\log p$ . Introduced in: Lesson 6
latent diffusion: A two-stage design: train an autoencoder that compresses images into a smaller latent, then train the diffusion model on the latents and decode once at the end. Introduced in: Lesson 13
latent variable model: A model with a hidden variable $z$ , a simple prior $p(z)$ and a simple likelihood (decoder) $p_\theta(x \mid z)$ . Its density is the marginal $p_\theta(x) = \int p(z)\, p_\theta(x \mid z)\, dz$ . Introduced in: Lesson 1
log score: The scoring rule $-\log f(y)$ . It is strictly proper but needs a density. Introduced in: Lesson 14
MADE: Masked autoencoder for distribution estimation. An MLP whose weight matrices are multiplied by binary masks built from unit degrees, so output $i$ depends only on inputs $1, \dots, i-1$ . Introduced in: Lesson 3
MAF (masked autoregressive flow): An autoregressive flow in the density direction, $u_i = (x_i - \mu_i(x_{<i}))\,e^{-\alpha_i(x_{<i})}$ . The density takes one pass; sampling takes $D$ passes. Introduced in: Lesson 4
marginal velocity: $v(x, t) = \mathbb E[\varepsilon - x_0 \mid x_t = x]$ , the average of the conditional velocities through $x$ at time $t$ . It generates the marginal path. Introduced in: Lesson 10
maximum likelihood estimation (MLE): Choosing the parameters that maximize the average log-density of the data, $\frac1n\sum_i \log p_\theta(x_i)$ . It is the same as minimizing the forward KL divergence $\mathrm{KL}(p_{\text{data}} \,\|\, p_\theta)$ . Introduced in: Lesson 1
maximum mean discrepancy (MMD): $\mathrm{MMD}^2 = \mathbb E\,k(X, X') + \mathbb E\,k(Y, Y') - 2\,\mathbb E\,k(X, Y)$ , the squared distance between the mean feature vectors of two distributions in the feature space of a kernel $k$ . Introduced in: Lesson 14
Metropolis-adjusted Langevin algorithm (MALA): ULA used as a proposal, followed by a Metropolis–Hastings accept/reject step. The accept step removes the bias of ULA. Introduced in: Lesson 6
Min-SNR- $\gamma$: A loss weight that caps the weight in $x_0$ units at $\min(\mathrm{SNR}, \gamma)$ , usually with $\gamma = 5$ . It lowers the weight of the easy low-noise levels. Introduced in: Lesson 8
minibatch optimal transport (OT): Re-pairing the data and noise points of each training batch by an exact assignment before flow matching, so the pairs cross less. Introduced in: Lesson 10
mode collapse: A GAN failure in which the generator maps many values of $z$ to one or a few modes of the data, so samples look good but have low diversity. Introduced in: Lesson 5
mode covering: The behavior of a model fitted by the forward KL $\mathrm{KL}(p \,\|\, q)$ . It must put mass on every region where the data has mass, so it spreads over all modes. Also called mass covering. Introduced in: Lesson 1
mode seeking: The behavior of a model fitted by the reverse KL $\mathrm{KL}(q \,\|\, p)$ . Mass where $p$ is tiny is expensive and missing regions cost nothing, so $q$ tends to sit inside one mode. Introduced in: Lesson 1
moment matching: The result that the Gaussian with the smallest forward KL to $p$ (the MLE Gaussian) has the same mean and covariance as $p$ . More generally, MLE in an exponential family matches the expected sufficient statistics. Introduced in: Lesson 1
network function evaluations (NFE): The number of network calls a sampler makes. Samplers are compared at equal NFE, because the network dominates the cost. Introduced in: Lesson 9
noise schedule: The sequence $\beta_1, \dots, \beta_T$ of per-step noise variances, with $\alpha_t = 1 - \beta_t$ and $\bar\alpha_t = \prod_{s \le t}\alpha_s$ . Common choices are the linear and the cosine schedule. Introduced in: Lesson 7
noise-contrastive estimation: Training an unnormalized model by a logistic classifier that separates data from samples of a known noise distribution; $\log Z$ is learned as one more parameter. Introduced in: Lesson 6
noise-prediction network: A network $\varepsilon_\theta(x_t, t)$ trained to predict the noise $\varepsilon$ that was added to $x_0$ . DDPM trains it with the simple loss $\|\varepsilon - \varepsilon_\theta(x_t, t)\|^2$ . Introduced in: Lesson 7
noisy classifier: The classifier $p_t(y \mid x_t)$ on noisy inputs. It is the clean classifier averaged over the denoising posterior $p(x_0 \mid x_t)$ . Introduced in: Lesson 11
non-saturating loss: The generator loss $-\log D(G(z))$ . It gives large gradients when the discriminator confidently rejects the fakes, unlike the original (saturating) loss $\log(1 - D(G(z)))$ . Introduced in: Lesson 5
normalizing flow: An invertible map $f$ from data to a simple base distribution. Its density follows from the change of variables, $\log p(x) = \log p_z(f(x)) + \log|\det \partial f/\partial x|$ . Introduced in: Lesson 4
null token: A special "no condition" input $\varnothing$ that gives the unconditional prediction of a conditional network. Introduced in: Lesson 11
order of a solver: The power $p$ in "global error $\approx C h^p$ ". Euler has order 1 and Heun has order 2. Introduced in: Lesson 9
partition function: The normalizing constant $Z_\theta = \int e^{-E_\theta(x)}\,dx$ of an energy-based model. In high dimension it cannot be computed. Introduced in: Lesson 6
patchify: Cutting a latent of side $I$ into $p \times p$ patches and embedding each one linearly, which gives $T = (I/p)^2$ tokens. Introduced in: Lesson 13
pathwise gradient estimator: A gradient estimator that differentiates through the sample using the reparameterization trick. It usually has a much lower variance than the score-function estimator. Introduced in: Lesson 2
perplexity: $\exp$ of the mean negative log-likelihood per token in nats, $e^{\bar\ell}$ . A uniform model over $K$ tokens has perplexity $K$ . Introduced in: Lesson 3
positive and negative phase: The two parts of the EBM log-likelihood gradient. The positive phase lowers the energy at the data points; the negative phase raises the energy at samples from the model. Introduced in: Lesson 6
posterior collapse: A failure of VAE training in which $q_\phi(z \mid x) \approx p(z)$ for all $x$ (in all or some latent dimensions), so $z$ carries no information and the decoder ignores it. Introduced in: Lesson 2
precision and recall (generative): Precision is the share of generated samples that lie in the support of the real data (fidelity). Recall is the share of real samples that lie in the support of the model (diversity). Introduced in: Lesson 14
prior: The distribution $p(z)$ of the latent variable before seeing $x$ , usually $\mathcal N(0, I)$ . Introduced in: Lesson 1
probability integral transform (PIT): The value $u = F(x^*)$ of the true value under the claimed distribution function. It is uniform on $[0, 1]$ when the distribution is calibrated. Introduced in: Lesson 15
probability path: A family of densities $p_t$ , $t \in [0, 1]$ , from the data ( $p_0 = p_{\text{data}}$ ) to the noise ( $p_1 = \mathcal N(0, I)$ ). Introduced in: Lesson 10
probability-flow ODE (PF-ODE): The deterministic ODE $dx/dt = f(x, t) - \frac12 g(t)^2\,\nabla_x \log p_t(x)$ . It has the same marginals as the forward SDE and maps noise to data by a deterministic, invertible map. Introduced in: Lesson 9
progressive distillation: Training a student whose one sampler step equals two steps of the teacher, then repeating with the student as the new teacher, to halve the number of steps each round. Introduced in: Lesson 12
proper scoring rule: A scoring rule whose expected value under the truth $G$ is smallest at $F = G$ . It is strictly proper if $F = G$ is the only minimizer. Introduced in: Lesson 14
R1 penalty: The term $\frac\gamma2\,\mathbb E_{p_{\text{data}}}\|\nabla_x l(x)\|^2$ added to the discriminator loss. It makes the GAN game converge locally, for example in the Dirac-GAN. Introduced in: Lesson 5
rank histogram: A histogram of the rank of the outcome among the $M$ ensemble members over many cases. It is flat for a calibrated ensemble, U-shaped when the ensemble is too narrow and dome-shaped when it is too wide. Introduced in: Lesson 14
raster order: The pixel order of PixelCNN, row by row and left to right within each row. Introduced in: Lesson 3
rate: The KL term of the negative ELBO, $\mathrm{KL}(q_\phi(z \mid x) \,\|\, p(z))$ . It is the price, in nats, of storing information about $x$ in $z$ . Introduced in: Lesson 2
rectified flow: The flow learned with the linear path. Reflow trains it again on the pairs (noise, generated sample) of the trained flow, which makes its paths straighter. Introduced in: Lesson 10
reflow: Training a new flow-matching model on the deterministic pairs $(Z_0, Z_1)$ produced by a trained flow. The lines cross less, so the new paths are straighter and need fewer steps. Introduced in: Lesson 10
reparameterization trick: Writing a sample as a differentiable function of the parameters and independent noise, for example $z = \mu + \sigma \odot \varepsilon$ with $\varepsilon \sim \mathcal N(0, I)$ , so that gradients flow through $z$ into the encoder. Introduced in: Lesson 2
reverse process: The learned Markov chain $p_\theta(x_{t-1} \mid x_t)$ that runs from $x_T \sim \mathcal N(0, I)$ back to data. Introduced in: Lesson 7
reverse-time SDE: $dx = [f(x, t) - g(t)^2\,\nabla_x \log p_t(x)]\,dt + g(t)\,d\bar w$ , solved from noise to data. It has the same marginals $p_t$ as the forward SDE. Introduced in: Lesson 9
scale factor: The constant that rescales the latents to about unit variance before diffusion, for example $0.18215$ for Stable Diffusion v1. Introduced in: Lesson 13
scheduled sampling: A fix for exposure bias that sometimes feeds the model's own predictions as the prefix during training. Introduced in: Lesson 3
score: The gradient of the log-density, $s(x) = \nabla_x \log p(x)$ . It does not depend on the normalizing constant; for an EBM it is $-\nabla_x E_\theta(x)$ . Introduced in: Lesson 6
score matching: Fitting a model score $s_\theta$ to the score of the data. Explicit score matching minimizes the Fisher divergence; implicit and denoising score matching remove the unknown target. Introduced in: Lesson 6
score-based data assimilation (SDA): A diffusion prior over whole trajectories, built from local scores on short windows, combined with observation guidance and predictor–corrector sampling. Introduced in: Lesson 15
score-function estimator: The gradient estimator $\mathbb E_q[f(z)\,\nabla_\phi \log q_\phi(z)]$ (REINFORCE). It needs only values of $f$ , so it works for discrete $z$ , but its variance is high. Introduced in: Lesson 2
scoring rule: A number $S(F, y)$ that rates a forecast distribution $F$ after the outcome $y$ is seen. In this track lower is better. Introduced in: Lesson 14
SDEdit: Noising a guess to a level $t_0$ and running the unconditional reverse process. It uses no likelihood; $t_0$ trades faithfulness against realism. Introduced in: Lesson 15
signal scale and noise scale: The numbers $a_t$ and $\sigma_t$ in $x_t = a_t x_0 + \sigma_t\varepsilon$ . In DDPM, $a_t = \sqrt{\bar\alpha_t}$ and $\sigma_t = \sqrt{1 - \bar\alpha_t}$ . Introduced in: Lesson 8
signal-to-noise ratio (SNR): $\mathrm{SNR}(t) = a_t^2/\sigma_t^2$ . It decreases as $t$ goes from data to noise. Its logarithm $\lambda_t$ is the log-SNR. Introduced in: Lesson 8
simulation-based calibration (SBC): A test of a posterior sampler. Draw $x^*$ from the prior, simulate $y$ , draw posterior samples, and record the rank of $x^*$ . For an exact sampler the ranks are uniform. Introduced in: Lesson 15
spectral normalization: Dividing each weight matrix by its largest singular value (estimated by power iteration), which bounds the Lipschitz constant of the network by 1. Introduced in: Lesson 5
spread–skill ratio: The ensemble spread divided by the RMSE of the ensemble mean. With the correction factor $\sqrt{(M+1)/M}$ it is about 1 for a calibrated ensemble. Introduced in: Lesson 14
step schedule: The noise levels at which a sampler calls the network, for example the Karras grid with parameter $\rho$ . Introduced in: Lesson 9
stochastic differential equation (SDE): An equation $dx = f(x, t)\,dt + g(t)\,dw$ for a random path, with drift $f$ , diffusion coefficient $g$ and Brownian motion $w$ . Introduced in: Lesson 9
straightness: $S = \int_0^1 \mathbb E\|(Z_1 - Z_0) - v(Z_t, t)\|^2\,dt$ , which measures how much the velocity changes along the flow's paths. $S = 0$ means straight paths, where one Euler step is exact. Introduced in: Lesson 10
target network: A copy of the network with parameters $\theta^-$ , a stop-gradient moving average of $\theta$ , used to compute the training target. Introduced in: Lesson 12
teacher forcing: Training an autoregressive model with the true prefix from the data as input, not the model's own samples. Introduced in: Lesson 3
tilted target: The density $\propto p(x)^{1-w}\,p(x \mid y)^w$ . CFG does not sample it exactly for $w \ne 0, 1$ , because the guided scores do not belong to one diffusion path. Introduced in: Lesson 11
train–test mismatch (zero terminal SNR): When the last noise level still has $\mathrm{SNR} > 0$ , training inputs at $t = T$ contain some signal, but sampling starts from pure noise. The fix is a schedule that reaches SNR 0. Introduced in: Lesson 8
Tweedie's formula: For $x = x_0 + \sigma\varepsilon$ , $\mathbb E[x_0 \mid x] = x + \sigma^2\,\nabla \log p_\sigma(x)$ . It links the best denoiser to the score of the noisy density. Introduced in: Lesson 6
unadjusted Langevin algorithm (ULA): The Euler–Maruyama step of the Langevin SDE with a fixed step $h$ , $x_{k+1} = x_k + h\,\nabla \log p(x_k) + \sqrt{2h}\,\xi_k$ . Its stationary distribution has an error of order $h$ . Introduced in: Lesson 6
uniform dequantization: Training a continuous density on $x + u$ with $u \sim \mathcal U[0, 1)^D$ for integer data $x$ . The continuous log-likelihood is then a lower bound on the log-probability of the discrete model. Introduced in: Lesson 14
value function: The GAN objective $V(D, G) = \mathbb E_{p_{\text{data}}}[\log D(x)] + \mathbb E_z[\log(1 - D(G(z)))]$ , minimized by $G$ and maximized by $D$ . Introduced in: Lesson 5
variance exploding (VE): A noising process with $a_t = 1$ , $x_t = x_0 + \sigma_t\varepsilon$ , where $\sigma_t$ grows to a large maximum. Introduced in: Lesson 8
variance preserving: A noising process with $a_t^2 + \sigma_t^2 = 1$ , which keeps the variance at 1 for data of unit variance. DDPM is variance preserving. Introduced in: Lesson 7
variational autoencoder (VAE): A latent variable model with a neural decoder $p_\theta(x \mid z)$ and a neural encoder $q_\phi(z \mid x)$ , trained together by stochastic gradient ascent on the ELBO. Introduced in: Lesson 2
variational distribution: A chosen distribution $q(z \mid x)$ over the latent that approximates the posterior $p(z \mid x)$ . Introduced in: Lesson 1
variogram score: A scoring rule that compares the expected differences $\mathbb E_F|X_i - X_j|^p$ between pairs of sites with the observed ones. It is sensitive to the correlation structure. Introduced in: Lesson 14
velocity ( $v$ -prediction): The target $v = a_t\varepsilon - \sigma_t x_0$ . In the VP case it is the rate of change of $x_t$ along the angle $\phi$ with $a_t = \cos\phi$ , $\sigma_t = \sin\phi$ . Introduced in: Lesson 8
VQ-VAE: An autoencoder whose latent is the nearest vector of a finite codebook. The gradient passes the quantization with a straight-through copy; a prior over the code indices is fitted afterwards. Introduced in: Lesson 2
Wasserstein-1 distance: $W_1(P, Q) = \inf_\pi \mathbb E_\pi\|x - y\|$ over all couplings $\pi$ of $P$ and $Q$ , the least work to move one distribution onto the other. It grows smoothly with the distance between non-overlapping distributions. Introduced in: Lesson 5
weight function: The function $w(\lambda) \ge 0$ that says how much a diffusion loss counts each log-SNR level. The noise schedule used for training and the prediction target both change it. Introduced in: Lesson 8
zero-shot guidance: Solving inverse problems with one unconditional diffusion prior and adding an approximate likelihood score at sampling time, so the same prior serves every $\mathcal A$ and $\sigma_y$ . Introduced in: Lesson 15
β-VAE: A VAE trained on distortion $+,\beta\cdot$rate. $\beta > 1$ makes information in $z$ more expensive, and $\beta < 1$ makes it cheaper. Introduced in: Lesson 2