Denoising diffusion (DDPM): forward process, forward posterior, ELBO and the noise-prediction loss

Before this lesson: lesson 01 (the ELBO), lesson 02 (the KL divergence between two Gaussians) and lesson 06 (denoising score matching). From other tracks: the multivariate Gaussian (sums of independent Gaussians, conditioning). Time: about 95 minutes, plus the notebook.

---

Why this matters #

DDPM is the starting point of modern diffusion models: Stable Diffusion, Imagen and DALL·E 2 for images, diffusion models for video and audio, GenCast for ensemble weather forecasts, and latent diffusion emulators of seismic ground motion. The loss of this lesson ("predict the noise") is still the most common training objective. The next lessons change one piece of what is built here: the prediction target and the loss weights (lesson 08), the sampler (lesson 09), the conditioning (lesson 11) and the number of steps (lesson 12). "Derive the DDPM loss" is also a standard interview question.

DDPM is a latent variable model with a fixed encoder, so it connects back to lesson 01: the training loss is the ELBO. It is also a score model: predicting the noise is denoising score matching at many noise levels, as in lesson 06.

1. Predict first #

2. The picture #

One coordinate, one data point. Take $x_0 = 2$. One forward step multiplies by $\sqrt{1 - \beta}$ (a number slightly below 1) and adds Gaussian noise of variance $\beta$:

The mean shrinks geometrically toward 0, and the noise accumulates. If $x_{t-1}$ has variance 1, then $x_t$ has variance $(1 - \beta_t) \cdot 1 + \beta_t = 1$: the process is variance preserving (for data of unit variance). After many steps, $x_T$ is close to $\mathcal N(0, 1)$ and has forgotten $x_0$.

In 2D. The figure applies the forward process to the track's 8 Gaussians. The heat map is the exact density of $x_t$; the dots are data points pushed to step $t$.

Interactive figure: loading…

The generative model runs the chain backward. Start from $x_T \sim \mathcal N(0, I)$ and apply learned Gaussian steps $p_\theta(x_{t-1} \mid x_t)$ down to $x_0$. A Gaussian reverse step is a good model only because each forward step is small. For a small step, the true reverse conditional $q(x_{t-1} \mid x_t)$ is close to Gaussian. For a large step it can have several modes, and no Gaussian can fit it.

Interactive figure: loading…

It is a latent variable model. The latents are $x_1, \dots, x_T$, each with the same dimension as the data. The "encoder" $q(x_{1:T} \mid x_0)$ is the fixed forward process and has no parameters. Only the "decoder" chain $p_\theta$ is learned. So DDPM is a hierarchical VAE with a fixed encoder, and the ELBO of lesson 01 applies without change.

The key trick. The reverse conditional $q(x_{t-1} \mid x_t)$ depends on the unknown data distribution. But once we also condition on $x_0$, the forward posterior $q(x_{t-1} \mid x_t, x_0)$ is a Gaussian with a closed form (the blue curves above). The ELBO compares each learned step with this known Gaussian: one KL divergence per step.

3. Definitions and main results #

Notation and schedules #

$x_0 \in \R^d$ is a data point. The noise schedule is $\beta_1, \dots, \beta_T \in (0, 1)$. Write

Always $\varepsilon \sim \mathcal N(0, I)$.

• Forward process: $q(x_t \mid x_{t-1}) = \mathcal N(\sqrt{\alpha_t}\,x_{t-1}, \beta_t I)$ and $q(x_{1:T} \mid x_0) = \prod_{t=1}^T q(x_t \mid x_{t-1})$, a Markov chain.
• Reverse model: $p(x_T) = \mathcal N(0, I)$ and $p_\theta(x_{t-1} \mid x_t) = \mathcal N(\mu_\theta(x_t, t), \sigma_t^2 I)$, with a network $\mu_\theta$ and fixed variances $\sigma_t^2$. The joint is $p_\theta(x_{0:T}) = p(x_T) \prod_{t=1}^T p_\theta(x_{t-1} \mid x_t)$.
• Linear schedule (Ho et al.): $\beta_t$ evenly spaced from $\beta_1 = 10^{-4}$ to $\beta_T = 0.02$, with $T = 1000$. This gives $\bar\alpha_T \approx 4.04 \times 10^{-5}$.
• Cosine schedule (Nichol and Dhariwal 2021): $\bar\alpha_t = f(t)/f(0)$ with $f(t) = \cos^2\!\big(\tfrac{t/T + s}{1 + s}\cdot\tfrac{\pi}{2}\big)$, $s = 0.008$, and each $\beta_t$ capped at $0.999$.
• Signal-to-noise ratio: $\mathrm{SNR}(t) = \bar\alpha_t / (1 - \bar\alpha_t)$. It decreases with $t$.

Interactive figure: loading…

Theorem 1: the closed-form marginal #

Why this is true. Each step scales the old noise and adds new independent noise. A sum of independent Gaussians is Gaussian, and the variances add up.

Proof (induction on $t$).

1. For $t = 1$: $\bar\alpha_1 = \alpha_1$ and $1 - \bar\alpha_1 = \beta_1$, so the claim is the definition of the first step.
2. Assume $x_{t-1} = \sqrt{\bar\alpha_{t-1}}\,x_0 + \sqrt{1 - \bar\alpha_{t-1}}\,\varepsilon'$ with $\varepsilon' \sim \mathcal N(0, I)$.
3. Take one more step with fresh noise $\varepsilon'' \sim \mathcal N(0, I)$, independent of $\varepsilon'$, and use $\sqrt{\alpha_t}\sqrt{\bar\alpha_{t-1}} = \sqrt{\bar\alpha_t}$:

4. The bracket is a sum of independent zero-mean Gaussians. Its covariance is $c\,I$ with $c = \alpha_t(1 - \bar\alpha_{t-1}) + \beta_t$. Expand: $c = \alpha_t - \bar\alpha_t + 1 - \alpha_t = 1 - \bar\alpha_t$. $\blacksquare$

So training never runs the chain: draw $t$ and $\varepsilon$, and compute $x_t$ in one line.

Theorem 2: the forward posterior #

For $t \ge 2$, $q(x_{t-1} \mid x_t, x_0) = \mathcal N\big(\tilde\mu_t(x_t, x_0), \tilde\beta_t I\big)$ with

Why this is true. Given $x_0$, the pair $(x_{t-1}, x_t)$ is jointly Gaussian: $x_{t-1}$ comes from Theorem 1, and $x_t$ is a linear function of $x_{t-1}$ plus Gaussian noise. Conditioning a Gaussian on a linear observation gives a Gaussian. The mean is a weighted average of "where $x_0$ says $x_{t-1}$ should be" and "where $x_t$ says it should be".

Proof (completing the square). It is enough to work with one coordinate: all coordinates have the same coefficients and are independent.

1. Bayes' rule and the Markov property, $q(x_t \mid x_{t-1}, x_0) = q(x_t \mid x_{t-1})$, give, as a function of $x_{t-1}$,

2. Write $y = x_{t-1}$. Up to a constant that does not depend on $y$, the log of the right side is the quadratic function of $y$ below. So the posterior is Gaussian:

3. Expand and collect the $y^2$ terms. The coefficient of $-\tfrac12 y^2$ is the precision (inverse variance). With the identity of Theorem 1, step 4, it is

4. The variance is the inverse of the precision, which is $\tilde\beta_t$. Now collect the terms linear in $y$: the coefficient of $y$ is

5. For a Gaussian, $\log q = -\tfrac12 \lambda y^2 + h y + \text{const}$ with precision $\lambda$ has mean $h/\lambda$. So the mean is $\tilde\beta_t\,h$:

At $t = 1$, $\bar\alpha_0 = 1$ gives $\tilde\beta_1 = 0$: given $x_0$, the last step is a point mass at $x_0$.

The ε-form of the mean. Theorem 1 gives $x_0 = (x_t - \sqrt{1 - \bar\alpha_t}\,\varepsilon)/\sqrt{\bar\alpha_t}$. Substitute this into $\tilde\mu_t$:

1. The $x_0$ coefficient becomes $\frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{(1 - \bar\alpha_t)\sqrt{\bar\alpha_t}} = \frac{\beta_t}{(1 - \bar\alpha_t)\sqrt{\alpha_t}}$.
2. The total coefficient of $x_t$ is $\frac{\beta_t + \alpha_t(1 - \bar\alpha_{t-1})}{(1 - \bar\alpha_t)\sqrt{\alpha_t}} = \frac{1 - \bar\alpha_t}{(1 - \bar\alpha_t)\sqrt{\alpha_t}} = \frac{1}{\sqrt{\alpha_t}}$.
3. The coefficient of $\varepsilon$ is $-\frac{\beta_t}{(1 - \bar\alpha_t)\sqrt{\alpha_t}}\sqrt{1 - \bar\alpha_t} = -\frac{\beta_t}{\sqrt{\alpha_t}\sqrt{1 - \bar\alpha_t}}$. So

Theorem 3: the ELBO as a sum of KL terms #

Proof.

1. The ELBO with latents $x_{1:T}$ (lesson 01): $\log p_\theta(x_0) \ge \mathbb E_q \log \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}$, with $q = q(x_{1:T} \mid x_0)$.
2. For $t \ge 2$, rewrite each forward factor with Bayes' rule, conditioned on $x_0$:

3. The ratios telescope: $\prod_{t=2}^T \frac{q(x_t \mid x_0)}{q(x_{t-1} \mid x_0)} = \frac{q(x_T \mid x_0)}{q(x_1 \mid x_0)}$. The denominator cancels the first factor $q(x_1 \mid x_0)$ of the chain, so

4. Insert this and $p_\theta(x_{0:T}) = p(x_T)\prod_t p_\theta(x_{t-1} \mid x_t)$ into step 1, and split the log:

5. Take $-\mathbb E_q$. Each of the first two groups is the expectation of $\log(q/p)$ over one variable, averaged over the others: a KL divergence. The last group is $L_0$. $\blacksquare$

Why this form. Every KL is between two Gaussians, so it has a closed form. This has much lower variance than a Monte Carlo estimate over the whole chain. $L_T$ has no parameters; for the linear schedule and $x_0 = 1$ it is $2.0 \times 10^{-5}$ nats. For images scaled to $[-1, 1]$, Ho et al. model $L_0$ with a discrete decoder over the 256 pixel values.

Theorem 4: each $L_{t-1}$ is a weighted noise-prediction error #

Parameterize the reverse mean like the ε-form of $\tilde\mu_t$, with a noise-prediction network $\varepsilon_\theta(x_t, t)$:

Then, for $t \ge 2$, with $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\varepsilon$ and a constant $C_t$ that does not depend on $\theta$,

Proof.

1. Use the KL divergence between two Gaussians in $\R^d$ with covariances $s_1^2 I$ and $s_2^2 I$ (lesson 02). Here $s_1^2 = \tilde\beta_t$ and $s_2^2 = \sigma_t^2$ are fixed, so the first part below is the constant $C_t$:

2. Write both means in ε-form. They differ only in the noise term: $\tilde\mu_t - \mu_\theta = \frac{\beta_t}{\sqrt{\alpha_t}\sqrt{1 - \bar\alpha_t}}\,(\varepsilon_\theta - \varepsilon)$.
3. Square and divide by $2\sigma_t^2$: $\frac{\beta_t^2}{2\sigma_t^2\alpha_t(1 - \bar\alpha_t)}\|\varepsilon - \varepsilon_\theta\|^2$. $\blacksquare$

So the network only has to predict the noise. This is denoising score matching at noise level $t$ (lesson 06): the score of the noised data is $\nabla \log p_t(x_t) \approx -\varepsilon_\theta(x_t, t)/\sqrt{1 - \bar\alpha_t}$. The weight depends on $\sigma_t^2$:

• with $\sigma_t^2 = \beta_t$:

• with $\sigma_t^2 = \tilde\beta_t$:

For the linear schedule, $w_2 = 0.27$ (with $\beta_t$) or $0.60$ (with $\tilde\beta_t$), while $w_t$ lies between about $0.005$ and $0.010$ for every $t \ge 100$. So the ELBO puts about 27 to 120 times more weight on $t = 2$ than on any $t \ge 100$.

$L_{\text{simple}}$: drop the weights #

• The $t \ge 2$ terms are the $L_{t-1}$ with weight 1 instead of $w_t$. The $t = 1$ term approximates $L_0$. $L_T$ is dropped because it has no parameters.
• Relative to the ELBO, $L_{\text{simple}}$ lowers the weight of small $t$. Those terms teach the network to remove tiny amounts of noise: easy, and almost invisible in samples. The effort moves to larger $t$, where the network must create structure.
• Result (Ho et al., CIFAR-10, Table 1): training on the true bound $L$ gives FID 13.51 and $\le 3.70$ bits/dim; $L_{\text{simple}}$ gives FID 3.17 and $\le 3.75$ bits/dim. Much better samples, slightly worse likelihood. (FID: lower is better; lesson 14.)
• Equal weights in ε also give targets of unit scale at every $t$, which is easy for a network. Lesson 08 writes every such choice as a weight over the log-SNR.

Training and ancestral sampling (Algorithms 1 and 2 of Ho et al.) #

Training. Repeat:

1. draw $x_0$ from the data;
2. draw $t \sim \mathcal U\{1, \dots, T\}$;
3. draw $\varepsilon \sim \mathcal N(0, I)$;
4. take a gradient step on $\|\varepsilon - \varepsilon_\theta(\sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\varepsilon, t)\|^2$.

Sampling. Draw $x_T \sim \mathcal N(0, I)$. For $t = T, \dots, 1$: draw $z \sim \mathcal N(0, I)$ if $t > 1$, else set $z = 0$; then

Return $x_0$. This costs $T$ network evaluations (1000) per sample; lesson 09 reduces this.

Another view of the same step. Let $\hat x_0 = (x_t - \sqrt{1 - \bar\alpha_t}\,\varepsilon_\theta)/\sqrt{\bar\alpha_t}$ be the clean sample implied by the noise prediction. Then the step mean equals $\tilde\mu_t(x_t, \hat x_0)$: predict the clean sample, then take the forward-posterior step toward it. (Substitute $\hat x_0$ into Theorem 2; it is the ε-form with $\varepsilon_\theta$ in place of $\varepsilon$.) Libraries such as diffusers compute the step this way, and can clip $\hat x_0$ to the data range.

Interactive figure: loading…

$\sigma_t^2 = \beta_t$ or $\sigma_t^2 = \tilde\beta_t$? #

The best fixed $\sigma_t^2$ is the variance of the true reverse conditional $q(x_{t-1} \mid x_t)$, with $x_0$ integrated out. It depends on the data. A 1D Gaussian example shows the range.

Let $x_0 \sim \mathcal N(0, c^2)$ and $v_t = \mathrm{Var}(x_t) = \bar\alpha_t c^2 + 1 - \bar\alpha_t$. Then $\mathrm{Cov}(x_{t-1}, x_t) = \sqrt{\alpha_t}\,v_{t-1}$ and $v_t = \alpha_t v_{t-1} + \beta_t$. Gaussian conditioning gives

• $c = 0$ (all data at one point): $v_t = 1 - \bar\alpha_t$, so the variance is $\tilde\beta_t$. This is exact for any single point, because then $q(x_{t-1} \mid x_t) = q(x_{t-1} \mid x_t, x_0)$.
• $c = 1$ (unit-variance Gaussian data): $v_t = 1$, so the variance is $\beta_t$.
• $0 < c < 1$: in between.

Ho et al. (§3.2) state it the same way: $\beta_t$ is optimal for $x_0 \sim \mathcal N(0, I)$, $\tilde\beta_t$ for a single point, and these are the two extremes for data with unit variance per coordinate. In practice both give similar samples with $T = 1000$, because $\tilde\beta_t/\beta_t = (1 - \bar\alpha_{t-1})/(1 - \bar\alpha_t)$ is close to 1 except for the first steps. For the linear schedule it is $0.45$ at $t = 2$, $0.85$ at $t = 10$ and above $0.99$ for $t \ge 168$. The variance matters more for the likelihood, because the ELBO terms are largest at small $t$. Nichol and Dhariwal (2021) learn $\Sigma_\theta = \exp\big(\nu \log\beta_t + (1 - \nu)\log\tilde\beta_t\big)$ with a network output $\nu$, trained by $L_{\text{simple}} + 0.001\,L$. This improves the log-likelihood and allows fewer sampling steps.

Why small steps make the reverse step Gaussian

Bayes' rule gives $q(x_{t-1} \mid x_t) \propto q(x_t \mid x_{t-1})\,q_{t-1}(x_{t-1})$, where $q_{t-1}$ is the density of the noised data at step $t - 1$. The first factor is a Gaussian in $x_{t-1}$ with variance $\beta_t/\alpha_t$, so for small $\beta_t$ it is very narrow. Over such a narrow window, $\log q_{t-1}$ is close to its first-order Taylor expansion, a linear function. A Gaussian times the exponential of a linear function is again a Gaussian, with its mean shifted by $(\beta_t/\alpha_t) \nabla \log q_{t-1}$: the score appears here. So the reverse step is Gaussian up to an error that shrinks with $\beta_t$ (Feller 1949; Sohl-Dickstein et al. 2015). For a large $\beta_t$, the window covers several modes of $q_{t-1}$, and the product keeps them all.

Answers to section 1. (1) No. Theorem 1 gives $x_{500}$ from $x_0$ and one draw of $\varepsilon$. (2) The likelihood gets slightly worse ($3.75$ vs $3.70$ bits/dim) and the samples get much better (FID $3.17$ vs $13.51$), because the weights moved from small $t$ to larger $t$. (3) Worse: with large steps the true reverse conditional has several modes, and a Gaussian step cannot follow it. With the exact noise predictor on the 8 Gaussians, 99% of the samples land within 0.45 of a mode with $T = 1000$, but only 85% with $T = 10$ and 55% with $T = 5$ (notebook, section 4).

4. Worked example #

A three-step schedule with $\bar\alpha_1 = 0.64$, $\bar\alpha_2 = 0.36$ and $\bar\alpha_3 = 0.04$.

1. $\alpha_t = \bar\alpha_t/\bar\alpha_{t-1}$: $\alpha_1 = 0.64$, $\alpha_2 = 0.36/0.64 = 0.5625$, $\alpha_3 = 0.04/0.36 = 1/9$.
2. $\beta_t = 1 - \alpha_t$: $\beta_1 = 0.36$, $\beta_2 = 0.4375$, $\beta_3 = 8/9$.
3. Signal scales: $\sqrt{\bar\alpha_1} = 0.8$, $\sqrt{\bar\alpha_2} = 0.6$, $\sqrt{\bar\alpha_3} = 0.2$. Noise scales: $\sqrt{1 - \bar\alpha_1} = 0.6$, $\sqrt{1 - \bar\alpha_2} = 0.8$, $\sqrt{1 - \bar\alpha_3} = 0.9798$.
4. $\mathrm{SNR}_t = \bar\alpha_t/(1 - \bar\alpha_t)$: $\mathrm{SNR}_1 = 1.778$, $\mathrm{SNR}_2 = 0.5625$, $\mathrm{SNR}_3 = 0.0417$.

Marginal. $q(x_3 \mid x_0) = \mathcal N(0.2\,x_0,\ 0.96)$.

Forward posterior at $t = 2$.

• Variance: $\tilde\beta_2 = \dfrac{1 - 0.64}{1 - 0.36} \cdot 0.4375 = \dfrac{0.36 \cdot 0.4375}{0.64}$, so $\tilde\beta_2 = 0.24609375$. It is smaller than $\beta_2 = 0.4375$.
• Check with the precision: $\alpha_2/\beta_2 = 1.2857$ and $1/(1 - \bar\alpha_1) = 2.7778$. The sum is $4.0635$, and $1/4.0635 = 0.24609$. ✓
• Coefficient of $x_0$: $\dfrac{0.8 \cdot 0.4375}{0.64} = 0.546875$.
• Coefficient of $x_2$: $\dfrac{\sqrt{0.5625} \cdot 0.36}{0.64} = 0.75 \cdot 0.5625 = 0.421875$.

One noisy sample. Take $x_0 = 1$ and $\varepsilon = 0.5$. Then $x_2 = 0.6 \cdot 1 + 0.8 \cdot 0.5 = 1.0$.

• Mean from $x_0 = 1$ and $x_2 = 1$: $\tilde\mu_2 = 0.546875 + 0.421875 = 0.96875$.
• Mean from the ε-form: the noise term is $\frac{0.4375}{0.8} \cdot 0.5 = 0.2734375$, and $\tilde\mu_2 = (1.0 - 0.2734375)/0.75 = 0.96875$. ✓

A network predicts $\varepsilon_\theta = 0.3$ (error $0.2$).

• Model mean: the noise term is $\frac{0.4375}{0.8} \cdot 0.3 = 0.1640625$, so $\mu_\theta = (1.0 - 0.1640625)/0.75 = 1.114583$.
• Implied clean sample: $\hat x_0 = (1.0 - 0.8 \cdot 0.3)/0.6 = 1.266667$. Then $\tilde\mu_2(x_2, \hat x_0) = 0.546875 \cdot 1.266667 + 0.421875$, which is $1.114583$. ✓
• With $\sigma_2^2 = \tilde\beta_2$: the mean part of the KL is $(\tilde\mu_2 - \mu_\theta)^2/(2\tilde\beta_2)$. Here $\tilde\mu_2 - \mu_\theta = -0.145833$ and $2\tilde\beta_2 = 0.4921875$, so it is $0.043210$.
• The same from Theorem 4: $w_2 = \tfrac12(\mathrm{SNR}_1/\mathrm{SNR}_2 - 1)$. Here $\mathrm{SNR}_1/\mathrm{SNR}_2 = 3.1605$, so $w_2 = 1.080247$, and $w_2 \cdot 0.2^2 = 0.043210$. ✓
• With $\sigma_2^2 = \beta_2$: $w_2 = \dfrac{0.4375}{2 \cdot 0.5625 \cdot 0.64} = 0.607639$. The mean part is $0.607639 \cdot 0.04 = 0.024306$ (plus a different constant $C_2$).

$L_T$ is not zero for three steps. For $x_0 = 1$, $q(x_3 \mid x_0) = \mathcal N(0.2, 0.96)$, and the Gaussian KL formula gives

For the linear schedule with $T = 1000$ it is $2.0 \times 10^{-5}$ nats.

Open the figure of section 2 with "worked example" to see this posterior: $\bar\alpha_{t-1} = 0.64$, $\beta_t = 0.4375$ and $x_t = 1$.

5. In code #

NumPy checks the worked example by plain Gaussian conditioning (no DDPM formula), checks Theorem 4 on the same numbers, and checks Theorems 1 and 2 on the linear schedule.

import numpy as np

# (a) Worked example: abar = (0.64, 0.36, 0.04). The forward posterior at t = 2 by plain
#     Gaussian conditioning of the joint (x_1, x_2) given x_0, without any DDPM formula.
ab1, ab2 = 0.64, 0.36
a2 = ab2 / ab1                                   # alpha_2 = 0.5625
x0, x2 = 1.0, 1.0
mean = np.array([np.sqrt(ab1) * x0, np.sqrt(ab2) * x0])
cov = np.array([[1 - ab1, np.sqrt(a2) * (1 - ab1)],
                [np.sqrt(a2) * (1 - ab1), 1 - ab2]])
k = cov[0, 1] / cov[1, 1]                        # regression coefficient of x_1 on x_2
print(f"{mean[0] + k * (x2 - mean[1]):.5f} {cov[0, 0] - k * cov[0, 1]:.8f}")
# 0.96875 0.24609375                            (mu_tilde_2 and beta_tilde_2)

# (b) The same mean from the eps-form, and the eps-MSE weight with sigma^2 = beta_tilde
b2, eps, eps_hat = 1 - a2, 0.5, 0.3
mu = (x2 - b2 / np.sqrt(1 - ab2) * eps) / np.sqrt(a2)
mu_th = (x2 - b2 / np.sqrt(1 - ab2) * eps_hat) / np.sqrt(a2)
bt2 = (1 - ab1) / (1 - ab2) * b2
w2 = b2**2 / (2 * bt2 * a2 * (1 - ab2))
print(f"{mu:.5f} {(mu - mu_th)**2 / (2 * bt2):.6f} {w2 * (eps - eps_hat)**2:.6f}")
# 0.96875 0.043210 0.043210

# (c) Theorems 1 and 2 on the linear schedule: chain vs jump, and the posterior variance
rng = np.random.default_rng(0)
T = 1000
betas = np.linspace(1e-4, 0.02, T)
abar = np.cumprod(1 - betas)
x = np.full(100000, 2.0)
for b in betas[:300]:                            # run the chain for 300 steps
    x = np.sqrt(1 - b) * x + np.sqrt(b) * rng.standard_normal(x.shape)
print(f"{x.mean():.3f} {np.sqrt(abar[299]) * 2:.3f} {x.var():.3f} {1 - abar[299]:.3f}")
# 1.262 1.259 0.602 0.604                       (chain vs closed form: mean, variance)
ratio = (1 - np.concatenate([[1.0], abar[:-1]])) / (1 - abar)   # beta_tilde_t / beta_t
print(f"{abar[-1]:.2e}", ratio[1].round(3), ratio[9].round(3), np.argmax(ratio > 0.99) + 1)
# 4.04e-05 0.455 0.853 168

The notebook writes the closed forms, the ELBO weights and the ancestral sampler, runs the sampler with the exact noise predictor of the 8 Gaussians, and (optionally, with PyTorch) trains a small noise predictor on two moons.

6. Where it shows up #

• Images, video, audio. Noise prediction with $L_{\text{simple}}$ is the original recipe of latent diffusion (Stable Diffusion 1.x), applied to autoencoder latents (lesson 13). Imagen, DALL·E 2 and many audio models (DiffWave, WaveGrad) started from the same loss.
• Weather and climate. GenCast generates each ensemble member as a diffusion sample, conditioned on the recent weather states (it uses the EDM form of lesson 08). CorrDiff downscales to km scale with a diffusion model of the residual of a regression network.
• Seismology. Latent diffusion emulators of ground motion run exactly this forward process on the latent code of a waveform autoencoder, with conditioning inputs such as magnitude, distance and site terms.
• Likelihoods. The full bound $L$ gives bits per dimension (lesson 14). Variational diffusion models train $L$ directly and reach strong likelihoods.
• Practice: data scale. The schedule assumes data of unit scale. For data with standard deviation $c$, the effective SNR is $c^2\,\bar\alpha_t/(1 - \bar\alpha_t)$. Data with $c = 10$ have 100 times the SNR at every step, as if the log-SNR were shifted by $\ln 100 \approx 4.6$. So standardize each channel (for physical fields: each variable and level).

7. Common mistakes #

• Indexing off by one. $\bar\alpha_t$ includes step $t$, $\bar\alpha_0 = 1$, and the posterior uses $\bar\alpha_{t-1}$. Storing schedules with length $T + 1$ and index 0 for clean data avoids most errors.
• Mixing up $\beta_t$ and $1 - \bar\alpha_t$. $\beta_t$ is the variance added by one step; $1 - \bar\alpha_t$ is the total noise variance at $t$. $x_t = \sqrt{1 - \beta_t}\,x_0 + \sqrt{\beta_t}\,\varepsilon$ is wrong.
• A variance where a standard deviation is needed. Multiply $\varepsilon$ by $\sqrt{1 - \bar\alpha_t}$, and $z$ by $\sigma_t$, not by $\sigma_t^2$.
• Noise at the last step. At $t = 1$, return the mean ($z = 0$).
• Reading $L_{\text{simple}}$ as a likelihood. Its value is not a bound on $-\log p(x_0)$. To report bits/dim, evaluate the weighted bound $L$.
• Expecting the loss to reach zero. Even the best network cannot know $\varepsilon$ exactly from $x_t$. The minimum of the loss is the conditional variance of $\varepsilon$ given $x_t$, summed over coordinates (lesson 08). It stays positive.
• Large steps with a Gaussian reverse step. With $T = 10$ and the same total noise, the true reverse conditional can be far from Gaussian (for multimodal data it can have several modes), and ancestral sampling gives poor samples. Few-step sampling needs other tools (lesson 09 and lesson 12).
• Data that are not standardized. The schedule then gives a different SNR than intended, and the last step may still contain a lot of signal.

8. Check yourself #

1. Why does $q(x_{t-1} \mid x_t, x_0)$ have a closed form, but $q(x_{t-1} \mid x_t)$ does not?
   Answer

   Given $x_0$, all variables are linear-Gaussian, so conditioning is Gaussian algebra. Without $x_0$, $q(x_{t-1} \mid x_t) = \int q(x_{t-1} \mid x_t, x_0)\, q(x_0 \mid x_t)\, dx_0$ needs the posterior over the data, which depends on the unknown data distribution.
2. A constant schedule $\beta_t = 0.01$. After how many steps is $\sqrt{\bar\alpha_t} < 0.01$?
   Answer

   $\bar\alpha_t = 0.99^t$, and we need $0.99^t < 10^{-4}$: $t > \ln 10^{-4}/\ln 0.99 = 916.4$, so $t = 917$.
3. Show that the DDPM step mean equals $\tilde\mu_t(x_t, \hat x_0)$ with $\hat x_0 = (x_t - \sqrt{1 - \bar\alpha_t}\,\varepsilon_\theta)/\sqrt{\bar\alpha_t}$.
   Answer

   The ε-form was obtained by substituting $x_0 = (x_t - \sqrt{1 - \bar\alpha_t}\,\varepsilon)/\sqrt{\bar\alpha_t}$ into $\tilde\mu_t$. The same algebra with $\varepsilon_\theta$ in place of $\varepsilon$ turns $\tilde\mu_t(x_t, \hat x_0)$ into $\frac{1}{\sqrt{\alpha_t}}\big(x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}}\varepsilon_\theta\big) = \mu_\theta$.
4. With $\sigma_t^2 = \tilde\beta_t$, write the ELBO weight $w_t$ of the ε-MSE in terms of the SNR. Is it larger at small or at large $t$ for the linear schedule?
   Answer

   $w_t = \tfrac12\big(\mathrm{SNR}(t-1)/\mathrm{SNR}(t) - 1\big)$. It is largest at small $t$ ($0.60$ at $t = 2$) and only about $0.005$ to $0.010$ for $t \ge 100$.
5. All data are one point, $x_0 = c$. What must $\varepsilon_\theta$ learn so that the reverse model with $\sigma_t^2 = \tilde\beta_t$ is exact?
   Answer

   $\varepsilon_\theta(x_t, t) = (x_t - \sqrt{\bar\alpha_t}\,c)/\sqrt{1 - \bar\alpha_t}$, the noise implied by the known $c$. Then $\hat x_0 = c$ and $\mu_\theta = \tilde\mu_t(x_t, c)$, so every reverse step equals the true one.
6. Why does $L_T$ not appear in training, and when is that a problem?
   Answer

   It has no parameters when the $\beta_t$ are fixed. It is a problem when $\bar\alpha_T$ is not small: then the prior $\mathcal N(0, I)$ does not match $q(x_T \mid x_0)$ (for $x_0 = 1$: 0.0204 nats for the three-step example, $2 \times 10^{-5}$ nats for the linear schedule). Lesson 08 shows a real case (zero terminal SNR).

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. What is $q(x_t \mid x_0)$ in DDPM?

   Show answer

2. How do you draw $x_t$ from $x_0$ in one line?

   Show answer

3. What is the variance $\tilde\beta_t$ of the forward posterior $q(x_{t-1} \mid x_t, x_0)$?

   Show answer

4. Write the forward posterior mean $\tilde\mu_t$ in terms of $x_t$ and the noise $\varepsilon$.

   Show answer

5. What is $L_{\text{simple}}$?

   Show answer

6. Why does $L_{\text{simple}}$ give better samples than the ELBO?

   Show answer

7. Write one step of DDPM ancestral sampling.

   Show answer

8. For which data is $\sigma_t^2 = \tilde\beta_t$ exact, and for which is $\sigma_t^2 = \beta_t$ exact?

   Show answer

9. Why must the forward steps of DDPM be small?

   Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Derive the DDPM loss." ELBO; rewrite $q(x_{1:T} \mid x_0)$ with forward posteriors (the ratios telescope); KL between Gaussians; ε-form of the mean; drop the weights.
• "Why predict the noise and not the mean?" With the ε-form, the KL becomes an ε-MSE with a unit-scale target at every $t$: denoising score matching. Ho et al. (Table 2) report that predicting $\tilde\mu_t$ directly works only with the true bound, and that training it with an unweighted MSE was unstable.
• "Why 1000 steps?" A Gaussian reverse step is accurate only for small steps.
• "Is DDPM a VAE?" A hierarchical VAE with a fixed encoder, latents of the data dimension, and one decoder network shared across steps (it takes $t$ as input).
• "What does $\sigma_t^2$ change?" Almost nothing for samples with $T = 1000$; it matters for the likelihood. $\tilde\beta_t$ is exact for a point mass, $\beta_t$ for $\mathcal N(0, I)$ data.
• "How do you get $x_t$ during training?" In one line from $x_0$ and one Gaussian draw (Theorem 1); the chain is never run during training.

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution, and an optional PyTorch demonstration:

1. Make $x_t$ two ways, by the one-step jump and by running the chain; compare the means and variances.
2. Write q_posterior; check it against the worked example and against Monte Carlo conditioning.
3. Compute the ELBO weights $w_t$ of the ε-MSE for the linear schedule, for both variance choices; plot them.
4. Write the ancestral sampler; run it with the exact noise predictor of the 8 Gaussians and count the samples per mode; then use fewer, larger steps.
5. Compare the schedules: when is half of the signal variance gone?
6. Optional (PyTorch): train a small noise predictor on two moons with $L_{\text{simple}}$ and sample from it.

11. Watch and read #

What are Diffusion Models?Ari Seff · 15 min

The forward and reverse processes, the closed-form marginal and the noise-prediction loss, with clear animations.

Diffusion Models: DDPM | Generative AI AnimatedDeepia · 32 min

The full derivation of this lesson: the ELBO, its simplification into KL terms, and the step from the ELBO to an L2 loss.

Diffusion Models | Paper Explanation | Math ExplainedOutlier · 33 min

A slower walk through the DDPM paper and its math, including the forward posterior.

But how do AI images and videos actually work? | Guest video by Welch Labs3Blue1Brown · 37 min

The big picture of diffusion for images and video, with excellent visual intuition.

• Ho, Jain and Abbeel, Denoising Diffusion Probabilistic Models (2020): §2 (eqs. 1–7), §3.2 (eqs. 8–12 and the two choices of $\sigma_t^2$), §3.4 ($L_{\text{simple}}$), Algorithms 1–2, Tables 1–2, Appendix A.
• Calvin Luo, Understanding Diffusion Models: A Unified Perspective (2022): the section "Variational Diffusion Models" (the ELBO with forward posteriors, the forward posterior, the loss as an SNR difference).
• Nichol and Dhariwal, Improved Denoising Diffusion Probabilistic Models (2021): §3.1 (learned variances, the hybrid loss), §3.2 (the cosine schedule).
• Sohl-Dickstein et al., Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015): the first diffusion model, and the argument for small steps.
• Lilian Weng, What are Diffusion Models?: a readable overview with all formulas of this lesson.