Diffusion parameterizations: $\varepsilon$, $x_0$, $v$ and the score; SNR weighting; EDM preconditioning

Before this lesson: lesson 06 (the score, denoising score matching, Tweedie's formula at one noise level) and lesson 07 (the forward process $q(x_t \mid x_0)$, the ELBO and the $\varepsilon$-loss, noise schedules). You also need conditional expectation and the Gaussian density (probability and statistics) and the change of variables in an integral. Time: about 100 minutes, plus the notebook.

---

Why this matters #

Lesson 07 trained a network to predict the noise $\varepsilon$ (DDPM, Stable Diffusion 1). Other models predict other things. EDM (Karras et al. 2022) predicts the clean sample through a preconditioned network. Progressive distillation, Imagen Video and the 768-pixel Stable Diffusion 2 model predict the velocity $v$. Score-based SDE models predict the score. Flow matching and rectified flow (lesson 10, Stable Diffusion 3) predict a flow velocity. These look like different methods. They are not: at the optimum they are one function, seen through different linear maps.

What really differs is (1) how the loss weights the noise levels, (2) the scale of the target the network must output, and (3) how much a network error grows when it is converted to a clean-sample estimate. These three points explain most training choices: cosine vs linear schedules, min-SNR weighting, $v$-prediction, EDM preconditioning, zero terminal SNR, and schedule shifts for large images. When you choose a target and a weighting for a new model (for example a latent diffusion model of weather fields or seismic waveforms), you choose a point in this design space.

1. Predict first #

2. The picture #

One equation, two unknowns #

Fix one noise level $t$. The noisy sample is

with signal scale $a_t$ and noise scale $\sigma_t$. In DDPM, $a_t = \sqrt{\bar\alpha_t}$ and $\sigma_t = \sqrt{1 - \bar\alpha_t}$ (lesson 07). The network sees $x_t$. It does not see $x_0$ or $\varepsilon$. But the equation links them: a guess of $x_0$ fixes the guess of $\varepsilon$, and the other way round:

So "predict the noise" and "predict the clean sample" are two ways to write the same answer.

The noise level as a rotation #

Look at one coordinate. Then $(x_0, \varepsilon)$ is a point in a plane. In the variance-preserving (VP) case $a_t^2 + \sigma_t^2 = 1$, so we can write $a_t = \cos\phi$ and $\sigma_t = \sin\phi$ for an angle $\phi$ between $0°$ (clean) and $90°$ (pure noise). Then $x_t = \cos\phi\, x_0 + \sin\phi\, \varepsilon$ is the coordinate of the point along the unit direction $(\cos\phi, \sin\phi)$.

Define the velocity as the coordinate along the perpendicular direction $(-\sin\phi, \cos\phi)$:

It is called the velocity because it is the rate at which $x_t$ moves as the angle $\phi$ grows. The pair $(x_t, v)$ is the pair $(x_0, \varepsilon)$ in a frame rotated by $\phi$. A rotation is easy to undo, so

Check the first one: $a_t x_t - \sigma_t v = a_t^2 x_0 + a_t\sigma_t\varepsilon - a_t\sigma_t\varepsilon + \sigma_t^2 x_0 = x_0$, because $a_t^2 + \sigma_t^2 = 1$.

Given $x_t$, the unknown point $(x_0, \varepsilon)$ lies on one line: the line $a_t x_0 + \sigma_t\varepsilon = x_t$, perpendicular to the direction of $x_t$. Any prediction picks one point on this line. An $\varepsilon$-network reports its vertical coordinate, an $x_0$-network its horizontal coordinate, a $v$-network its position along the line.

Interactive figure: loading…

How errors grow when you convert #

A real network makes errors. The figure shows what happens to them:

• From $\varepsilon$ to $x_0$: an error $\delta$ in $\hat\varepsilon$ becomes an error $(\sigma_t/a_t)\,\delta$ in $\hat x_0$. At high noise $a_t \to 0$, so this blows up.
• From $x_0$ to $\varepsilon$: an error $\delta$ in $\hat x_0$ becomes $(a_t/\sigma_t)\,\delta$ in $\hat\varepsilon$. At low noise $\sigma_t \to 0$, so this blows up.
• From $v$: $\hat x_0 = a_t x_t - \sigma_t \hat v$ and $\hat\varepsilon = \sigma_t x_t + a_t \hat v$, so a $v$-error is multiplied by $\sigma_t \le 1$ or by $a_t \le 1$. It never grows.

This is the main practical reason to prefer $v$ (or the EDM preconditioning of section 3) over $\varepsilon$ when the model must work at very high noise.

The best prediction is a conditional mean #

Many pairs $(x_0, \varepsilon)$ give the same $x_t$. A network that only sees $x_t$ cannot know which pair was used. So what is the best it can do? For a squared-error loss the answer is always the same: the minimizer of $\mathbb E\|\text{target} - f(x_t)\|^2$ over all functions $f$ is the conditional mean $\mathbb E[\text{target} \mid x_t]$ (proof in section 3). The best $x_0$-prediction is the average of all clean samples that could have produced $x_t$, weighted by how likely they are.

This optimal $x_0$-prediction is called the denoiser $D(x_t) = \mathbb E[x_0 \mid x_t]$. Every other target is a linear function of $(x_0, \varepsilon)$ for a fixed $x_t$, and the conditional mean is linear. So the optimal $\varepsilon$, $v$ and score predictions are the same linear maps applied to $D$. One function determines all of them, and Tweedie's formula says which one: the score of the noisy density. In the simplest variance-exploding (VE) form $x = x_0 + \sigma\varepsilon$ (no scaling, $a = 1$), it reads

The denoiser is one step from $x$ along the score, with step size $\sigma^2$. You met this form in lesson 06.

Interactive figure: loading…

3. Definitions and main results #

Conversions #

From $x_t$ and one prediction you get all the others (VP case):

given	$\hat x_0$	$\hat\varepsilon$	$\hat v$	score $\hat s$
$\hat\varepsilon$	$(x_t - \sigma_t\hat\varepsilon)/a_t$	—	$(\hat\varepsilon - \sigma_t x_t)/a_t$	$-\hat\varepsilon/\sigma_t$
$\hat x_0$	—	$(x_t - a_t\hat x_0)/\sigma_t$	$(a_t x_t - \hat x_0)/\sigma_t$	$-(x_t - a_t\hat x_0)/\sigma_t^2$
$\hat v$	$a_t x_t - \sigma_t\hat v$	$\sigma_t x_t + a_t\hat v$	—	$-(\sigma_t x_t + a_t\hat v)/\sigma_t$

The first two columns use only $x_t = a_t x_0 + \sigma_t\varepsilon$. The $v$ entries also use $a_t^2 + \sigma_t^2 = 1$; for example, $v = a_t\varepsilon - \sigma_t(x_t - \sigma_t\varepsilon)/a_t = (\varepsilon - \sigma_t x_t)/a_t$. The score column is exact for the optimal predictions (Theorem 1 below); for a trained network it defines the score model.

The squared loss is minimized by the conditional mean #

Lemma. Let $Y$ be a target and $X$ an input with $\mathbb E\|Y\|^2 < \infty$. Over all functions $f$, $\mathbb E\|Y - f(X)\|^2$ is smallest for $f^*(x) = \mathbb E[Y \mid X = x]$, and the minimum is $\mathbb E\,\operatorname{tr}\operatorname{Cov}(Y \mid X)$.

Proof.

1. Write $Y - f(X) = (Y - f^*(X)) + (f^*(X) - f(X))$.
2. Expand the square. The cross term is $2\,\mathbb E\big[(f^*(X) - f(X))^\top (Y - f^*(X))\big]$.
3. Condition on $X$: the first factor is fixed, and $\mathbb E[Y - f^*(X) \mid X] = 0$. So the cross term is 0.
4. Hence $\mathbb E\|Y - f(X)\|^2 = \mathbb E\|Y - f^*(X)\|^2 + \mathbb E\|f^*(X) - f(X)\|^2$. The second term is $\ge 0$ and is 0 only for $f = f^*$. The first term is $\mathbb E\,\operatorname{tr}\operatorname{Cov}(Y \mid X)$. $\blacksquare$

So with unlimited capacity, the $\varepsilon$-network learns $\mathbb E[\varepsilon \mid x_t]$, the $x_0$-network learns $\mathbb E[x_0 \mid x_t]$, and so on, at every noise level.

Theorem 1 (Tweedie's formula) #

For any data distribution with a finite mean, and $x_t = a_t x_0 + \sigma_t\varepsilon$ with $a_t > 0$, $\sigma_t > 0$,

Why this is true (intuition). The noisy density $p_t$ is the data density blurred by a Gaussian. Its score at $x_t$ points toward where the clean data that could explain $x_t$ is concentrated. Moving along the score by $\sigma_t^2$ (the noise variance) moves you to the center of that mass.

Proof.

1. $x_t$ given $x_0$ is $\mathcal N(a_t x_0, \sigma_t^2 I)$, so $p_t(x) = \int \mathcal N(x; a_t x_0, \sigma_t^2 I)\, p_{\text{data}}(x_0)\, dx_0$.
2. The gradient of a Gaussian density in $x$ is $\nabla_x \mathcal N(x; a_t x_0, \sigma_t^2 I) = -\frac{x - a_t x_0}{\sigma_t^2}\,\mathcal N(x; a_t x_0, \sigma_t^2 I)$.
3. Differentiate step 1 under the integral (allowed for a Gaussian kernel): $\nabla p_t(x) = -\frac{1}{\sigma_t^2}\int (x - a_t x_0)\,\mathcal N(x; a_t x_0, \sigma_t^2 I)\,p_{\text{data}}(x_0)\,dx_0$.
4. Bayes' rule: $p(x_0 \mid x_t = x) = \mathcal N(x; a_t x_0, \sigma_t^2 I)\,p_{\text{data}}(x_0)/p_t(x)$. Divide step 3 by $p_t(x)$; the integral becomes an expectation over this posterior: $\nabla \log p_t(x) = \frac{\nabla p_t(x)}{p_t(x)} = -\frac{x - a_t\,\mathbb E[x_0 \mid x_t = x]}{\sigma_t^2}$.
5. Solve step 4 for $\mathbb E[x_0 \mid x_t]$: this is the first formula.
6. $\varepsilon = (x_t - a_t x_0)/\sigma_t$ is linear in $x_0$ for fixed $x_t$, so $\mathbb E[\varepsilon \mid x_t] = (x_t - a_t\,\mathbb E[x_0 \mid x_t])/\sigma_t = -\sigma_t\,\nabla \log p_t(x_t)$ by step 4. $\blacksquare$

With $a_t = 1$, $\sigma_t = \sigma$ this is the VE form of section 2.

Consequences.

• All optima are one function. $\varepsilon^* = -\sigma_t\nabla\log p_t$, $x_0^* = (x_t + \sigma_t^2\nabla\log p_t)/a_t$, $v^* = a_t\varepsilon^* - \sigma_t x_0^*$. With unlimited capacity, the choice of target does not change the learned model at any single noise level. It changes how the levels are weighted and how training behaves (Theorem 2).
• Denoising score matching is the same thing. Regressing $\hat s$ on $-\varepsilon/\sigma_t$ is the $\varepsilon$-loss divided by $\sigma_t^2$; its optimum is $-\mathbb E[\varepsilon \mid x_t]/\sigma_t = \nabla\log p_t$.
• The loss has a floor. By the lemma, the minimum of $\mathbb E\|\varepsilon - \hat\varepsilon\|^2$ is $\mathbb E\,\operatorname{tr}\operatorname{Cov}(\varepsilon \mid x_t) > 0$. A loss value alone does not say how good a model is.

Second-order Tweedie: the posterior covariance

Write $m(x) = \mathbb E[x_0 \mid x_t = x]$. Two facts give the covariance of the posterior.

1. The posterior is $p(x_0 \mid x) \propto p_{\text{data}}(x_0)\exp\!\big(a_t x_0^\top x/\sigma_t^2 - a_t^2\|x_0\|^2/(2\sigma_t^2)\big)$. As a function of $x$, $x$ enters only through $\eta = a_t x/\sigma_t^2$, like the natural parameter of an exponential family. For such a family the derivative of the mean with respect to $\eta$ is the covariance. So $\partial m/\partial x = (a_t/\sigma_t^2)\operatorname{Cov}(x_0 \mid x)$.
2. Differentiate Tweedie's formula: $\partial m/\partial x = (I + \sigma_t^2\,\nabla^2 \log p_t(x))/a_t$.

Together:

$$\operatorname{Cov}(x_0 \mid x_t) = \frac{\sigma_t^2}{a_t^2}\big(I + \sigma_t^2\,\nabla^2 \log p_t(x_t)\big).$$

Posterior sampling for inverse problems uses it (lesson 15).

Theorem 2 (the losses differ only by an SNR weight) #

Let $\hat x_0$, $\hat\varepsilon$ be one network output written in two forms, so that $x_t = a_t\hat x_0 + \sigma_t\hat\varepsilon$. Then for every example

In the VP case, with $\hat v = a_t\hat\varepsilon - \sigma_t\hat x_0$,

Proof.

1. The true pair and the predicted pair give the same $x_t$: $a_t x_0 + \sigma_t\varepsilon = a_t\hat x_0 + \sigma_t\hat\varepsilon$. So $a_t(x_0 - \hat x_0) = -\sigma_t(\varepsilon - \hat\varepsilon)$.
2. Take squared norms: $a_t^2\|x_0 - \hat x_0\|^2 = \sigma_t^2\|\varepsilon - \hat\varepsilon\|^2$. Divide by $a_t^2$: this is the first identity, since $\sigma_t^2/a_t^2 = 1/\mathrm{SNR}$. (No VP assumption so far.)
3. For $v$: $v - \hat v = a_t(\varepsilon - \hat\varepsilon) - \sigma_t(x_0 - \hat x_0)$. Insert step 1, $x_0 - \hat x_0 = -(\sigma_t/a_t)(\varepsilon - \hat\varepsilon)$: $v - \hat v = \big(a_t + \sigma_t^2/a_t\big)(\varepsilon - \hat\varepsilon) = \frac{a_t^2 + \sigma_t^2}{a_t}(\varepsilon - \hat\varepsilon) = \frac{1}{a_t}(\varepsilon - \hat\varepsilon)$, using VP.
4. Square: $\|v - \hat v\|^2 = \|\varepsilon - \hat\varepsilon\|^2/a_t^2$, and $1/a_t^2 = (a_t^2 + \sigma_t^2)/a_t^2 = 1 + 1/\mathrm{SNR}$. The last form follows from step 2. $\blacksquare$

Read it in "$x_0$ units" (the factor in front of $\|x_0 - \hat x_0\|^2$): the $x_0$-loss has weight $1$, the $\varepsilon$-loss has weight $\mathrm{SNR}$, the $v$-loss has weight $1 + \mathrm{SNR}$, and the plain score loss $\|\hat s + \varepsilon/\sigma_t\|^2$ has weight $\mathrm{SNR}/\sigma_t^2$. So, compared with the $x_0$-loss, the $\varepsilon$-loss puts much more weight on low noise (high SNR), and much less on high noise.

All losses as one weighted integral over $\lambda$ (Kingma and Gao 2023) #

Theorem 2 lets us write every loss in $\varepsilon$ units. Write $x_\lambda$ for the noisy sample at log-SNR $\lambda$. Every common objective is a special case of

for some weight function $w(\lambda) \ge 0$.

How a schedule turns into a weight, step by step. Training draws $t \sim \mathcal U(0, 1)$ and computes $\lambda = \lambda_t$. Because $\lambda_t$ decreases in $t$, this gives $\lambda$ a density $p(\lambda) = -dt/d\lambda$ (change of variables for densities). Then

So a loss with weight $w$ can be trained with any schedule, if each sample is multiplied by $w(\lambda)/p(\lambda)$. In the other direction, the plain loss $L_{\text{simple}} = \mathbb E_t\|\varepsilon - \hat\varepsilon\|^2$ has $w(\lambda) = p(\lambda)$: for $L_{\text{simple}}$ the schedule is the weight.

Example: the cosine schedule. Without the small offset, $a_t = \cos(\pi t/2)$ and $\sigma_t = \sin(\pi t/2)$.

1. $\mathrm{SNR} = \cot^2(\pi t/2)$, so $\lambda = -2\log\tan(\pi t/2)$.
2. Solve for $t$: $t = \frac{2}{\pi}\arctan(e^{-\lambda/2})$.
3. Differentiate: $\frac{dt}{d\lambda} = \frac{2}{\pi}\cdot\frac{1}{1 + e^{-\lambda}}\cdot\big(-\tfrac12 e^{-\lambda/2}\big) = -\frac{1}{\pi}\cdot\frac{1}{e^{\lambda/2} + e^{-\lambda/2}}$.
4. So $p(\lambda) = \frac{1}{2\pi}\operatorname{sech}(\lambda/2)$, a bell shape centered at $\lambda = 0$.

Implied weights (up to a constant factor), using Theorem 2 to move each loss to $\varepsilon$ units:

objective	$w(\lambda)$	decreasing in $\lambda$?
continuous-time ELBO	$1$	yes (constant)
$\varepsilon$-loss, cosine schedule ($L_{\text{simple}}$)	$p(\lambda) \propto \operatorname{sech}(\lambda/2)$	no
$x_0$-loss, cosine schedule	$p(\lambda)\,e^{-\lambda}$	yes
$v$-loss, cosine schedule	$p(\lambda)(1 + e^{-\lambda}) = e^{-\lambda/2}/\pi$	yes
flow matching, $t \sim \mathcal U(0, 1)$	$\tfrac12 e^{-\lambda/2}$	yes
EDM	$\mathcal N(\lambda;\, 2.4,\, 2.4^2)\,(e^{-\lambda} + \sigma_{\text{data}}^2)/\sigma_{\text{data}}^2$	no
min-SNR-$\gamma$, $\varepsilon$-loss, cosine	$p(\lambda)\,\min(1, \gamma e^{-\lambda})$	no

Min-SNR-$\gamma$ (Hang et al. 2023, usually $\gamma = 5$) caps the weight in $x_0$ units at $\min(\mathrm{SNR}, \gamma)$. In $\varepsilon$ units that is $\min(\mathrm{SNR}, \gamma)/\mathrm{SNR} = \min(1, \gamma/\mathrm{SNR})$. It lowers the weight of the easy, low-noise levels.

Monotone weights are ELBOs (Kingma and Gao 2023, Theorem 1). If $w(\lambda)$ decreases as $\lambda$ grows (it increases with $t$), then $\mathcal L_w$ equals, up to a constant, an average over noise levels of the negative ELBO of noise-perturbed data. That is maximum likelihood with Gaussian noise as data augmentation. The $v$-loss with the cosine schedule and flow matching have monotone weights; $L_{\text{simple}}$ with the cosine schedule and EDM do not.

Interactive figure: loading…

Theorem 3 (the continuous-time ELBO depends on the schedule only through its end points) #

The ELBO of a diffusion model with $T$ steps contains a sum of terms $L_{t-1}$, one per step (lesson 07). As $T \to \infty$ with a fixed function $\mathrm{SNR}(t)$ on $t \in [0, 1]$, this sum becomes (Kingma et al. 2021, VDM, eq. 15)

$\mathrm{SNR}'(t) < 0$, so $L_\infty \ge 0$. Claim. Suppose the network uses its input only through the rescaled sample $x_t/a_t$ and the value $\mathrm{SNR}(t)$. Then

where $\mathrm{SNR}_{\max} = \mathrm{SNR}(0)$, $\mathrm{SNR}_{\min} = \mathrm{SNR}(1)$ and $\tilde x(z, u)$ is the network written as a function of $z = x_t/a_t$ and $u = \mathrm{SNR}$. Only the two end points of the schedule enter. The other two ELBO terms (the prior term at $t = 1$ and the reconstruction term at $t = 0$) also depend only on the end points.

Why this is true (intuition). The integrand is "the denoising error at signal-to-noise ratio $u$". The schedule only says when (at which $t$) each $u$ is visited and how fast. Integrating over $u$ instead of $t$ removes the "when".

Proof (from the discrete ELBO to the end points)

1. One step of the discrete ELBO. Ho et al. (2020, eq. 12) give $L_{t-1} = \frac{\beta_t^2}{2\tilde\beta_t\,\alpha_t(1 - \bar\alpha_t)}\,\mathbb E\|\varepsilon - \hat\varepsilon\|^2 + C_t$ for the reverse variance $\tilde\beta_t = \frac{1 - \bar\alpha_{t-1}}{1 - \bar\alpha_t}\beta_t$. Insert $\tilde\beta_t$: the weight is $\frac{\beta_t}{2\alpha_t(1 - \bar\alpha_{t-1})}$.
2. In SNR form. With $\mathrm{SNR}(t) = \bar\alpha_t/(1 - \bar\alpha_t)$ and $\bar\alpha_t = \alpha_t\bar\alpha_{t-1}$: $\frac{\mathrm{SNR}(t-1)}{\mathrm{SNR}(t)} = \frac{1 - \bar\alpha_t}{\alpha_t(1 - \bar\alpha_{t-1})}$, so $\frac{\mathrm{SNR}(t-1)}{\mathrm{SNR}(t)} - 1 = \frac{1 - \bar\alpha_t - \alpha_t + \bar\alpha_t}{\alpha_t(1 - \bar\alpha_{t-1})} = \frac{\beta_t}{\alpha_t(1 - \bar\alpha_{t-1})}$. So the weight is $\tfrac12\big(\mathrm{SNR}(t-1)/\mathrm{SNR}(t) - 1\big)$.
3. In $x_0$ units. Theorem 2, step 2 (no VP needed) gives $\|\varepsilon - \hat\varepsilon\|^2 = \mathrm{SNR}(t)\,\|x_0 - \hat x_0\|^2$. Hence $L_{t-1} = \tfrac12\big(\mathrm{SNR}(t-1) - \mathrm{SNR}(t)\big)\,\mathbb E\|x_0 - \hat x_0\|^2 + C_t$.
4. The limit. Put the steps on a grid $t_i = i/T$. Then $\mathrm{SNR}(t_{i-1}) - \mathrm{SNR}(t_i) \approx -\mathrm{SNR}'(t_i)/T$, and the sum over $i$ is a Riemann sum of the integral $L_\infty$.
5. Change of variables. $\mathrm{SNR}(t)$ is strictly decreasing, so it is invertible. Set $u = \mathrm{SNR}(t)$, $du = \mathrm{SNR}'(t)\,dt$; $t = 0 \mapsto \mathrm{SNR}_{\max}$ and $t = 1 \mapsto \mathrm{SNR}_{\min}$: $L_\infty = -\frac12 \int_{\mathrm{SNR}_{\max}}^{\mathrm{SNR}_{\min}} \mathbb E\|x_0 - \hat x_0\|^2\,du = \frac12 \int_{\mathrm{SNR}_{\min}}^{\mathrm{SNR}_{\max}} \mathbb E\|x_0 - \hat x_0\|^2\,du$.
6. The integrand depends only on $u$. $x_t/a_t = x_0 + (\sigma_t/a_t)\,\varepsilon = x_0 + \varepsilon/\sqrt{u}$. The law of this rescaled input depends only on $u$. By assumption the network output is $\tilde x(x_t/a_t, u)$. So the integrand is a function of $u$ alone, and the schedule enters only through the limits. $\blacksquare$

The same in $\lambda$. With $du = e^\lambda d\lambda$ and $e^\lambda\|x_0 - \hat x_0\|^2 = \|\varepsilon - \hat\varepsilon\|^2$ (Theorem 2), $L_\infty = \frac12 \int \mathbb E\|\varepsilon - \hat\varepsilon\|^2\, d\lambda$: the weight $w(\lambda) = 1$ of the table. The same change of variables shows that every $\mathcal L_w$ with a fixed $w(\lambda)$ does not depend on the schedule between the end points (Kingma and Gao 2023, section 3.2).

What the schedule still changes.

• The variance of training. The schedule is the importance-sampling distribution $p(\lambda)$ of the Monte Carlo estimate. It sets the variance of the loss and gradient estimates, and with it the training speed.
• The weight, for losses defined in $t$. For $L_{\text{simple}}$, $w(\lambda) = p(\lambda)$, so a new schedule is a new objective.
• The end points. $\mathrm{SNR}_{\min}$ must be close to 0 so that $x_1$ matches the Gaussian prior (see zero terminal SNR below), and $\mathrm{SNR}_{\max}$ sets how clean the last step is.
• The time grid of the sampler (lesson 09).
• Resolution. More pixels average the noise, so the coarse structure survives longer at the same SNR. The shifted cosine schedule (Hoogeboom et al. 2023) uses $\lambda(t) + 2\log(64/d)$ for resolution $d$; for $d = 256$ the shift is $2\log(1/4) = -2.77$.

EDM preconditioning (Karras et al. 2022, section 5) #

EDM works in the VE form $x = y + \sigma n$: clean data $y$ with standard deviation $\sigma_{\text{data}}$, noise $n \sim \mathcal N(0, I)$, noise level $\sigma$. (The link to VP: $y = x_0$, $\sigma = \sigma_t/a_t$, $x = x_t/a_t$.) The network $F_\theta$ is wrapped in a denoiser

The four coefficients are chosen so that the network always sees inputs and targets of unit size. The loss is $\mathbb E_{\sigma, y, n}\,\omega(\sigma)\,\|D_\theta(y + \sigma n; \sigma) - y\|^2$. (Karras et al. call the loss weight $\lambda(\sigma)$; we write $\omega$ to avoid a clash with the log-SNR.) Since $D_\theta - y = c_{\text{out}}\big(F_\theta - F_{\text{target}}\big)$ with the effective target

the loss is $\mathbb E\,\omega\,c_{\text{out}}^2\,\|F_\theta - F_{\text{target}}\|^2$. The coefficients follow in four steps.

1. Unit input variance. $\operatorname{Var}(c_{\text{in}} x) = c_{\text{in}}^2(\sigma_{\text{data}}^2 + \sigma^2) = 1$, so $c_{\text{in}} = 1/\sqrt{\sigma^2 + \sigma_{\text{data}}^2}$.
2. Unit target variance. $y - c_{\text{skip}}x = (1 - c_{\text{skip}})\,y - c_{\text{skip}}\sigma n$ has variance $(1 - c_{\text{skip}})^2\sigma_{\text{data}}^2 + c_{\text{skip}}^2\sigma^2$ (per coordinate; $y$ and $n$ are independent). For $\operatorname{Var}(F_{\text{target}}) = 1$, $c_{\text{out}}^2$ must equal this variance.
3. Smallest error amplification. An error of $F_\theta$ reaches $D_\theta$ multiplied by $c_{\text{out}}$. So choose $c_{\text{skip}}$ to make the variance of step 2 as small as possible. Set its derivative to zero: $-2(1 - c_{\text{skip}})\sigma_{\text{data}}^2 + 2c_{\text{skip}}\sigma^2 = 0$, so $c_{\text{skip}} = \frac{\sigma_{\text{data}}^2}{\sigma^2 + \sigma_{\text{data}}^2}$. Insert it into step 2: $c_{\text{out}} = \frac{\sigma\,\sigma_{\text{data}}}{\sqrt{\sigma^2 + \sigma_{\text{data}}^2}}$.
4. Equal effective weight. Choose $\omega(\sigma) = 1/c_{\text{out}}^2 = \frac{\sigma^2 + \sigma_{\text{data}}^2}{(\sigma\,\sigma_{\text{data}})^2}$, so every noise level has weight 1 on the loss of $F_\theta$.
5. Empirical choices (not derived): $c_{\text{noise}} = \tfrac14\ln\sigma$; training levels $\ln\sigma \sim \mathcal N(-1.2, 1.2^2)$; $\sigma_{\text{data}} = 0.5$ for images; sampling from $\sigma_{\max} = 80$ down to $\sigma_{\min} = 0.002$.

What $F_\theta$ learns. Insert $x = y + \sigma n$ and the coefficients:

For $\sigma \to 0$ this tends to $-n$: the network predicts the noise, like $\varepsilon$-prediction. For $\sigma \to \infty$ it tends to $y/\sigma_{\text{data}}$: the network predicts the signal, like $x_0$-prediction. In between it mixes the two, and its variance is always $\frac{\sigma^2}{\sigma^2 + \sigma_{\text{data}}^2} + \frac{\sigma_{\text{data}}^2}{\sigma^2 + \sigma_{\text{data}}^2} = 1$.

Interactive figure: loading…

The skip path is the best linear denoiser. For Gaussian data $y \sim \mathcal N(0, \sigma_{\text{data}}^2)$, $\mathbb E[y \mid x] = \frac{\sigma_{\text{data}}^2}{\sigma_{\text{data}}^2 + \sigma^2}\,x = c_{\text{skip}}\,x$ (the Wiener filter) and $\operatorname{Var}(y \mid x) = c_{\text{out}}^2$. So $c_{\text{skip}}x$ is already the optimal denoiser for Gaussian data, and $F_\theta$ learns only the correction for the non-Gaussian structure.

Older choices as special cases of the EDM form

(Karras et al., Table 1; Kingma and Gao, Appendix E.2.)

• $\varepsilon$-prediction: $c_{\text{skip}} = 1$, $c_{\text{out}} = -\sigma$. Network errors are multiplied by $\sigma$, by 80 at EDM's $\sigma_{\max}$.
• $x_0$-prediction: $c_{\text{skip}} = 0$, $c_{\text{out}} = 1$.
• $v$-prediction is EDM with $\sigma_{\text{data}} = 1$. With $a_t = 1/\sqrt{1 + \sigma^2}$, $\sigma_t = \sigma/\sqrt{1+\sigma^2}$ and $x_t = a_t x$: $\hat x_0 = a_t x_t - \sigma_t\hat v = \frac{1}{1 + \sigma^2}\,x - \frac{\sigma}{\sqrt{1 + \sigma^2}}\,\hat v$. So $c_{\text{skip}} = 1/(1 + \sigma^2)$ and $c_{\text{out}} = -\sigma/\sqrt{1 + \sigma^2}$ (the minus sign only flips the sign of $F$), and the network input $x_t = a_t x$ is exactly $c_{\text{in}}x$.
• In $\lambda$ form, EDM's weight in $\varepsilon$ units is $\omega(\sigma)\,\sigma^2 = (e^{-\lambda} + \sigma_{\text{data}}^2)/\sigma_{\text{data}}^2$, and its training density is $\lambda = -2\ln\sigma \sim \mathcal N(2.4, 2.4^2)$: the EDM row of the table.

Zero terminal SNR (Lin et al. 2024) #

Common schedules end with $\mathrm{SNR}(T) > 0$:

schedule ($T = 1000$)	$\sqrt{\bar\alpha_T}$	$\mathrm{SNR}(T)$
linear, $\beta_t$ from $10^{-4}$ to $0.02$	$0.0064$	$4.0 \times 10^{-5}$
cosine (offset $0.008$, $\beta_t \le 0.999$)	$4.9 \times 10^{-5}$	$2.4 \times 10^{-9}$
Stable Diffusion "scaled linear"	$0.068$	$0.0047$

• So in training, $x_T = \sqrt{\bar\alpha_T}\,x_0 + \sqrt{1 - \bar\alpha_T}\,\varepsilon$ still contains some signal. At inference, sampling starts from $x_T \sim \mathcal N(0, I)$, which contains none. This is a train–test mismatch.
• The leaked signal is mostly the lowest frequency, such as the mean of each channel. Averaging $D$ values divides the noise by $\sqrt D$ but keeps the signal. The model learns to read the mean brightness from $x_T$. At inference it reads a mean near 0, so Stable Diffusion made only images of medium brightness: it could not make a very dark or very bright image.
• The fix (Lin et al.): (1) shift and rescale $\sqrt{\bar\alpha_t}$ so that $\sqrt{\bar\alpha_T} = 0$ while $\sqrt{\bar\alpha_1}$ stays the same; (2) train with $v$-prediction; (3) start sampling at $t = T$; (4) rescale classifier-free guidance (lesson 11) to avoid over-exposure.
• Why not $\varepsilon$ at $t = T$. With $a_T = 0$, $x_T = \varepsilon$ exactly. The $\varepsilon$-target equals the input, so it is trivial, and $\hat x_0 = (x_T - \sigma_T\hat\varepsilon)/a_T$ is $0/0$: the network gives no information about $x_0$. The $v$-target there is $v = -x_0$, so a $v$-network predicts minus the conditional data mean, which is well defined.
• EDM avoids the problem differently: it starts from $\mathcal N(0, \sigma_{\max}^2 I)$ with $\sigma_{\max} = 80 \gg \sigma_{\text{data}}$, so the remaining mismatch is tiny.

Interactive figure: loading…

Answers to section 1. (1) Yes, at every noise level. Both optima are conditional means, linked by the conversions (Theorem 1). But the two averaged losses weight the levels differently, by a factor $\mathrm{SNR}$ (Theorem 2). So with finite capacity and finite training they spend their effort differently. (2) The factor is $\sigma_t/a_t \approx 0.995/0.1 \approx 10$, so the $\hat x_0$-error is about $0.5$: ten times larger. (3) It must output $x_T$ itself, because $x_T = \varepsilon$ when $a_T = 0$. The implied $\hat x_0 = (x_T - \sigma_T\hat\varepsilon)/0$ is undefined, so an $\varepsilon$-network carries no information about $x_0$ at that step; use $v$ (or $x_0$).

4. Worked example #

One VP noise level with $\bar\alpha = 0.64$:

Data: $x_0 \sim \mathcal N(0, \sigma_{\text{data}}^2)$ in one dimension with $\sigma_{\text{data}} = 0.5$.

(a) Tweedie.

1. The noisy density: $x_t = 0.8\,x_0 + 0.6\,\varepsilon$ is a sum of independent Gaussians, so $p_t = \mathcal N(0,\ 0.8^2 \cdot 0.25 + 0.6^2) = \mathcal N(0,\ 0.16 + 0.36) = \mathcal N(0, 0.52)$.
2. Observe $x_t = 1.3$. The score of $\mathcal N(0, 0.52)$ is $-x/0.52$, so it is $-1.3/0.52 = -2.5$.
3. Tweedie: $\mathbb E[x_0 \mid x_t] = (1.3 + 0.36 \cdot (-2.5))/0.8 = (1.3 - 0.9)/0.8 = 0.5$.
4. Check with Gaussian conditioning: $\mathbb E[x_0 \mid x_t] = \frac{\operatorname{Cov}(x_0, x_t)}{\operatorname{Var}(x_t)}\,x_t = \frac{0.8 \cdot 0.25}{0.52} \cdot 1.3 = 0.5$ ✓.
5. $\mathbb E[\varepsilon \mid x_t] = -\sigma_t \cdot \text{score} = -0.6 \cdot (-2.5) = 1.5$. Check: $(1.3 - 0.8 \cdot 0.5)/0.6 = 0.9/0.6 = 1.5$ ✓.
6. $\mathbb E[v \mid x_t] = a_t \cdot 1.5 - \sigma_t \cdot 0.5 = 1.2 - 0.3 = 0.9$. Check with the table: $(\hat\varepsilon - \sigma_t x_t)/a_t = (1.5 - 0.78)/0.8 = 0.9$ ✓, and back: $a_t x_t - \sigma_t \hat v = 1.04 - 0.54 = 0.5$ ✓.

(b) Loss floors. The best possible losses are the posterior variances (the lemma).

• $\operatorname{Var}(\varepsilon \mid x_t) = 1 - \frac{\operatorname{Cov}(\varepsilon, x_t)^2}{\operatorname{Var}(x_t)} = 1 - \frac{0.6^2}{0.52} = 1 - 0.6923 = 0.3077$.
• $\operatorname{Var}(x_0 \mid x_t) = 0.25 - \frac{(0.8 \cdot 0.25)^2}{0.52} = 0.25 - \frac{0.04}{0.52} = 0.1731$.
• Theorem 2 says the second is the first divided by $\mathrm{SNR}$: $0.3077 \cdot 9/16 = 0.1731$ ✓.
• Second-order Tweedie gives the same: $\frac{0.36}{0.64}\big(1 + 0.36 \cdot (-\tfrac{1}{0.52})\big) = 0.5625 \cdot 0.3077 = 0.1731$ ✓.

(c) Error conversion. Suppose the network outputs $\hat\varepsilon = 1.6$: an error of $0.1$ against the optimum $1.5$. (Theorem 2 holds for any two pairs on the line $a_t x_0 + \sigma_t\varepsilon = x_t$.)

• $\hat x_0 = (1.3 - 0.6 \cdot 1.6)/0.8 = 0.34/0.8 = 0.425$: error $-0.075$, squared $0.005625 = 0.01/\mathrm{SNR}$ ✓.
• $\hat v = (1.6 - 0.6 \cdot 1.3)/0.8 = 0.82/0.8 = 1.025$: error $0.125 = 0.1/a_t$, squared $0.015625 = 0.01 \cdot (1 + 9/16)$ ✓.

(d) The same point in EDM form. $\sigma = \sigma_t/a_t = 0.75$ and $x = x_t/a_t = 1.625$. With $\sigma_{\text{data}} = 0.5$, $\sigma^2 + \sigma_{\text{data}}^2 = 0.5625 + 0.25 = 0.8125$:

• $c_{\text{in}} = 1/\sqrt{0.8125} = 1.109$, $c_{\text{skip}} = 0.25/0.8125 = 0.3077$, $c_{\text{out}} = 0.75 \cdot 0.5/\sqrt{0.8125} = 0.375/0.9014 = 0.4160$, $c_{\text{noise}} = \tfrac14\ln 0.75 = -0.0719$.
• $\omega(\sigma) = 0.8125/(0.75 \cdot 0.5)^2 = 0.8125/0.140625 = 5.778 = 1/c_{\text{out}}^2$.
• $c_{\text{skip}}\,x = 0.3077 \cdot 1.625 = 0.5 = \mathbb E[x_0 \mid x_t]$, and $c_{\text{out}}^2 = 0.1731 = \operatorname{Var}(x_0 \mid x_t)$. For Gaussian data the skip path is already optimal: the best $F_\theta$ is $0$.
• At $\sigma = 80$: $c_{\text{out}} = 40/\sqrt{6400.25} = 0.49999$ for EDM, but $80$ for $\varepsilon$-prediction. An $F$-error of $0.01$ moves $\hat x_0$ by $0.005$ with EDM and by $0.8$ with $\varepsilon$-prediction.

5. In code #

NumPy and SciPy check the worked example, Tweedie's formula for non-Gaussian data, Theorem 2, the EDM target and Theorem 3. The notebook has these cells and the exercises.

import numpy as np
from scipy.integrate import quad
from scipy.stats import norm

a, s, sd = 0.8, 0.6, 0.5          # VP level (alpha_bar = 0.64) and sigma_data
xt = 1.3

# (a) Tweedie for Gaussian data: p_t = N(0, a^2 sd^2 + s^2)
score = -xt / (a**2 * sd**2 + s**2)
x0_hat = (xt + s**2 * score) / a
eps_hat = -s * score
v_hat = a * eps_hat - s * x0_hat
print(round(score, 4), round(x0_hat, 4), round(eps_hat, 4), round(v_hat, 4))  # -2.5 0.5 1.5 0.9

# (b) Tweedie for a two-component mixture, against Bayes' rule by quadrature
w, mu, sk = np.array([0.3, 0.7]), np.array([-1.0, 1.5]), np.array([0.2, 0.5])
xm = 0.7
var = a**2 * sk**2 + s**2                         # each component of p_t
r = w * norm.pdf(xm, a * mu, np.sqrt(var)); r /= r.sum()
score_m = -np.sum(r * (xm - a * mu) / var)        # exact score of p_t
p0 = lambda x0: np.sum(w * norm.pdf(x0, mu, sk))
joint = lambda x0: p0(x0) * norm.pdf(xm, a * x0, s)
post_mean = quad(lambda x0: x0 * joint(x0), -10, 10)[0] / quad(joint, -10, 10)[0]
print(round(post_mean, 6), round((xm + s**2 * score_m) / a, 6))  # 1.235454 1.235454

# (c) Theorem 2 on random outputs: x0- and v-errors implied by an eps-error
rng = np.random.default_rng(0)
x0, eps, eps_n = rng.normal(size=(3, 100_000))
x_t = a * x0 + s * eps
x0_n = (x_t - s * eps_n) / a                       # the same output, in x0 form
v, v_n = a * eps - s * x0, (eps_n - s * x_t) / a   # ... and in v form
L_eps, L_x0, L_v = [np.mean((p - q)**2) for p, q in [(eps, eps_n), (x0, x0_n), (v, v_n)]]
snr = a**2 / s**2
print(np.isclose(L_x0, L_eps / snr), np.isclose(L_v, L_eps * (1 + 1 / snr)))  # True True

# (d) EDM at the same point: sigma = s/a, x = x_t/a
sig = s / a
c_skip, c_out = sd**2 / (sig**2 + sd**2), sig * sd / np.sqrt(sig**2 + sd**2)
print(round(c_skip, 4), round(c_out, 4), round(c_skip * xt / a, 4))  # 0.3077 0.416 0.5
y, n = sd * rng.normal(size=100_000), rng.normal(size=100_000)
F = (y - c_skip * (y + sig * n)) / c_out           # effective target
print(round(F.var(), 2))                           # 1.0

# (e) Theorem 3: two schedules with the same end points (lambda from 10 to -10)
mmse = lambda u: sd**2 / (1 + u * sd**2)          # error of E[x0 | x_t] at SNR u
for lam, dlam in [(lambda t: 10 - 20 * t, lambda t: -20.0),
                  (lambda t: 10 - 20 * t**3, lambda t: -60 * t**2)]:
    f = lambda t: -0.5 * np.exp(lam(t)) * dlam(t) * mmse(np.exp(lam(t)))
    print(round(quad(f, 0, 1, limit=200)[0], 6))   # 4.306938 (both)
print(round(0.5 * np.log((1 + np.exp(10) * sd**2) / (1 + np.exp(-10) * sd**2)), 6))  # 4.306938

In (e), for Gaussian data the best $x_0$-error at SNR $u$ is $\mathrm{mmse}(u) = \sigma_{\text{data}}^2/(1 + u\,\sigma_{\text{data}}^2)$, so $L_\infty = \tfrac12\ln\frac{1 + \mathrm{SNR}_{\max}\sigma_{\text{data}}^2}{1 + \mathrm{SNR}_{\min}\sigma_{\text{data}}^2}$ in closed form. The linear and the cubic schedule give the same value. Zero terminal SNR for the Stable Diffusion schedule and the rescaling of Lin et al.:

import numpy as np

betas = np.linspace(0.00085**0.5, 0.012**0.5, 1000)**2   # Stable Diffusion "scaled linear"
sab = np.sqrt(np.cumprod(1 - betas))                      # sqrt(alpha_bar_t), t = 1..T
print(round(sab[-1], 4), round(sab[-1]**2 / (1 - sab[-1]**2), 4))   # 0.0683 0.0047

# The leak: channel mean of x_T for a 64 x 64 latent channel with clean mean 0.5
signal = sab[-1] * 0.5
noise_std = np.sqrt(1 - sab[-1]**2) / 64                  # mean of 4096 noise values
print(round(signal, 4), round(noise_std, 4), round(signal / noise_std, 1))  # 0.0341 0.0156 2.2

# Lin et al., Algorithm 1: shift so the last value is 0, scale so the first is kept
new = (sab - sab[-1]) * sab[0] / (sab[0] - sab[-1])
print(np.isclose(new[0], sab[0]), new[-1])                # True 0.0

A clean channel mean of $0.5$ is $2.2$ noise standard deviations away from 0 at $t = T$: easy for a network to detect.

6. Where it shows up #

• Image and video models. Stable Diffusion 1.x uses $\varepsilon$-prediction. The 768-pixel Stable Diffusion 2 model and Imagen Video use $v$-prediction. Progressive distillation (Salimans and Ho 2022) needs $v$ or $x_0$, because $\varepsilon$-prediction breaks down at high noise when only a few sampling steps are left (lesson 12).
• Flow matching and rectified flow (lesson 10). With $x_t = (1 - t)\,x_0 + t\,\varepsilon$ and the velocity target $u = \varepsilon - x_0$, the conversions are $\hat x_0 = x_t - t\,\hat u$ and $\hat\varepsilon = x_t + (1 - t)\,\hat u$: one more linear target. With $t \sim \mathcal U(0, 1)$ its weight is $w(\lambda) = \tfrac12 e^{-\lambda/2}$, the same shape as $v$ with the cosine schedule (Kingma and Gao 2023, Appendix D). Stable Diffusion 3 trains rectified flow with a logit-normal distribution of $t$ and shifts it for higher resolutions (Esser et al. 2024).
• Weather and climate. GenCast and NVIDIA's CorrDiff build on the EDM formulation: a denoiser $D_\theta(x; \sigma)$ trained over noise levels $\sigma$, and samplers on a $\sigma$ grid.
• Inverse problems and data assimilation. Diffusion posterior sampling uses the Tweedie estimate $\hat x_0(x_t)$ to approximate the likelihood of observations; second-order Tweedie gives the posterior covariance (lesson 15).
• Scientific data in practice. Standardize the data and set $\sigma_{\text{data}}$ from it; do not copy $0.5$ from image models. For long records or large grids, think about the resolution shift: more values per sample need lower SNR to destroy the coarse structure.

7. Common mistakes #

• Mixing VP and VE variables. $\sigma = \sigma_t/a_t$ and $x = x_t/a_t$. A VP-trained network inside an EDM-style sampler needs this rescaling of its input.
• Sign conventions. Here $v = a_t\varepsilon - \sigma_t x_0$ (Salimans and Ho). The rectified-flow velocity $\varepsilon - x_0$ is a different target with a different time variable.
• Converting $\varepsilon$ to $x_0$ at high noise. $\hat x_0 = (x_t - \sigma_t\hat\varepsilon)/a_t$ divides by a small $a_t$. Clip $\hat x_0$ to the data range, or use $v$ or $x_0$.
• Comparing loss values across parameterizations. They are in different units. Compare in one unit, for example the $x_0$-error at each noise level.
• "The schedule does not matter." True only for the continuous-time loss with a fixed $w(\lambda)$. It always matters for the variance of training and for sampling, and for $L_{\text{simple}}$ it is the weight.
• $\varepsilon$-prediction with a zero-terminal-SNR schedule. The last step is undefined.
• Wrong noise conditioning. Feeding a network a $t$ or $c_{\text{noise}}$ from a different schedule than in training gives wrong denoisers at every level.
• Reading $D(x)$ as a sample. The optimal denoiser is a posterior mean. At high noise it is a blurry average between modes, not a realistic clean sample.

8. Check yourself #

1. VP level with $\bar\alpha = 0.36$ ($a_t = 0.6$, $\sigma_t = 0.8$). Given $x_t = (0.5, -1.0)$ and $\hat v = (0.2, 0.4)$, find $\hat x_0$ and $\hat\varepsilon$.
   Answer

   $\hat x_0 = a_t x_t - \sigma_t\hat v = (0.3 - 0.16,\ -0.6 - 0.32) = (0.14, -0.92)$. $\hat\varepsilon = \sigma_t x_t + a_t\hat v = (0.4 + 0.12,\ -0.8 + 0.24) = (0.52, -0.56)$. Check: $0.6\,\hat x_0 + 0.8\,\hat\varepsilon = (0.084 + 0.416,\ -0.552 - 0.448) = (0.5, -1.0)$ ✓.
2. At $\mathrm{SNR} = 0.25$, an $\varepsilon$-prediction has squared error $0.04$. What are the squared errors of the implied $\hat x_0$ and $\hat v$ (VP)?
   Answer

   $\hat x_0$: $0.04/\mathrm{SNR} = 0.04/0.25 = 0.16$. $\hat v$: $0.04\,(1 + 1/0.25) = 0.04 \cdot 5 = 0.2$.
3. Which $w(\lambda)$ does the continuous-time ELBO use, and which does the $\varepsilon$-loss with $t \sim \mathcal U(0, 1)$ use?
   Answer

   The ELBO: $w(\lambda) = 1$. The $\varepsilon$-loss: $w(\lambda) = p(\lambda) = -dt/d\lambda$, the density of log-SNR values that the schedule produces ($\operatorname{sech}(\lambda/2)/(2\pi)$ for the cosine schedule).
4. Why does min-SNR-$\gamma$ use the weight $\min(1, \gamma/\mathrm{SNR})$ on the $\varepsilon$-loss?
   Answer

   It caps the weight in $x_0$ units at $\min(\mathrm{SNR}, \gamma)$. The $\varepsilon$-loss equals $\mathrm{SNR}$ times the $x_0$-loss (Theorem 2), so the $\varepsilon$-weight is $\min(\mathrm{SNR}, \gamma)/\mathrm{SNR} = \min(1, \gamma/\mathrm{SNR})$. It lowers the weight of the low-noise levels.
5. In step 3 of the EDM derivation, why minimize the target variance over $c_{\text{skip}}$?
   Answer

   $c_{\text{out}}$ scales the target to unit variance, so $c_{\text{out}}^2$ equals that variance. An error of $F_\theta$ reaches $D_\theta$ multiplied by $c_{\text{out}}$. The smallest variance gives the smallest $c_{\text{out}}$, so network errors are amplified as little as possible.
6. Explain why the channel mean leaks through $x_T$ when $\mathrm{SNR}(T) > 0$, and why it leaks more at higher resolution.
   Answer

   The channel mean of $x_T$ is $\sqrt{\bar\alpha_T}\,\bar x_0$ plus the mean of $D$ noise values, whose standard deviation is $\sqrt{1 - \bar\alpha_T}/\sqrt D$. The signal part does not shrink with $D$, but the noise part does. So with more values per channel the mean is easier to read. For Stable Diffusion's schedule and $64 \times 64$ latents, a clean mean of $0.5$ is $2.2$ noise standard deviations from 0.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Write $\hat x_0$ in terms of $x_t$ and $\hat\varepsilon$.

   Show answer

2. Define the $v$-target and recover $x_0$ from it (VP).

   Show answer

3. How is the optimal $\varepsilon$-prediction related to the score?

   Show answer

4. State Tweedie's formula for $x_t = a_t x_0 + \sigma_t\varepsilon$.

   Show answer

5. Why do the $\varepsilon$-, $x_0$- and $v$-networks have the same optimum?

   Show answer

6. An $\hat\varepsilon$ has error $\delta$. What is the error of the implied $\hat x_0$?

   Show answer

7. For one output, how is $\|x_0 - \hat x_0\|^2$ related to $\|\varepsilon - \hat\varepsilon\|^2$?

   Show answer

8. Which weight $w(\lambda)$ does the continuous-time ELBO put on the $\varepsilon$-loss, and which does $L_{\text{simple}}$ use?

   Show answer

9. Give EDM's $c_{\text{skip}}$ and $c_{\text{out}}$.

   Show answer

10. What goes wrong when $\mathrm{SNR}(T) > 0$, and what is the fix?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "$\varepsilon$, $x_0$ or $v$: which and why?" — The same optimum (Tweedie). Different weights over SNR ($\mathrm{SNR}$, $1$, $1 + \mathrm{SNR}$ in $x_0$ units) and different error amplification. $\varepsilon$ is bad at high noise, $x_0$ at low noise; $v$ (or EDM preconditioning) is safe at both ends.
• "State and prove Tweedie's formula." — $\mathbb E[x_0 \mid x_t] = (x_t + \sigma_t^2\nabla\log p_t)/a_t$. Differentiate the Gaussian convolution under the integral and use Bayes' rule.
• "Does the noise schedule matter?" — Not for the continuous-time ELBO, except its end points. Yes for the variance of training (importance sampling), for losses weighted in $t$, and for sampling.
• "Why could Stable Diffusion not make very dark images?" — $\mathrm{SNR}(T) > 0$ leaks the image mean in training; at inference the start has mean 0. Fix: zero terminal SNR, $v$-prediction, start at $t = T$.
• "What do $c_{\text{skip}}$ and $c_{\text{out}}$ do in EDM?" — They give a unit-variance target with the smallest error amplification. The skip path is the Wiener filter, and the network predicts noise at low $\sigma$ and signal at high $\sigma$.

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution:

1. Write the six conversions ($\varepsilon \leftrightarrow x_0$, $v$ from and to both, score from $\varepsilon$) and pass round-trip tests.
2. Tweedie for a mixture: compute $D(x) = x + \sigma^2\nabla\log p_\sigma(x)$ from the exact score and compare it with a Monte Carlo estimate of $\mathbb E[x_0 \mid x]$ by binning noisy samples.
3. Check the loss-weight identities of Theorem 2 on random predictions at several noise levels.
4. Write the EDM coefficients and check by Monte Carlo that the effective target has unit variance for two different data distributions.
5. Compute the continuous-time ELBO of the Gaussian toy for two schedules with the same end points (equal) and for one with a different end point (not equal).

11. Watch and read #

[M2L 2025] 5.2 Diffusion models - Sander DielemanMediterranean Machine Learning (M2L) summer school · 74 min

The chapters "How diffusion models work" and "Training and score matching" cover prediction targets and noise levels; "Frequency perspective" explains why resolution changes the schedule.

[EEML'24] Sander Dieleman - Generative modelling through iterative refinementEEML Community · 64 min

Start at the chapter "Sampling and training"; the chapter "Image spectrum and tuning" connects noise schedules to image frequencies.

Sander Dieleman - Diffusion models for image and video generation | ML in PL 2025ML in PL · 52 min

The chapters "Diffusion modeling" and "Training techniques" show how these choices are made in large latent diffusion models.

• Kingma and Gao, Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation (2023): section 2 (schedules as $\lambda(t)$ and $p(\lambda)$), section 3 (the weighted loss and its invariance), section 4 (monotone weights are ELBOs), Appendix D (the implied weights).
• Karras, Aittala, Aila, Laine, Elucidating the Design Space of Diffusion-Based Generative Models (EDM, 2022): section 5, Table 1 and Appendix B.6 (the derivation of $c_{\text{in}}, c_{\text{out}}, c_{\text{skip}}$). Reference code: NVlabs/edm (training/networks.py, training/loss.py).
• Kingma, Salimans, Poole, Ho, Variational Diffusion Models (2021): the continuous-time ELBO and its invariance to the schedule.
• Salimans and Ho, Progressive Distillation for Fast Sampling of Diffusion Models (2022): the $v$-parameterization.
• Lin, Liu, Li, Yang, Common Diffusion Noise Schedules and Sample Steps are Flawed (2024): zero terminal SNR (Algorithm 1), $v$-prediction, sampling from the last step, rescaled guidance.
• Hang et al., Efficient Diffusion Training via Min-SNR Weighting Strategy (2023).
• Hoogeboom, Heek, Salimans, Simple diffusion (2023): shifted schedules for high resolution.
• Bradley Efron, Tweedie's Formula and Selection Bias (JASA 2011, author's PDF): the formula in its statistical origin.
• Sander Dieleman, blog posts Perspectives on diffusion and Noise schedules considered harmful.