Few-step generation: progressive distillation, consistency models and adversarial distillation

Before this lesson: lesson 08 ($v$-prediction, error amplification, EDM's $c_{\text{skip}}$ and $c_{\text{out}}$), lesson 09 (DDIM, the probability-flow ODE, the Gaussian toy, the number of function evaluations NFE), lesson 10 (rectified flow and reflow; not repeated here) and lesson 11 (classifier-free guidance doubles the NFE). Time: about 80 minutes, plus the notebook.

---

Why this matters #

A diffusion sampler costs (number of network evaluations, NFE) × (cost of one evaluation). A good ODE solver needs 10–50 NFE (lesson 09), and classifier-free guidance doubles that (lesson 11). For one image this is fine. For real-time image generation, for video, or for a weather ensemble with hundreds of members per forecast, it is not.

Few-step methods spend extra training compute to cut sampling compute to 1–4 NFE. Products such as SDXL-Turbo, latent consistency models and InstaFlow are built this way. Each method fails in its own way: blur, a quality ceiling set by the teacher, or lost diversity. In science, lost diversity means an ensemble that is too narrow, with extremes that are too rare. So you must know which failure to test for.

1. Predict first #

2. The picture #

One deterministic map from noise to data #

The probability-flow ODE (PF-ODE) maps each noise sample to one data sample (lesson 09). We use the variance-exploding (VE) form of lesson 08: $x = x_0 + \sigma n$ with $n \sim \mathcal N(0, 1)$ and noise level $\sigma$. With the denoiser $D(x; \sigma) = \mathbb E[x_0 \mid x]$, the PF-ODE is

A sampler starts at a large $\sigma_{\max}$ and solves this ODE down to $\sigma \approx 0$. It replaces the true path by many short solver steps. Each step is accurate only locally, because the path bends.

The simplest case: one data point #

If the data is a single point $y$, then $D(x; \sigma) = y$ for every input. The ODE becomes $dx/d\sigma = (x - y)/\sigma$, whose solutions are straight lines into $y$: $x(\sigma) = y + (x_{\max} - y)\,\sigma/\sigma_{\max}$. One Euler step from any noise level lands exactly on $y$. Few-step sampling is trivial.

Two data points: the path bends #

Now let the data be $-1$ or $+1$ with equal probability. The posterior weights of the two points are proportional to $e^{-(x \mp 1)^2/(2\sigma^2)}$, so the denoiser is

At high noise, $x/\sigma^2$ is small and $D \approx 0$: the denoiser cannot tell which point a sample came from, so it predicts the mean. A path first moves toward $0$. When the noise is small enough to decide, it bends toward $-1$ or $+1$. One large Euler step follows the first direction and lands near $0$: an average of the two modes, not a sample.

Interactive figure: loading…

Multimodal data bends paths, and bent paths need many steps. This is the core problem. Real data (images, weather fields) has a huge number of modes, so the bends are everywhere.

Four ways out #

• Learn a longer step. Train a student whose one step equals two teacher steps. Repeat (progressive distillation).
• Learn the end point. Train a network that maps any point of a path to its end (consistency model).
• Straighten the paths. Re-pair noise and data with the model's own map and retrain (reflow, lesson 10).
• Give up the path. Train a one-step generator whose outputs have the data distribution, with a discriminator or a score-based divergence (adversarial or distribution-matching distillation).

The first three keep (approximately) the PF-ODE map from a noise seed to a sample. So inversion (finding the seed of an image) and editing still work. The fourth only matches the distribution, so a seed no longer gives the teacher's sample.

3. Definitions and main results #

The DDIM step with an $x_0$-prediction #

Deterministic DDIM ($\eta = 0$, lesson 09) goes from time $t$ to an earlier time $u$. It uses the prediction $\hat x_0$ and the noise it implies, $\hat\varepsilon = (x_t - a_t\hat x_0)/\sigma_t$:

In words: keep the predicted clean part and the predicted noise, and mix them with the coefficients of time $u$.

Theorem 1 (the progressive-distillation target) #

Let a teacher take two DDIM steps $t \to t' \to t''$ from $x_t$ and arrive at $x_{t''}$. The student prediction $\tilde x$ for which one DDIM step $t \to t''$ also arrives at $x_{t''}$ is

Proof.

1. One DDIM step with prediction $\tilde x$ gives $x_{t''} = a_{t''}\tilde x + (\sigma_{t''}/\sigma_t)(x_t - a_t\tilde x)$.
2. Move the terms without $\tilde x$ to the left: $x_{t''} - (\sigma_{t''}/\sigma_t)\,x_t = \big(a_{t''} - (\sigma_{t''}/\sigma_t)\,a_t\big)\,\tilde x$.
3. The bracket is positive. It equals $\sigma_{t''}\big(a_{t''}/\sigma_{t''} - a_t/\sigma_t\big)$, and $a/\sigma = \sqrt{\mathrm{SNR}}$ is larger at the less noisy time $t''$. Divide by it. $\blacksquare$

Why this target is easier than $x_0$. The usual target $x_0$ is random given $x_t$: many data points could have produced $x_t$. So the best regression output is their average $\mathbb E[x_0 \mid x_t]$, which is blurry (lesson 08). The target $\tilde x$ is a deterministic function of $x_t$ (two deterministic teacher steps). So the student can fit it exactly. Salimans and Ho call this a "sharp" prediction.

The halving procedure (Salimans and Ho 2022, Algorithm 2).

1. Train a teacher $\hat x_\eta$ in the usual way (cosine schedule $a_t = \cos(\pi t/2)$), with an output that stays stable at low SNR: $x_0$, $(x_0, \varepsilon)$ or $v$ (see below).
2. Copy it: student $\theta \leftarrow \eta$. The student will take $N$ steps on the grid $t = i/N$.
3. Repeat: draw $x_0$, $i \in \{1, \dots, N\}$ and $\varepsilon$. Set $t = i/N$, $t' = t - 0.5/N$, $t'' = t - 1/N$. Run two teacher DDIM steps from $x_t$. Compute $\tilde x$ (Theorem 1). Minimize $w(\lambda_t)\,\|\tilde x - \hat x_\theta(x_t)\|^2$.
4. The student becomes the teacher ($\eta \leftarrow \theta$), and $N \leftarrow N/2$. Go to step 2.

Costs: two teacher calls and one student call per example. Salimans and Ho used 50k updates per round (100k for the 2- and 1-step rounds), starting from 8192 (CIFAR-10) or 1024 steps. Reaching 4 steps costs about as much as training the original model, and no synthetic data set is needed. The top time has $a_1 = 0$, so the student sees pure noise, exactly as at test time. Results on CIFAR-10 (FID, lower is better; lesson 14): 2.57 at 8 steps, 3.00 at 4, 4.51 at 2, 9.12 at 1. Quality holds down to 4–8 steps and then degrades.

Interactive figure: loading…

Why $\varepsilon$-prediction breaks, and what replaces it #

1. Error amplification. From $\hat\varepsilon$, the clean estimate is $\hat x_0 = (x_t - \sigma_t\hat\varepsilon)/a_t$, so an $\varepsilon$-error is multiplied by $\sigma_t/a_t = 1/\sqrt{\mathrm{SNR}}$ (lesson 08). Each halving moves the student's inputs toward low SNR. A 1-step student is only evaluated at $t = 1$.
2. No information at SNR 0. At $a_t = 0$ the input is $x_t = \varepsilon$. Predicting $\varepsilon$ is then trivial and says nothing about $x_0$; $\hat x_0 = (x_t - \sigma_t\hat\varepsilon)/a_t$ is $0/0$.
3. Zero weight where it matters. The $\varepsilon$-loss equals $\mathrm{SNR}\cdot\|x_0 - \hat x_0\|^2$ (lesson 08, Theorem 2). At $\mathrm{SNR} = 0$ it puts zero weight on the only step a 1-step model takes.
4. Fixes (Salimans and Ho, section 4). Predict $x_0$ directly; or predict $x_0$ and $\varepsilon$ in two output channels and merge them; or predict $v = a_t\varepsilon - \sigma_t x_0$, which gives $\hat x_0 = a_t x_t - \sigma_t\hat v$ and never amplifies errors. Use a loss weight that stays positive at SNR 0: "truncated SNR" $\max(\mathrm{SNR}, 1)$, or "SNR + 1" (the $v$-loss). In $x_0$ units the three weights are $\mathrm{SNR}$ ($\varepsilon$), $\max(\mathrm{SNR}, 1)$ and $1 + \mathrm{SNR}$.

Consistency models (Song, Dhariwal, Chen and Sutskever 2023) #

Use the VE form with the EDM settings of lesson 08: $\sigma_{\min} = 0.002$, $\sigma_{\max} = 80$, $\sigma_{\text{data}} = 0.5$. (Song et al. write $t$ for $\sigma$ and $\epsilon$ for $\sigma_{\min}$.)

• Consistency function. $f(x_\sigma, \sigma) = x_{\sigma_{\min}}$: the end point of the PF-ODE path that passes through $x_\sigma$ at noise level $\sigma$.
• Self-consistency. $f(x_\sigma, \sigma) = f(x_{\sigma'}, \sigma')$ for any two points of the same path.
• Boundary condition. $f(x, \sigma_{\min}) = x$.
• Parameterization. $f_\theta(x, \sigma) = c_{\text{skip}}(\sigma)\,x + c_{\text{out}}(\sigma)\,F_\theta(x, \sigma)$ with

These are EDM's coefficients (lesson 08) shifted by $\sigma_{\min}$, so that $c_{\text{skip}}(\sigma_{\min}) = 1$ and $c_{\text{out}}(\sigma_{\min}) = 0$. The boundary condition then holds for every network $F_\theta$, and $f_\theta$ stays differentiable in $\sigma$.

• Sampling. One step: draw $x \sim \mathcal N(0, \sigma_{\max}^2)$ and output $f_\theta(x, \sigma_{\max})$. Multistep (their Algorithm 1): for decreasing levels $\tau_1 > \tau_2 > \dots$, add noise again, $\hat x = x + \sqrt{\tau_n^2 - \sigma_{\min}^2}\,z$, and apply $x \leftarrow f_\theta(\hat x, \tau_n)$. Each extra step trades one NFE for quality.

Interactive figure: loading…

Consistency distillation (CD) #

Take the grid $\sigma_{\min} = \sigma_1 < \dots < \sigma_N = \sigma_{\max}$ (the Karras grid with $\rho = 7$ of lesson 09; $N = 18$ on CIFAR-10) and a pretrained score model $s_\phi(x, \sigma) \approx \nabla\log p_\sigma(x)$.

1. Draw $x_0$ from the data, $n \in \{1, \dots, N-1\}$ and $x_{\sigma_{n+1}} = x_0 + \sigma_{n+1} z$.
2. One teacher solver step back to $\sigma_n$. With Euler and $dx/d\sigma = -\sigma\, s_\phi(x, \sigma)$: $\hat x_{\sigma_n} = x_{\sigma_{n+1}} - (\sigma_n - \sigma_{n+1})\,\sigma_{n+1}\,s_\phi(x_{\sigma_{n+1}}, \sigma_{n+1})$ (Heun is better).
3. Loss: $\lambda(\sigma_n)\,d\big(f_\theta(x_{\sigma_{n+1}}, \sigma_{n+1}),\ f_{\theta^-}(\hat x_{\sigma_n}, \sigma_n)\big)$, with a distance $d$ (squared $\ell_2$, $\ell_1$ or LPIPS) and $\lambda \equiv 1$.
4. $\theta^-$ is a stop-gradient exponential moving average of $\theta$ (a target network, as in DQN and BYOL): $\theta^- \leftarrow \mathrm{stopgrad}(\mu\theta^- + (1 - \mu)\theta)$.

The PF-ODE in score form is the same ODE as before: Tweedie gives $D = x + \sigma^2\nabla\log p_\sigma$, so $(x - D)/\sigma = -\sigma\nabla\log p_\sigma$.

Theorem 2 (zero CD loss gives the teacher's consistency function) #

This is Song et al., Theorem 1. Assume $f_\theta(\cdot, \sigma)$ is $L$-Lipschitz for every $\sigma$, the teacher's solver has local error $O(h^{p+1})$ with $p \ge 1$ for step size $h$, the weight $\lambda(\sigma) > 0$, the distance satisfies $d(x, y) = 0 \Leftrightarrow x = y$, and the CD loss is zero with $\theta^- = \theta$. Let $f$ be the consistency function of the teacher's PF-ODE and $\Delta = \max_n(\sigma_{n+1} - \sigma_n)$. Then

Why this is true (intuition). The boundary condition fixes the error to $0$ at $\sigma_{\min}$. Zero loss says that each grid level copies the next lower one, through one solver step. So the error at a level is the error one level down plus one local solver error. Adding up $N$ local errors of size $O(h^{p+1})$ gives $O(\Delta^p)$, exactly as for the global error of an ODE solver.

Proof (along one teacher path with exact points $x_{\sigma_n}$)

1. Let $e_n = f_\theta(x_{\sigma_n}, \sigma_n) - f(x_{\sigma_n}, \sigma_n)$. The boundary condition gives $e_1 = 0$, because both terms equal $x_{\sigma_{\min}}$.
2. Zero loss: the density of $x_{\sigma_{n+1}} = x_0 + \sigma_{n+1}z$ is positive everywhere and $\lambda > 0$, so $d = 0$ at every point. Hence $f_\theta(x_{\sigma_{n+1}}, \sigma_{n+1}) = f_\theta(\hat x_{\sigma_n}, \sigma_n)$, where $\hat x_{\sigma_n}$ is one solver step from $x_{\sigma_{n+1}}$. Self-consistency of $f$ gives $f(x_{\sigma_{n+1}}, \sigma_{n+1}) = f(x_{\sigma_n}, \sigma_n)$.
3. Subtract the two equations: $e_{n+1} = \big[f_\theta(\hat x_{\sigma_n}, \sigma_n) - f_\theta(x_{\sigma_n}, \sigma_n)\big] + e_n$.
4. Lipschitz and local error: the bracket is at most $L\,|\hat x_{\sigma_n} - x_{\sigma_n}| = L\cdot O\big((\sigma_{n+1} - \sigma_n)^{p+1}\big)$.
5. Sum the steps: $|e_n| \le L \sum_k O\big((\sigma_{k+1} - \sigma_k)^{p+1}\big) \le L\,O(\Delta^p)\sum_k (\sigma_{k+1} - \sigma_k) = O(\Delta^p)\,(\sigma_{\max} - \sigma_{\min})$. $\blacksquare$

Two consequences. The boundary condition anchors the argument: without it, a constant $f_\theta$ has zero loss. And CD can at best reproduce the teacher's ODE, so the teacher sets a quality ceiling. Results on CIFAR-10: FID 3.55 at 1 step and 2.93 at 2 steps (progressive distillation: 8.34 and 5.58 in the same comparison).

Consistency training (CT): no teacher #

Replace the teacher score in step 2 by the one-sample estimate $-(x_{\sigma_{n+1}} - x_0)/\sigma_{n+1}^2$. It is unbiased given $x_{\sigma_{n+1}}$, because $\nabla\log p_\sigma(x) = -\mathbb E[(x - x_0)/\sigma^2 \mid x]$ (Tweedie, lesson 08). With $x_{\sigma_{n+1}} - x_0 = \sigma_{n+1} z$, the Euler step becomes

So CT compares two points that share the same noise $z$: $\lambda\,d\big(f_\theta(x_0 + \sigma_{n+1} z, \sigma_{n+1}),\ f_{\theta^-}(x_0 + \sigma_n z, \sigma_n)\big)$.

• Song et al. prove (their Theorem 2) that $\mathcal L_{\text{CD}} = \mathcal L_{\text{CT}} + o(\Delta)$ for an exact teacher score with Euler steps (assuming $d$ and $f_{\theta^-}$ are twice continuously differentiable with bounded second derivatives). The CT pair is a noisy estimate of the ODE step: right on average, but with variance.
• Small $N$ (large steps) means low variance but high bias; large $N$ the opposite. CT therefore grows $N$ during training (a curriculum). CIFAR-10: FID 8.70 at 1 step.
• Improved CT (Song and Dhariwal 2023): $\theta^- = \mathrm{stopgrad}(\theta)$ without the moving average, the pseudo-Huber distance $\sqrt{\|x - y\|^2 + c^2} - c$ instead of LPIPS, log-normal sampling of noise levels, and doubling $N$ on a schedule. CIFAR-10: FID 2.51 at 1 step, better than CD. Without a teacher there is no teacher ceiling.

	CD	CT
needs	pretrained diffusion model	data only
pair for the loss	ODE step of the teacher	same noise $z$ at two levels
quality ceiling	the teacher's ODE (Theorem 2)	none from a teacher
main difficulty	teacher cost per step	variance of the target, $N$ schedule

Continuous-time consistency models and flow maps

• Continuous-time CMs (Lu and Song 2024, sCM): let $N \to \infty$; the loss then needs the time derivative of $f_\theta$ along the path, computed with a Jacobian–vector product. With stabilized training they scaled to 1.5B parameters and reached a 2-step FID of 1.88 on ImageNet $512 \times 512$, within 10% of the best diffusion models of that time.
• Flow maps. A consistency function jumps from $\sigma$ to $\sigma_{\min}$. A flow map $X_{s, t}(x)$ jumps from any time $s$ to any other time $t$ along the ODE. Consistency models, progressive distillation and "mean-flow" models (2025) are special cases or relatives. They can be trained by self-distillation, without a separate teacher (see the Boffi video below).

Adversarial distillation, distribution matching and reflow #

• Adversarial diffusion distillation (ADD, Sauer et al. 2023; SDXL-Turbo). The student starts from the teacher. Its loss is a hinge GAN loss (lesson 05) with a discriminator made of frozen DINOv2 ViT-S features and small trainable heads (R1 penalty), plus a distillation loss. For the distillation loss, add noise to the student's output, let the teacher denoise it, and pull the student's output toward the teacher's result (a form of score distillation). It gives sharp $512 \times 512$ images in 1–4 steps. The authors report lower sample diversity than the teacher.
• Distribution matching distillation (DMD, Yin et al. 2024): a one-step generator whose KL gradient is the difference of two scores (the teacher's, and a score model fitted to the generator's own samples), plus a regression loss.
• Latent consistency models (LCM, Luo et al. 2023): consistency distillation in the latent space of Stable Diffusion (lesson 13), with classifier-free guidance built into the teacher ODE; 2–4 steps.
• Reflow (lesson 10). Simulate the current flow to get noise–data pairs $(Z_0, Z_1)$, then retrain flow matching on these pairs. Transport costs do not increase and paths become straighter; then distill into one step. On CIFAR-10, a 1-rectified flow sampled with one Euler step has FID 378; 2-rectified flow plus distillation reaches 4.85. InstaFlow (Liu et al. 2024) applies reflow and distillation to Stable Diffusion: one step, FID 23.3 on MS COCO 2017-5k (progressive distillation: 37.2), for 199 A100 GPU-days.

Comparison (as of 2024).

method	needs	objective	typical failure
progressive distillation	teacher, $\log_2(N/K)$ rounds	regress onto two teacher steps	blur at 1–2 steps
consistency distillation	teacher, target network	self-consistency of adjacent ODE points	capped by the teacher
consistency training	data only	same, with a shared-noise pair	sensitive to the $N$ schedule and to $d$
adversarial distillation	teacher, discriminator	GAN loss + score distillation	lost diversity, GAN instability
reflow + distillation	generated pairs	flow matching on the model's coupling	cost of the pairs, accumulated errors

The general pattern: regression objectives blur at 1–2 steps. They average a multimodal target that the student cannot fit exactly. Distribution-level objectives are sharp but mode-seeking: they lose diversity.

Answers to section 1. (1) Near $0$: one Euler step from $\sigma$ to $0$ lands on $D(1.2; 3) = \tanh(1.2/9) = 0.133$, the posterior mean, not a data point. (2) $\log_2(1024/4) = 8$ rounds. (3) A constant, for example $f_\theta \equiv 0$: it is consistent everywhere. The boundary condition $f_\theta(x, \sigma_{\min}) = x$, built into $c_{\text{skip}}$ and $c_{\text{out}}$, rules it out.

4. Worked example #

(a) A progressive-distillation target. 1-D data $x_0 \sim \mathcal N(0, \sigma_{\text{data}}^2)$ with $\sigma_{\text{data}} = 0.5$. The teacher is the exact denoiser $\hat x_0(x_t) = a\,\sigma_{\text{data}}^2 x_t/(a^2\sigma_{\text{data}}^2 + \sigma^2)$ (VP, level with coefficients $a, \sigma$). Three levels: $t$ with $(a, \sigma) = (0.6, 0.8)$, $t'$ with $(0.8, 0.6)$, $t''$ with $(0.96, 0.28)$. Start at $x_t = 1$.

1. Teacher at $t$: $\hat x_0 = 0.6 \cdot 0.25 \cdot 1/(0.36 \cdot 0.25 + 0.64) = 0.15/0.73 = 0.2055$.
2. DDIM to $t'$: $x_{t'} = 0.8 \cdot 0.2055 + (0.6/0.8)(1 - 0.6 \cdot 0.2055) = 0.1644 + 0.6575 = 0.8219$.
3. Teacher at $t'$: $\hat x_0 = 0.8 \cdot 0.25 \cdot 0.8219/(0.64 \cdot 0.25 + 0.36) = 0.1644/0.52 = 0.3161$.
4. DDIM to $t''$: $x_{t''} = 0.96 \cdot 0.3161 + (0.28/0.6)(0.8219 - 0.8 \cdot 0.3161) = 0.3035 + 0.2655 = 0.5690$.
5. Target: $\sigma_{t''}/\sigma_t = 0.28/0.8 = 0.35$, so $\tilde x = (0.5690 - 0.35 \cdot 1)/(0.96 - 0.35 \cdot 0.6) = 0.2190/0.75 = 0.2920$.
6. Check: one DDIM step with $\tilde x$ gives $0.96 \cdot 0.2920 + 0.35\,(1 - 0.6 \cdot 0.2920) = 0.2803 + 0.2887 = 0.5690$ ✓. One step with the teacher's own $\hat x_0 = 0.2055$ gives only $0.5041$.
7. The exact PF-ODE solution at $t''$: in VE coordinates $x/a$ with $\sigma_{\text{VE}} = \sigma/a$, the Gaussian-toy paths are $x(\sigma) \propto \sqrt{\sigma_{\text{data}}^2 + \sigma^2}$ (lesson 09). Here $\sigma_{\text{VE}}$ goes from $0.8/0.6 = 1.3333$ to $0.28/0.96 = 0.2917$, so $x_{t''} = 0.96 \cdot (1/0.6) \cdot \sqrt{0.25 + 0.2917^2}/\sqrt{0.25 + 1.3333^2} = 0.6504$. The student copies the teacher's two-step error ($0.5690$ instead of $0.6504$). This is why distillation starts from a teacher with many steps.
8. Note that $\tilde x = 0.292$ is larger than the posterior mean $0.2055$: the student's prediction is "sharper" than the average.

(b) Error amplification. At $a_t = 0.05$ ($\mathrm{SNR} = 0.0025$, $\sigma_t = 0.99875$), an $\varepsilon$-error of $0.01$ becomes an $x_0$-error of $0.01 \cdot 0.99875/0.05 = 0.20$. A $v$-error of $0.01$ becomes an $x_0$-error of $0.01 \cdot 0.99875 = 0.010$.

(c) A consistency function in closed form. Same Gaussian data, now in the VE form. The PF-ODE is $dx/d\sigma = \sigma x/(\sigma_{\text{data}}^2 + \sigma^2)$, with solutions $x(\sigma) \propto \sqrt{\sigma_{\text{data}}^2 + \sigma^2}$. So

1. From $x = 40$ at $\sigma = 80$: $f = 40 \cdot 0.500004/80.0016 = 0.25000$.
2. The same path passes $\sigma = 1$ at $x = 40 \cdot \sqrt{1.25}/80.0016 = 0.5590$, and $f(0.5590, 1) = 0.5590 \cdot 0.500004/1.1180 = 0.25000$: self-consistency ✓.
3. Coefficients: $c_{\text{skip}}(\sigma_{\min}) = 1$, $c_{\text{out}}(\sigma_{\min}) = 0$; $c_{\text{skip}}(1) = 0.2006$, $c_{\text{out}}(1) = 0.4463$; $c_{\text{skip}}(80) = 3.9 \times 10^{-5}$, $c_{\text{out}}(80) = 0.49998$. At high noise the output is almost all network.
4. One Euler step of CD from $\sigma = 1$ to $0.5$ with the exact score: $0.5590 - 0.5 \cdot (0.5590 - 0.25 \cdot 0.5590/1.25)/1 = 0.5590 - 0.5 \cdot 0.4472 = 0.3354$. The exact ODE point is $0.5590 \cdot \sqrt{0.5}/\sqrt{1.25} = 0.3536$. A perfect CD fit would map $0.5590$ to $f(0.3354, 0.5) = 0.2372$ instead of $0.25$: the local solver error enters the target, as in Theorem 2. Smaller steps (larger $N$) or Heun shrink it.
5. A CT pair with $x_0 = 0.3$, $z = 0.7$: $(x_0 + 1 \cdot z,\ x_0 + 0.5 z) = (1.0, 0.65)$. Given $x_\sigma = 1$ at $\sigma = 1$, $\mathbb E[x_0 \mid x_\sigma] = 0.25/1.25 = 0.2$, so the expected CT point is $0.2 + 0.5\,(1 - 0.2) = 0.6$. This is exactly the Euler point $1 - 0.5 \cdot (1 - 0.2)/1 = 0.6$ ✓.

5. In code #

NumPy and SciPy check the worked example: Theorem 1, the consistency function of the Gaussian toy, the CT pair as a noisy Euler step, and why one-step regression blurs. The notebook has these cells and the exercises.

import numpy as np
from scipy.integrate import solve_ivp

# (a) Theorem 1 on the Gaussian toy N(0, 0.5^2), VP levels (a, sigma).
sd = 0.5
teacher = lambda x, a, s: a * sd**2 * x / (a**2 * sd**2 + s**2)        # E[x0 | x_t]
ddim = lambda x, x0, a, s, a2, s2: a2 * x0 + s2 * (x - a * x0) / s
(a0, s0), (a1, s1), (a2, s2) = (0.6, 0.8), (0.8, 0.6), (0.96, 0.28)    # t, t', t''
xt = 1.0
x_mid = ddim(xt, teacher(xt, a0, s0), a0, s0, a1, s1)                  # teacher step 1
x_end = ddim(x_mid, teacher(x_mid, a1, s1), a1, s1, a2, s2)            # teacher step 2
r = s2 / s0
target = (x_end - r * xt) / (a2 - r * a0)                              # Theorem 1
print(f"{x_mid:.4f} {x_end:.4f} {target:.4f}")                         # 0.8219 0.5690 0.2920
print(f"{ddim(xt, target, a0, s0, a2, s2):.4f}")                       # 0.5690: one student step
print(f"{ddim(xt, teacher(xt, a0, s0), a0, s0, a2, s2):.4f}")          # 0.5041: one teacher step

# (c) Consistency function of the Gaussian toy, VE form, and self-consistency.
s_min = 0.002
D = lambda x, s: sd**2 * x / (sd**2 + s**2)
ode = lambda s, x: (x - D(x, s)) / s
f_cf = lambda x, s: x * np.sqrt(sd**2 + s_min**2) / np.sqrt(sd**2 + s**2)
path = solve_ivp(ode, (80.0, s_min), [40.0], t_eval=[80.0, 10.0, 1.0, 0.1], rtol=1e-10, atol=1e-12)
print(np.round([f_cf(x, s) for x, s in zip(path.y[0], path.t)], 6))     # [0.249997 0.249997 0.249997 0.249997]

c_skip = lambda s: sd**2 / ((s - s_min)**2 + sd**2)
c_out = lambda s: sd * (s - s_min) / np.sqrt(sd**2 + s**2)
print(c_skip(s_min), c_out(s_min), f"{c_skip(1.0):.4f} {c_out(1.0):.4f}")  # 1.0 0.0 0.2006 0.4463

# CT: given x_t1, the partner x0 + t0 z is random; its mean is the exact-score Euler point.
rng = np.random.default_rng(0)
t1, t0, x_t1 = 1.0, 0.5, 1.0
post_sd = np.sqrt(sd**2 * t1**2 / (sd**2 + t1**2))                     # std of x0 | x_t1
x0 = D(x_t1, t1) + post_sd * rng.standard_normal(1_000_000)
z = (x_t1 - x0) / t1
ct = x0 + t0 * z
print(f"{ct.mean():.3f} {x_t1 + (t0 - t1) * ode(t1, x_t1):.3f} {ct.std():.3f}")  # 0.600 0.600 0.224

Why one-step regression blurs: data $\pm 1$, the exact ODE map (what a consistency model must learn) against the best one-step regression onto $x_0$ (the posterior mean at $\sigma_{\max}$):

import numpy as np
from scipy.integrate import solve_ivp

s_min, s_max = 0.002, 80.0
D = lambda x, s: np.tanh(x / s**2)                         # E[x0 | x] for data -1 or +1
ode = lambda s, x: (x - D(x, s)) / s
rng = np.random.default_rng(0)
x_T = rng.choice([-1.0, 1.0], 2000) + s_max * rng.standard_normal(2000)
f = solve_ivp(ode, (s_max, s_min), x_T, method="LSODA", rtol=1e-8, atol=1e-10).y[:, -1]
reg = D(x_T, s_max)
print(f"ODE map:    mean |x| = {np.abs(f).mean():.3f}, share > 0 = {(f > 0).mean():.3f}")
print(f"regression: mean |x| = {np.abs(reg).mean():.3f}, max |x| = {np.abs(reg).max():.3f}")
# ODE map:    mean |x| = 1.000, share > 0 = 0.482   -> samples at -1 and +1
# regression: mean |x| = 0.010, max |x| = 0.040   -> every "sample" sits near the data mean 0

6. Where it shows up #

• Text-to-image products. SDXL-Turbo (adversarial diffusion distillation, 1–4 steps), latent consistency models and LCM-LoRA (2–4 steps), InstaFlow (reflow and distillation, 1 step). Many distilled models fold classifier-free guidance into the student (guided distillation, LCM), so each step costs one network call instead of two.
• Ensembles in science. The sampling cost is members × steps × calls per step. Weather ensembles (GenCast samples each member with a diffusion model) and hazard studies that need thousands of realizations per scenario are exactly this case.
• The science failure mode. A sharper, mode-seeking student gives an under-dispersed ensemble: too little spread, extremes too rare. After any distillation, check calibration again with rank histograms, the spread–skill ratio and recall (lesson 14), not only a sample-quality score.
• Inverse problems. Consistency models allow zero-shot inpainting and super-resolution (Song et al., section 3). Guidance methods that need many small steps (lesson 15) fit less well with 1–2 steps.
• Practice. Before you retrain, try a better solver: Heun or DPM-Solver at 10–20 NFE needs no training (lesson 09).

7. Common mistakes #

• Distilling with $\varepsilon$-prediction and the $\varepsilon$-loss weight. The weight is zero at SNR 0 and errors are amplified by $1/\sqrt{\mathrm{SNR}}$. Use $v$ or $x_0$ with a truncated-SNR or SNR + 1 weight.
• Dropping the boundary condition. Without $c_{\text{skip}}(\sigma_{\min}) = 1$, $c_{\text{out}}(\sigma_{\min}) = 0$, a constant function has zero consistency loss.
• Expecting a distilled student to beat its teacher. Progressive distillation and CD approximate the teacher's sampler or ODE (Theorem 2). Only teacher-free methods (CT) or methods with an extra data signal (adversarial losses on real data) can exceed it.
• Comparing 1-step FIDs without context. Report the teacher, the NFE of every method, and the training distance. LPIPS in training can bias an evaluation with Inception-based features (one reason improved CT replaced it).
• Many rounds of reflow. Each round needs a new generated set of pairs and fits model output, so errors accumulate. Use one or two rounds, then distill.
• Judging a fast model by sharpness alone. Check diversity (recall, ensemble spread). A GAN-type loss can collapse modes.

8. Check yourself #

1. VP levels $t$: $(a, \sigma) = (0.8, 0.6)$ and $t''$: $(0.96, 0.28)$. From $x_t = 0.5$ the teacher's two DDIM steps reach $x_{t''} = 0.45$. Compute the student target $\tilde x$ and check it.
   Answer

   $\sigma_{t''}/\sigma_t = 0.28/0.6 = 0.4667$. $\tilde x = (0.45 - 0.4667 \cdot 0.5)/(0.96 - 0.4667 \cdot 0.8) = 0.2167/0.5867 = 0.3693$. Check: $0.96 \cdot 0.3693 + 0.4667\,(0.5 - 0.8 \cdot 0.3693) = 0.3545 + 0.0955 = 0.45$ ✓.
2. Why can the student fit $\tilde x$ sharply, when a network trained on $x_0$ predicts a blurry average?
   Answer

   Given $x_t$, the target $x_0$ is random (many data points are consistent with $x_t$), so the squared-loss optimum is $\mathbb E[x_0 \mid x_t]$. The target $\tilde x$ is a deterministic function of $x_t$ (two deterministic teacher steps), so zero error is possible.
3. With $\sigma_{\text{data}} = 0.5$ and $\sigma_{\min} = 0.002$, compute $c_{\text{skip}}(0.5)$ and $c_{\text{out}}(0.5)$. Why must $c_{\text{out}}(\sigma_{\min}) = 0$?
   Answer

   $c_{\text{skip}} = 0.25/(0.498^2 + 0.25) = 0.5020$; $c_{\text{out}} = 0.5 \cdot 0.498/\sqrt{0.5} = 0.3521$. With $c_{\text{skip}}(\sigma_{\min}) = 1$ and $c_{\text{out}}(\sigma_{\min}) = 0$, $f_\theta(x, \sigma_{\min}) = x$ for every network $F_\theta$. The boundary condition then holds by construction. It rules out the constant solution and anchors the error bound of Theorem 2.
4. Show that the consistency-training pair is an Euler step of the PF-ODE with a one-sample score estimate.
   Answer

   With $x_{\sigma_{n+1}} = x_0 + \sigma_{n+1}z$, the estimate is $\hat s = -(x_{\sigma_{n+1}} - x_0)/\sigma_{n+1}^2 = -z/\sigma_{n+1}$. Euler: $x_{\sigma_{n+1}} - (\sigma_n - \sigma_{n+1})\,\sigma_{n+1}\hat s = x_0 + \sigma_{n+1}z + (\sigma_n - \sigma_{n+1})z = x_0 + \sigma_n z$. Since $\mathbb E[\hat s \mid x_{\sigma_{n+1}}]$ is the true score, the CT point equals the exact-score Euler point on average.
5. A team needs 1,000-member ensembles; today each member costs 25 steps with classifier-free guidance. What does a 4-step distilled model with guidance folded in save, and what must they check?
   Answer

   $1000 \cdot 25 \cdot 2 = 50{,}000$ calls against $1000 \cdot 4 = 4{,}000$: 12.5 times fewer. They must check that the ensemble is still calibrated (rank histograms, spread–skill ratio), that extremes and spectra match held-out data, and diversity (recall). First they should compare with a training-free solver at 10–20 NFE.
6. On CIFAR-10, one Euler step of a 1-rectified flow gives FID 378, but 2-rectified flow plus distillation gives 4.85. Explain.
   Answer

   With the independent coupling of noise and data, the marginal paths are curved. One Euler step follows the initial velocity, an average direction, and lands near conditional means (blur). Reflow retrains on the model's own deterministic coupling, which makes the paths much straighter (lesson 10), so a one-step distilled student can match the map.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Why do multimodal data distributions need many PF-ODE solver steps?

   Show answer

2. Data $\pm 1$, VE form. Where does one Euler step from $(\sigma, x)$ to $\sigma = 0$ land?

   Show answer

3. What is the idea of progressive distillation?

   Show answer

4. Give the progressive-distillation target $\tilde x$.

   Show answer

5. Why does a 1-step distilled student need $v$- or $x_0$-prediction?

   Show answer

6. Define the consistency function of a PF-ODE.

   Show answer

7. How do consistency models enforce the boundary condition $f_\theta(x, \sigma_{\min}) = x$?

   Show answer

8. What pair of points does consistency training compare, and why does it work?

   Show answer

9. Name two few-step families that keep the seed-to-sample map.

   Show answer

10. What do adversarial and distribution-matching distillation give up?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "How would you make a diffusion model 10× faster at inference?" — First a better solver (training-free, 10–20 NFE). Then distillation: progressive or consistency distillation to 4–8 steps; adversarial or distribution matching for 1–2 steps. Measure quality and diversity at equal NFE.
• "Derive the progressive-distillation target." — Write one DDIM step with prediction $\tilde x$, set it equal to the teacher's two-step result, solve (Theorem 1).
• "Why can't you distill an $\varepsilon$-prediction model to one step?" — At SNR 0 the input is pure noise, $\varepsilon$ is trivial, $\hat x_0$ divides by $a_t = 0$, and the $\varepsilon$-loss weight on $x_0$ is SNR = 0.
• "What is a consistency model, and how is the boundary condition enforced?" — A map from any point of a PF-ODE path to its end; $c_{\text{skip}}(\sigma_{\min}) = 1$, $c_{\text{out}}(\sigma_{\min}) = 0$.
• "CD or CT?" — CD needs a teacher and is capped by it; CT needs only data, has a noisier target and needs an $N$ curriculum; improved CT beat CD on CIFAR-10.

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution:

1. Write an Euler sampler for the PF-ODE and show that one step on two-point data lands near $0$, while many steps land on $\pm 1$.
2. Write the progressive-distillation target and check that one student step equals two teacher DDIM steps on the Gaussian toy.
3. Write the closed-form consistency function of the Gaussian toy and check self-consistency and the boundary condition against a numerical ODE solution.
4. Consistency training without a teacher: build CT pairs with shared noise and fit a small parametric consistency model, level by level, on two-mode data; sample in one step.
5. Plot NFE against error for plain DDIM and for an ideal distilled student on the Gaussian toy.

11. Watch and read #

CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow MatchingKelly He · 71 min

A full lecture on this lesson: progressive distillation, consistency distillation and training, and flow maps.

How to build a consistency model: Learning flow maps via self-distillation | Nicholas BoffiValence Labs · 76 min

A research talk that frames consistency models as flow maps and trains them by self-distillation; watch after the lesson.

The Rise of Single-Step Generative Models [MeanFlow]Jia-Bin Huang · 10 min

A short overview of newer one-step models (2025) that build on the ideas of this lesson.

• Salimans and Ho, Progressive Distillation for Fast Sampling of Diffusion Models (ICLR 2022): section 3 and Algorithm 2 (the procedure), section 4 (parameterizations, truncated-SNR and SNR + 1 weights), Table 1 (ablation), Appendix G (the target).
• Song, Dhariwal, Chen and Sutskever, Consistency Models (ICML 2023): section 3 (definition, Algorithm 1), section 4 (Theorem 1, Algorithm 2), section 5 (Theorem 2, Algorithm 3), Appendix C (the boundary-condition coefficients).
• Song and Dhariwal, Improved Techniques for Training Consistency Models (2023).
• Lu and Song, Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models (2024).
• Sauer, Lorenz, Blattmann and Rombach, Adversarial Diffusion Distillation (2023).
• Yin et al., One-step Diffusion with Distribution Matching Distillation (CVPR 2024).
• Liu et al., InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (ICLR 2024).
• Reference code: openai/consistency_models (cm/karras_diffusion.py: get_scalings_for_boundary_condition, consistency_losses, progdist_losses).
• Sander Dieleman, The paradox of diffusion distillation (2024): a clear tour of distillation methods and of the question "where does the free lunch come from?".