Flow matching and rectified flow

Before this lesson: lesson 04 (continuous flows and the continuity equation), lesson 08 (conversions between targets, Tweedie's formula) and lesson 09 (the probability-flow ODE, Euler sampling, DDIM = Euler, NFE). Optimal transport appears only in the last part; the ML for science track treats it in depth. Time: about 100 minutes, plus the notebook.

Why this matters #

Continuous normalizing flows (lesson 04) are trained by maximum likelihood, which needs an ODE solve and a trace estimate for every training example: slow and noisy. Flow matching (Lipman et al. 2023; Liu et al. 2023; Albergo and Vanden-Eijnden 2023, as "stochastic interpolants") removes the ODE from training. The loss is a mean squared error, as simple as the diffusion loss of lesson 07.

Rectified-flow transformers generate images in Stable Diffusion 3 (Esser et al. 2024) and Flux; Meta's Movie Gen is trained with flow matching. In science, flow matching emulates weather fields, molecules and seismic waveforms, and Riemannian flow matching works on spheres and other manifolds. Because sampling cost is (steps) × (network cost), the straightness of the paths decides the cost of every sample. Reflow, minibatch optimal transport and distillation (lesson 12) all aim at straighter paths.

1. Predict first #

2. The picture #

One pair: a straight line #

Fix one data point $x_0$ and one noise point $\varepsilon$. The straight line

starts at the data ($t = 0$) and ends at the noise ($t = 1$). Its velocity is constant: $\frac{d x_t}{dt} = \varepsilon - x_0$. If you knew the pair, one Euler step of size 1 from $\varepsilon$ would land exactly on $x_0$. This line is called the conditional path (it is conditioned on the pair), and $\varepsilon - x_0$ is the conditional velocity.

Many pairs: the lines cross #

Now draw many pairs: data points from $p_{\text{data}}$, noise points from $\mathcal N(0, I)$, independently. Their lines cross. At one place and one time, different lines move in different directions. But a network sees only the place $x_t$ and the time $t$; it does not know which pair a point came from. So it cannot follow every line.

The best guess is an average #

What is the best single velocity at the point $x$ and time $t$? Under the squared error, the best prediction is the conditional mean (lesson 08): the average of the velocities of all lines that pass through $x$ at time $t$, each weighted by how likely its pair is to be there. This average is the marginal velocity

The next figure makes this concrete in one dimension. With the two data points $\pm 1$, exactly two lines pass through each point $(t, x)$: one that ends at $+1$ and one that ends at $-1$.

3. Definitions and main results #

Flows, paths and the continuity equation #

A velocity field $v(x, t)$ defines a flow: $dx/dt = v(x, t)$. If $x(1)$ has density $p_1$ and you solve the ODE, the density $p_t$ of $x(t)$ solves the continuity equation

(lesson 04). We say that $v$ generates the path $p_t$. A probability path is a family of densities $p_t$ from the data ($p_0 = p_{\text{data}}$) to the noise ($p_1 = \mathcal N(0, I)$).

Fix the linear path: draw $x_0 \sim p_{\text{data}}$ and $\varepsilon \sim \mathcal N(0, I)$ independently and set $x_t = (1 - t)\,x_0 + t\,\varepsilon$. Its density $p_t$ is the marginal path. If we knew a velocity $u(x, t)$ that generates it, we could train a network $v_\theta$ with the flow matching (FM) loss

But we do not know $u$. The next two theorems find it and remove it from the loss.

Theorem 1 (the marginalization trick) #

The marginal velocity $v(x, t) = \mathbb E[\varepsilon - x_0 \mid x_t = x]$ generates the marginal path $p_t$.

Intuition. The density at $x$ changes because points flow in and out. Every pair moves with its own velocity, but for the flux of mass through $x$ only the average velocity of the mass at $x$ matters.

Proof (weak form; $\varphi$ is any smooth test function with compact support).

1. Each pair moves on its line, so by the chain rule $\frac{d}{dt}\varphi(x_t) = \nabla\varphi(x_t)\cdot(\varepsilon - x_0)$. Take expectations: $\frac{d}{dt}\mathbb E[\varphi(x_t)] = \mathbb E\big[\nabla\varphi(x_t)\cdot(\varepsilon - x_0)\big]$.
2. Condition on $x_t$ (the tower property): $\nabla\varphi(x_t)$ is fixed given $x_t$, so the right side is

3. In integrals: $\frac{d}{dt}\int\varphi\,p_t\,dx = \int\nabla\varphi\cdot v\,p_t\,dx = -\int\varphi\,\nabla\cdot(p_t v)\,dx$ (integration by parts; $\varphi$ vanishes far away).
4. This holds for every $\varphi$, so $\partial_t p_t = -\nabla\cdot(p_t v)$: the continuity equation. $\blacksquare$

When $v$ is regular enough (for example, Lipschitz in $x$), the ODE $dx/dt = v(x, t)$ started from $x(1) \sim \mathcal N(0, I)$ therefore has the marginals $p_t$ and ends at $p_0 = p_{\text{data}}$.

Theorem 2 (conditional flow matching, CFM) #

Define the conditional flow matching loss

Then $L_{\text{CFM}}(\theta) = L_{\text{FM}}(\theta) + V$, where $u = v$ is the marginal velocity and $V \ge 0$ does not depend on $\theta$. So the two losses have the same gradient and the same minimizer $v_\theta = v$.

Proof. Fix $t$ and write $c = \varepsilon - x_0$.

1. Expand: $\|v_\theta - c\|^2 = \|v_\theta\|^2 - 2\langle v_\theta, c\rangle + \|c\|^2$.
2. The first term is the same in both losses, because $x_t \sim p_t$ in both.
3. The cross term, by the tower property:

This is the cross term of $L_{\text{FM}}$. 4. So $L_{\text{CFM}} - L_{\text{FM}} = \mathbb E\|c\|^2 - \mathbb E\|v(x_t, t)\|^2$, which does not contain $\theta$. Written as one term:

This is the same argument as denoising score matching (lesson 06): a conditional target whose conditional mean is the marginal target. The floor $V$ is the variance of the direction given the position. It is zero only when no two lines cross. So the CFM loss never reaches zero for a model with crossing lines, even a perfect one; do not read its value as an error.

Sampling #

Training gives $v_\theta \approx v$. To sample, draw $x \sim \mathcal N(0, I)$ at $t = 1$ and integrate $dx/dt = v_\theta(x, t)$ backward to $t = 0$. With Euler on a grid $1 = t_0 > t_1 > \dots > t_n = 0$:

Each step costs one NFE, as in lesson 09. Heun and other solvers work the same way.

Why one step lands on the mean. At $t = 1$ the point $x_1 = \varepsilon$ says nothing about $x_0$, so $\mathbb E[x_0 \mid x_1] = \mathbb E[x_0]$, and $v(x, 1) = x - \mathbb E[x_0]$ (from the formula $v = (x - D)/t$ below). One step of size 1 gives $x - v(x, 1) = \mathbb E[x_0]$: the same point for every noise. The first velocity is an average direction toward the data mean, and the paths turn only later. This is the price of crossing lines.

Flow matching and diffusion #

Take a general Gaussian path $x_t = a_t\,x_0 + \sigma_t\,\varepsilon$ (lessons 08–09); the linear path has $a_t = 1 - t$ and $\sigma_t = t$. Write $D(x, t) = \mathbb E[x_0 \mid x_t = x]$ (the denoiser) and $\hat\varepsilon(x, t) = \mathbb E[\varepsilon \mid x_t = x] = (x - a_t D)/\sigma_t$.

Proposition 3 (the velocity is a denoiser). The marginal velocity of the path is

Proof.

1. The conditional velocity is $\frac{d}{dt}x_t = \dot a_t x_0 + \dot\sigma_t\varepsilon$. Its conditional mean given $x_t = x$ is $\dot a_t D + \dot\sigma_t\hat\varepsilon$, because the expectation is linear.
2. Linear path: $\dot a_t = -1$, $\dot\sigma_t = 1$, so $v = -D + (x - (1 - t)D)/t = (x - D)/t$.
3. Tweedie's formula (lesson 08) for $x = (1 - t)x_0 + t\varepsilon$ is $D = (x + t^2\nabla\log p_t(x))/(1 - t)$. Put it into step 2: $v = \big((1 - t)x - x - t^2\nabla\log p_t\big)/\big(t(1 - t)\big) = -(x + t\nabla\log p_t)/(1 - t)$. $\blacksquare$

So a velocity network, a denoiser and a score network carry the same information: they are linear maps of each other at every $t$. In lesson 08's language, the velocity is one more prediction target, with $\hat x_0 = x_t - t\,\hat v$ and $\hat\varepsilon = x_t + (1 - t)\,\hat v$.

The PF-ODE is a flow-matching ODE. For the VP path, the probability-flow ODE of lesson 09 has the drift $f - \tfrac12 g^2\nabla\log p_t$ with $f = (\dot a_t/a_t)\,x$ and $g^2 = \frac{d\sigma_t^2}{dt} - 2\frac{\dot a_t}{a_t}\sigma_t^2$. With $\nabla\log p_t = -\hat\varepsilon/\sigma_t$ this drift is $\frac{\dot a_t}{a_t}x + \big(\dot\sigma_t - \frac{\dot a_t}{a_t}\sigma_t\big)\hat\varepsilon = \frac{\dot a_t}{a_t}(x - \sigma_t\hat\varepsilon) + \dot\sigma_t\hat\varepsilon = \dot a_t D + \dot\sigma_t\hat\varepsilon$, the marginal velocity of Proposition 3. So diffusion's PF-ODE is the flow-matching ODE of its own Gaussian path. Diffusion and flow matching differ in the path ($a_t, \sigma_t$), in the loss weight over noise levels (lesson 08) and in the sampler grid, not in the kind of object they learn.

Proposition 4 (an Euler step of flow matching is a DDIM step). For the linear path, one Euler step from $t$ to $t' < t$ with the exact velocity gives the same point as the $\eta = 0$ DDIM step with the same denoiser output.

Proof. Euler: $x + (t' - t)(x - D)/t = \frac{t'}{t}x + \frac{t - t'}{t}D$. DDIM (lesson 09) with $a = 1 - t$, $\sigma = t$:

This is Theorem 3 of lesson 09 again: on the linear path, $x/a_t$ and $\sigma_t/a_t = t/(1 - t)$ are already straight-line coordinates. Flow matching with Euler steps and DDIM are one sampler on different time grids.

Straight conditional paths, curved marginal paths #

Every conditional path is a straight line. The marginal paths are not, because the average direction at a point changes with time: early (near $t = 1$) all lines through a point point toward the data mean on average; late (near $t = 0$) only the lines from one nearby data point carry weight. The marginal paths also cannot cross: an ODE with a Lipschitz velocity has exactly one solution through each point $(x, t)$, so two paths that meet at one time were the same path all along. Crossing lines are replaced by paths that bend away from each other.

Curvature costs steps. Euler is exact on a straight path with constant speed (lesson 09: a step is exact when the slope does not change along it), and it errs where the path turns. So we want a coupling of noise and data whose lines do not cross. Two ways to get one: rectified flow and optimal transport.

Rectified flow and reflow #

Let $(X_0, X_1)$ be any coupling of the data and the noise (at first, the independent one), let $v$ be its marginal velocity, and let $Z$ solve $dZ_t/dt = v(Z_t, t)$ backward from $Z_1 = X_1$. Liu, Gong and Liu (2023) call $(Z_0, Z_1)$ the rectified coupling. By Theorem 1, $Z_0$ has the data distribution, so $(Z_0, Z_1)$ is again a coupling of the same two distributions, now a deterministic one: $Z_0$ is a function of $Z_1$.

Theorem 5 (rectified flow does not increase convex transport costs). If the ODE has unique solutions, then for every convex function $c$,

Reflow. Train a new flow-matching model on the rectified pairs $(Z_0, Z_1)$ instead of independent pairs. Its lines come from a deterministic, non-crossing map, so they cross much less: $V$ is small and the new paths are straighter. Repeating gives the $k$-rectified flow. By the identity in the proof, the squared transport cost falls by $S + V$ in each round and can never fall below 0; Liu et al. derive from this that $\min_{k \le K} S(Z^k) \le \mathbb E\|X_1 - X_0\|^2/K$.

What reflow costs. Each round must simulate the current flow to produce many pairs (pairs × steps × network cost), and it fits a model to model outputs, so errors accumulate. In practice one or two rounds are used, often followed by distillation into a one-step model (lesson 12). On CIFAR-10, Liu et al. report FID 378 for one Euler step of the 1-rectified flow, and 4.85 for the 2-rectified flow plus distillation. InstaFlow (Liu et al. 2024) applies reflow and distillation to Stable Diffusion for one-step text-to-image generation.

Answers to section 1. (1) All samples land at $0$, the data mean: one Euler step gives $x - v(x, 1) = \mathbb E[x_0]$ for every noise point. (2) No. The marginal ODE has exactly one path through each point $(x, t)$; crossing training lines are replaced by curved paths that bend away from each other. (3) No. It outputs the conditional mean $\mathbb E[\varepsilon - x_0 \mid x_t]$, the average over all pairs that could have produced $x_t$. The loss that remains at the optimum is the floor $V$ of Theorem 2.

4. Worked example #

Data $x_0 \in \{-1, +1\}$ with probability $\tfrac12$ each, noise $\varepsilon \sim \mathcal N(0, 1)$, linear path.

(a) The marginal velocity at $t = 0.5$, $x = 0.3$.

1. Given $x_0 = \pm 1$, $x_t \sim \mathcal N(\pm(1 - t), t^2) = \mathcal N(\pm 0.5, 0.25)$.
2. Posterior weights by Bayes' rule (equal priors): $\log\frac{w_+}{w_-} = \frac{(x + 0.5)^2 - (x - 0.5)^2}{2 \cdot 0.25} = \frac{2x}{0.5} = 4x = 1.2$ (in general $2x(1 - t)/t^2$). So $w_+ = 1/(1 + e^{-1.2}) = 0.7685$ and $w_- = 0.2315$.
3. Conditional velocities: the line to $+1$ has $\varepsilon = (0.3 - 0.5)/0.5 = -0.4$, so $u_+ = \varepsilon - x_0 = -1.4$; the line to $-1$ has $\varepsilon = (0.3 + 0.5)/0.5 = 1.6$, so $u_- = 1.6 + 1 = 2.6$. (Check: $u_\pm = (x \mp 1)/t$.)
4. Marginal velocity: $v = 0.7685 \cdot (-1.4) + 0.2315 \cdot 2.6 = -1.0759 + 0.6018 = -0.4741$.
5. Check with the denoiser: $D = w_+ - w_- = 0.5370 = \tanh(1.2/2) = \tanh(0.6)$ ✓, and $(x - D)/t = (0.3 - 0.5370)/0.5 = -0.4741$ ✓.

The velocity is negative: moving toward $t = 0$ (sampling), $x$ grows toward the more likely data point $+1$.

(b) Euler with one and two steps. At $t = 1$, $D = \tanh(0) = 0$, so $v(x, 1) = x$.

• One step: $x_0 = \varepsilon - 1 \cdot v(\varepsilon, 1) = 0$ for every $\varepsilon$.
• Two steps: first $x_{0.5} = \varepsilon - 0.5\,\varepsilon = 0.5\,\varepsilon$. At $t = 0.5$, $D = \tanh(0.5\varepsilon \cdot 0.5/0.25) = \tanh\varepsilon$ and $v = (0.5\varepsilon - \tanh\varepsilon)/0.5 = \varepsilon - 2\tanh\varepsilon$. Then $x_0 = 0.5\varepsilon - 0.5(\varepsilon - 2\tanh\varepsilon) = \tanh\varepsilon$.
• For $\varepsilon = 1$: $\tanh 1 = 0.7616$, an error of $0.24$ against the exact end point $+1$. The mean error over $\varepsilon \sim \mathcal N(0, 1)$ is $\mathbb E[1 - \tanh|\varepsilon|] = 0.444$.

(c) Reflow in 1D. The exact flow keeps the order of points, so it sends $\varepsilon < 0$ to $-1$ and $\varepsilon > 0$ to $+1$: $Z_0 = \operatorname{sign}(Z_1)$.

• Independent pairs: $\mathbb E(\varepsilon - x_0)^2 = \mathbb E\varepsilon^2 + \mathbb E x_0^2 = 1 + 1 = 2$ (the cross term vanishes by independence).
• Rectified pairs: $\mathbb E(\varepsilon - \operatorname{sign}\varepsilon)^2 = \mathbb E\varepsilon^2 - 2\,\mathbb E|\varepsilon| + 1 = 2 - 2\sqrt{2/\pi} = 0.404$, using $\mathbb E|\varepsilon| = \sqrt{2/\pi} = 0.798$.
• The drop $2 - 0.404 = 1.596$ equals $S + V$ of the first rectification. After reflow the lines from $\varepsilon$ to $\operatorname{sign}\varepsilon$ never cross, so the new velocity is $v_2(x, t) = (x - \operatorname{sign} x)/t$, constant along each line, and one Euler step is exact.

(d) Converting a velocity. A model in the reversed convention outputs $v_{\text{paper}} = 0.8$ at $s = 0.25$ for some $x$. In this track's time $t = 1 - 0.25 = 0.75$ and $v = -0.8$. The implied clean sample at $x = 0.5$ is $\hat x_0 = x - t\,v = 0.5 + 0.75 \cdot 0.8 = 1.1$, and the implied noise is $\hat\varepsilon = x + (1 - t)v = 0.5 - 0.2 = 0.3$. Check: $(1 - t)\hat x_0 + t\hat\varepsilon = 0.275 + 0.225 = 0.5 = x$ ✓.

5. In code #

NumPy checks the worked example, Theorem 2, the Euler formulas, Proposition 4 and the minibatch OT costs. The notebook has these cells, a small trained model and the exercises.

import numpy as np
from scipy.optimize import linear_sum_assignment

# Two-point data x0 in {-1, +1}, noise eps ~ N(0, 1), path x_t = (1 - t) x0 + t eps
def v_marg(x, t):                                   # exact marginal velocity (x - D) / t
    return (x - np.tanh(x * (1 - t) / t**2)) / t

# (a) the worked example at t = 0.5, x = 0.3
t, x = 0.5, 0.3
w_plus = 1 / (1 + np.exp(-2 * x * (1 - t) / t**2))
u_plus, u_minus = (x - 1) / t, (x + 1) / t
print(round(w_plus, 4), round(w_plus * u_plus + (1 - w_plus) * u_minus, 4), round(v_marg(x, t), 4))
# 0.7685 -0.4741 -0.4741

# (b) CFM = FM + constant: shift the exact v by 0.1; the CFM loss rises by exactly 0.1^2 = 0.01
rng = np.random.default_rng(0)
n = 2_000_000
x0 = rng.choice([-1.0, 1.0], n)
eps = rng.standard_normal(n)
tt = rng.uniform(0.05, 1.0, n)                      # avoid t near 0, where v is steep
xt = (1 - tt) * x0 + tt * eps
target = eps - x0
cfm = lambda v: np.mean((v - target) ** 2)
print(round(cfm(v_marg(xt, tt)), 4), round(cfm(v_marg(xt, tt) + 0.1) - cfm(v_marg(xt, tt)), 4))
# 1.0707 0.01   (the first number is the floor V > 0; the second is the FM loss of the shifted model)

# (c) Euler from t = 1: one step gives the data mean, two steps give tanh(eps)
e = np.array([-1.5, 0.2, 1.0])
x1 = e - 1.0 * v_marg(e, 1.0)
xh = e - 0.5 * v_marg(e, 1.0)
x2 = xh - 0.5 * v_marg(xh, 0.5)
print(x1, np.round(x2, 4), np.round(np.tanh(e), 4))  # [0. 0. 0.] [-0.9051  0.1974  0.7616] (twice)

# (d) an Euler step of flow matching is a DDIM step (a_t = 1 - t, sigma_t = t)
D, x, t, t2 = 0.4, 1.3, 0.7, 0.45                    # any denoiser output, state and two times
euler = x + (t2 - t) * (x - D) / t
eps_hat = (x - (1 - t) * D) / t
ddim = (1 - t2) * D + t2 * eps_hat
print(round(euler, 6), round(ddim, 6))              # 0.978571 0.978571

# (e) minibatch OT: pair B data points with B noise points by an exact assignment
K, s = 8, 0.15
MU = np.array([[2 * np.cos(2 * np.pi * k / K), 2 * np.sin(2 * np.pi * k / K)] for k in range(K)])
for B in (1, 16, 64, 256):
    costs = []
    for _ in range(4096 // B):
        xb = MU[rng.integers(0, K, B)] + s * rng.standard_normal((B, 2))
        eb = rng.standard_normal((B, 2))
        C = ((xb[:, None] - eb[None]) ** 2).sum(-1)
        r, c = linear_sum_assignment(C)
        costs.append(C[r, c].mean())
    print(B, round(np.mean(costs), 2))
# 1 6.03 / 16 1.68 / 64 1.1 / 256 0.97   (independent pairs: E|eps|^2 + E|x0|^2 = 2 + 4.045)

The CFM loss of the exact velocity is about 1.07 here: the model is perfect, but the loss is far from 0. Shifting the velocity by 0.1 raises the loss by exactly $0.1^2$, the FM loss of the shifted model, as Theorem 2 says.

6. Where it shows up #

• Image and video models. Stable Diffusion 3 trains a rectified-flow transformer with the linear path and samples $t$ from a logit-normal distribution, shifted toward high noise for large images (Esser et al. 2024). Flux uses the same objective. Meta's Movie Gen is trained with flow matching.
• Few-step generation. Reflow, InstaFlow and other straightening methods prepare a flow for distillation into one or a few steps (lesson 12).
• Guidance and latent models. Classifier-free guidance (lesson 11) works on velocities as on scores, since they are linear maps of each other; most large flow models run in the latent space of an autoencoder (lesson 13).
• Science. Flow matching emulates weather and climate fields, generates molecules and protein structures (often with Riemannian flow matching on rotations and spheres), and samples seismic waveforms. When an emulator must produce many ensemble members, the straightness of its paths sets the compute budget.
• Likelihoods. A trained flow is a continuous normalizing flow, so its exact log-likelihood can be computed with the instantaneous change of variables of lesson 04, at the cost of an accurate ODE solve.

7. Common mistakes #

• Thinking the model learns $\varepsilon - x_0$. It learns the conditional mean. When lines cross, the best CFM loss is $V > 0$, and the marginal paths are curved although every training path is straight.
• Mixing time conventions. Papers use "data at 0" or "data at 1". Converting maps $t \mapsto 1 - t$ and flips the sign of the velocity. A sampler that runs the wrong way produces noise, not data.
• Reading the CFM loss as an error. Its floor depends on the data and the coupling. Compare samples, not loss values, across couplings or datasets.
• One large Euler step on an independent-coupling model. It lands on the data mean (blur). Use several steps, a better solver, or straighten the flow first.
• Sampling $t$ only near the ends, or uniformly without thought. The distribution of $t$ in training is a loss weight over noise levels (lesson 08). Stable Diffusion 3 found logit-normal sampling better than uniform.
• Minibatch OT done once, or reflow done many times. The OT pairing must be recomputed in every batch; each reflow round fits a model to model output, so errors accumulate.
• Dividing by $t$ at $t = 0$. The formula $v = (x - D)/t$ is singular at $t = 0$; a network that predicts $v$ directly has no problem, but code that converts from a denoiser must stop at a small $t_{\min}$ or take the last step to $D$.

8. Check yourself #

1. Write the CFM loss for the linear path, and say which quantity a perfect network outputs.
   Answer

   $L_{\text{CFM}} = \mathbb E_{t, x_0, \varepsilon}\|v_\theta((1 - t)x_0 + t\varepsilon, t) - (\varepsilon - x_0)\|^2$ with $t \sim \mathcal U[0, 1]$. A perfect network outputs the marginal velocity $v(x, t) = \mathbb E[\varepsilon - x_0 \mid x_t = x] = (x - D(x, t))/t$.
2. Show that the CFM gradient equals the FM gradient. Which step of the proof uses the tower property?
   Answer

   Expand both squares. The $\|v_\theta\|^2$ terms agree because $x_t \sim p_t$. The cross term $\mathbb E\langle v_\theta(x_t), \varepsilon - x_0\rangle$ equals $\mathbb E\langle v_\theta(x_t), \mathbb E[\varepsilon - x_0 \mid x_t]\rangle$ by the tower property, which is the FM cross term. The remaining terms do not depend on $\theta$.
3. Data $\mathcal N(0, 1)$ in 1D, noise $\mathcal N(0, 1)$, independent coupling, linear path. Compute the marginal velocity.
   Answer

   $x_t = (1 - t)x_0 + t\varepsilon$ has variance $(1 - t)^2 + t^2$. Gaussian conditioning: $D = \mathbb E[x_0 \mid x_t] = \frac{1 - t}{(1 - t)^2 + t^2}x$. So $v = (x - D)/t = \frac{(1 - t)^2 + t^2 - (1 - t)}{t\,((1 - t)^2 + t^2)}x = \frac{2t - 1}{(1 - t)^2 + t^2}x$. At $t = 1$: $v = x$; at $t = \tfrac12$: $v = 0$ (the variance is smallest there); at $t = 0$: $v = -x$.
4. Why can paths of the marginal ODE not cross, although the training lines do?
   Answer

   With a Lipschitz velocity the ODE has exactly one solution through each point $(x, t)$. If two paths met at one time, they would be the same solution at all times. The training lines are not solutions of one ODE; each follows its own pair's velocity.
5. In the proof that reflow lowers convex costs, where is convexity used, and why twice?
   Answer

   Once in time: $Z_1 - Z_0$ is the time average of $v(Z_t, t)$, so Jensen gives $c(Z_1 - Z_0) \le \int c(v)\,dt$. Once in conditional expectation: $v(X_t, t)$ is the conditional mean of $X_1 - X_0$, so $c(v) \le \mathbb E[c(X_1 - X_0) \mid X_t]$. The equality of the laws of $Z_t$ and $X_t$ links the two steps.
6. Show that for the linear path $\hat x_0 = x_t - t\,\hat v$ and $\hat\varepsilon = x_t + (1 - t)\,\hat v$.
   Answer

   From $x_t = (1 - t)x_0 + t\varepsilon$ and $v = \varepsilon - x_0$: $x_t - t v = (1 - t)x_0 + t\varepsilon - t\varepsilon + t x_0 = x_0$, and $x_t + (1 - t)v = (1 - t)x_0 + t\varepsilon + (1 - t)\varepsilon - (1 - t)x_0 = \varepsilon$. The same identities hold for the conditional means, so they hold for the predictions.

9. Interview angle #

• "Flow matching or diffusion: what is the difference?" — With Gaussian paths they learn the same object up to a linear map (velocity, denoiser, score). They differ in the path ($a_t, \sigma_t$), the loss weight over noise levels and the sampler grid. Flow matching regresses a velocity on straight lines and is simulation-free.
• "Why is the CFM loss valid if the network never sees the marginal velocity?" — The conditional target's mean given $x_t$ is the marginal velocity; by the tower property the two losses differ by a constant.
• "Why can't you sample a flow-matching model in one step?" — Lines of independent pairs cross; the learned velocity is an average; the first step heads toward the data mean. Straighten (reflow, OT couplings) or distill.
• "What does rectified flow guarantee?" — The rectified coupling has no higher cost for every convex cost, and repeated reflow makes the paths straight in the limit. It is not in general the optimal transport map in $d \ge 2$, but in 1D it is.
• "You load a flow model from another code base and get noise. First thing to check?" — The time convention and the sign of the velocity, then the noise scale of the prior.

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution:

1. Estimate the marginal velocity of the two-point data by Monte Carlo (binning $x_t$) and compare it with $(x - D)/t$.
2. Write the CFM batch (inputs and targets) and the loss; check Theorem 2 numerically. A demonstration then fits a small velocity model on two moons by least squares and samples it with Euler.
3. Write a flow-matching Euler sampler for the 8 Gaussians with the exact velocity; measure the sample error against the number of steps.
4. 1D reflow: compute the ODE map numerically for a two-Gaussian mixture and check that it equals the quantile (optimal transport) map; after reflow one step is exact.
5. Minibatch OT with linear_sum_assignment: measure how the pairing cost and the CFM floor fall with the batch size.

11. Watch and read #

• Lipman, Havasi, Holderrieth, Shaul, Le, Karrer, Chen, Lopez-Paz, Ben-Hamu, Gat, Flow Matching Guide and Code (2024): section 3.5 (the continuity equation), section 4.4 (the marginalization trick), section 4.5 (FM and CFM gradients agree), section 4.7 (the linear path and optimal transport).
• Lipman, Chen, Ben-Hamu, Nickel, Le, Flow Matching for Generative Modeling (ICLR 2023).
• Liu, Gong, Liu, Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow (ICLR 2023): Theorems 3.3 and 3.5 (marginals and convex costs), straightness and reflow.
• Albergo and Vanden-Eijnden, Building Normalizing Flows with Stochastic Interpolants (ICLR 2023).
• Tong, Fatras, Malkin, Huguet, Zhang, Rector-Brooks, Wolf, Bengio, Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport (TMLR 2024).
• Esser et al., Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (Stable Diffusion 3, 2024): section 3 (the linear path and its weighting), the logit-normal time distribution.
• Liu, Zhang, Ma, Peng, Liu, InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (ICLR 2024).
• Holderrieth and Erives, An Introduction to Flow Matching and Diffusion Models (MIT 6.S184 lecture notes, 2025).