Normalizing flows: change of variables, coupling, autoregressive and continuous flows

Before this lesson: lesson 01 (maximum likelihood) and lesson 03 (MADE, used inside MAF and IAF). From other tracks: change of variables for densities and Jacobians (calculus), Hutchinson's trace estimator (calculus), and the determinant (linear algebra). Time: about 90 minutes, plus the notebook.

---

Why this matters #

Flows are the second family with an exact likelihood, after autoregressive models (lesson 03). Unlike AR models, a coupling flow samples in one pass. Flows are standard tools for density estimation and variational inference (flexible posteriors), for simulation-based inference (MAF and neural spline flows estimate posteriors of simulator parameters), and for sampling physical systems (Boltzmann generators). The continuous version matters even more today: the probability-flow ODE of diffusion models and the ODE learned by flow matching (lesson 10) are continuous flows, and the instantaneous change of variables of this lesson is how you compute their exact log-likelihood.

1. Predict first #

2. The picture #

One affine map in 1D #

Let $z \sim \mathcal N(0, 1)$ and $x = az + b$ with $a \ne 0$. An interval of length $\ell$ in $z$-space becomes an interval of length $|a|\ell$ in $x$-space, and it holds the same probability. The same mass on a longer interval means a lower density, so the density is divided by $|a|$:

Now write $f(x) = (x - b)/a$ for the map from data to base. Then $|f'(x)| = 1/|a|$, and the formula reads: the density of $x$ is the base density at $f(x)$, times $|f'(x)|$.

A curved map. The same holds point by point for any increasing map $f$: near $x$, the map looks like a line with slope $f'(x)$. Where $f$ is steep, a small interval of $x$ covers a large interval of $z$, so it collects a lot of probability: the density of $x$ is high. Where $f$ is flat, the density of $x$ is low.

Interactive figure: loading…

A flow is many such maps #

A normalizing flow makes $f$ a composition of invertible layers with learnable parameters: $f = f_K \circ \dots \circ f_1$. It is used in two directions.

• Density direction (data $\to$ base, $f$): compute $z = f(x)$ and add up the $\log|\det J|$ of each layer. This gives $\log p_X(x)$, and the training loss.
• Sampling direction (base $\to$ data, $f^{-1}$): draw $z \sim \mathcal N(0, I)$ and invert the layers in reverse order.

In this track, $f$ always goes from data to base, $z = f(x)$. The review by Papamakarios et al. names the sampling map $T$, with $x = T(u)$: it is the same object seen from the other side, $T = f^{-1}$ and $u = z$.

The design problem. Each layer must be (1) invertible, (2) cheap in $\log|\det J|$ (a general $D \times D$ determinant costs $O(D^3)$), and (3) expressive. There are two answers. Make the Jacobian triangular (coupling and autoregressive layers), or make the flow continuous in time, where the log-determinant becomes an integral of a trace.

Coupling layers bend space one coordinate at a time #

An affine coupling layer keeps half of the coordinates and uses them to shift and scale the other half. In 2D: keep $x_1$; replace $x_2$ by $x_2 e^{s(x_1)} + t(x_1)$. The next layer keeps $x_2$ and changes $x_1$. Each layer is simple, but a stack of them can bend a Gaussian into a curved, skewed shape, and the density stays exact.

Interactive figure: loading…

3. Definitions and main results #

3.1 The change-of-variables formula #

Theorem (change of variables). Let $f: \R^D \to \R^D$ be a bijection, with $f$ and $f^{-1}$ continuously differentiable, and let $z = f(x)$ have density $p_Z$. Then

Why this is true. A small box around $x$ with volume $dV$ is mapped to a small parallelogram around $f(x)$ with volume $|\det J_f(x)|\,dV$ (the determinant is the volume factor of the linear map $J_f(x)$). Both hold the same probability: $p_X(x)\,dV = p_Z(f(x))\,|\det J_f(x)|\,dV$.

Proof.

1. 1D, $f$ increasing: $F_X(x) = P(X \le x) = P(Z \le f(x)) = F_Z(f(x))$. Differentiate: $p_X(x) = p_Z(f(x))\,f'(x)$.
2. 1D, $f$ decreasing: $F_X(x) = P(Z \ge f(x)) = 1 - F_Z(f(x))$, so $p_X(x) = -p_Z(f(x))\,f'(x)$. Both cases give $p_Z(f(x))\,|f'(x)|$.
3. $D$ dimensions: for every set $A$, $P(X \in A) = P(Z \in f(A)) = \int_{f(A)} p_Z(z)\,dz = \int_A p_Z(f(x))\,|\det J_f(x)|\,dx$, by the substitution $z = f(x)$ of multivariable calculus.
4. A density is fixed by its integrals over all sets, so the integrand is $p_X$. $\blacksquare$

The calculus lesson on change of variables gives the substitution rule of step 3 in full.

Composition. Let $f = f_K \circ \dots \circ f_1$, with $z_0 = x$, $z_k = f_k(z_{k-1})$ and $z = z_K$. The chain rule gives $J_f = J_{f_K} \cdots J_{f_1}$, and the determinant of a product is the product of the determinants. So

Training minimizes the mean of $-\log p_X(x)$ over the data: maximum likelihood, the same as minimizing $\mathrm{KL}(p_{\text{data}} \,\|\, p_X)$. You never form $J_f$ itself; each layer reports its own log-determinant.

The other direction. In the sampling direction, $x = f^{-1}(z)$ and $\log p_X(x) = \log p_Z(z) - \log|\det J_{f^{-1}}(z)|$: the sign flips, because $J_{f^{-1}}(z) = J_f(x)^{-1}$. Check with $x = 2z$: the data are more spread out than the base, so the density must go down, and indeed $\log p_X(x) = \log p_Z(z) - \log 2$.

3.2 Affine coupling layers (RealNVP and Glow) #

Split $x = (x_A, x_B)$. Let $s$ and $t$ be any networks from $x_A$ to vectors of the size of $x_B$, and $\odot$ the elementwise product. The affine coupling layer (RealNVP; Dinh et al. 2017) is

1. Jacobian. $y_A$ does not depend on $x_B$, so the upper-right block is 0. Each $y_{B,j}$ depends on $x_B$ only through $x_{B,j}$, so the lower-right block is diagonal:

2. Log-determinant. The matrix is block lower triangular, so its determinant is the product of the diagonal entries: $\log|\det J| = \sum_j s_j(x_A)$. The cost is $O(D)$. The block $\partial y_B/\partial x_A$, which holds all the derivatives of the networks, never enters.
3. Inverse. $x_A = y_A$ and $x_B = \big(y_B - t(y_A)\big) \odot \exp\big(-s(y_A)\big)$. It needs only forward passes of $s$ and $t$, so $s$ and $t$ never have to be inverted: they can be arbitrary networks.
4. Stacking. One layer leaves $x_A$ unchanged, so the roles of $A$ and $B$ alternate (or the coordinates are permuted) between layers. The product of the layer Jacobians is a full matrix, but its log-determinant is still the sum of the layer terms.

In practice the log-scale is bounded, for example $s = \tanh(s_{\text{raw}})$ or a soft clamp, so each factor $e^{s_j}$ stays in a safe range early in training.

Glow (Kingma and Dhariwal 2018) adds two layers for feature maps of size $H \times W$ with $c$ channels:

• Actnorm: a per-channel scale $s_c$ and bias, initialized on the first batch so that each channel has zero mean and unit variance. Log-determinant: $HW \sum_c \log|s_c|$.
• Invertible $1 \times 1$ convolution: the same $c \times c$ matrix $W$ mixes the channels at every pixel. The Jacobian of the whole layer is block diagonal with $HW$ copies of $W$, so $\log|\det J| = HW \log|\det W|$. With the LU form $W = PL(U + \operatorname{diag}(s))$ ($P$ a fixed permutation, $L$ unit lower triangular, $U$ strictly upper triangular), $\log|\det W| = \sum_i \log|s_i|$: cost $O(c)$ instead of $O(c^3)$. It generalizes the fixed permutation between coupling layers.

3.3 Autoregressive flows (MAF and IAF) #

An affine autoregressive layer transforms coordinate $i$ with a shift $\mu_i$ and a log-scale $\alpha_i$. Both come from one MADE network (lesson 03) with two outputs per dimension.

MAF (masked autoregressive flow; Papamakarios et al. 2017) works in the density direction:

1. Jacobian. $u_i$ depends on $x_1, \dots, x_i$ only, so $\partial u/\partial x$ is lower triangular with diagonal entries $e^{-\alpha_i}$. Hence $\log|\det \partial u/\partial x| = -\sum_i \alpha_i(x_{<i})$.
2. Density: one pass. All $\mu_i, \alpha_i$ depend on the observed $x$, so one MADE pass gives all of them.
3. Sampling: $D$ passes. The inverse is $x_i = u_i e^{\alpha_i(x_{<i})} + \mu_i(x_{<i})$, and $x_{<i}$ must be known before $x_i$. This is the ancestral sampling of lesson 03. An AR model with Gaussian conditionals is exactly a one-layer MAF.

IAF (inverse autoregressive flow; Kingma et al. 2016) works in the sampling direction. Its parameters depend on the base variables:

1. Sampling: one pass, because $u$ is known.
2. Density of its own samples: one pass. $\log q(x) = \log \mathcal N(u; 0, I) - \sum_i \alpha_i(u_{<i})$, and $u$ is kept from the sampling step.
3. Density of an outside $x$: $D$ passes. Recovering $u_i$ needs $u_{<i}$ first.

Interactive figure: loading…

Flow	$\log p$ of a data point	Sampling	Typical use
Coupling (RealNVP, Glow)	1 pass	1 pass	images, audio (WaveGlow); needs many layers
MAF	1 pass	$D$ passes	density estimation, simulation-based inference
IAF	$D$ passes (own samples: 1)	1 pass	VAE posteriors; fast students (Parallel WaveNet)
Continuous (FFJORD)	ODE solve + trace	ODE solve	free-form vector field; cost = number of evaluations

A coupling layer is an autoregressive layer with only two blocks: block $B$ is conditioned on block $A$, and $A$ on nothing. That is why it is fast in both directions, and less expressive per layer.

3.4 Continuous flows: the instantaneous change of variables #

A continuous normalizing flow (CNF) defines the map by an ordinary differential equation

Any network $f$ that is Lipschitz in $x$ (and continuous in $t$) works. The solution map is invertible (solve the ODE backwards in time), and $f$ needs no special structure: no triangular Jacobian, no coupling.

Picture: one Euler step. The step $x_{t+h} = x_t + h\,f(x_t, t)$ has Jacobian $I + hJ$, with $J = \partial f/\partial x$, and $\log\det(I + hJ) = h \operatorname{tr}(J) + O(h^2)$. The change of variables for this small step gives $\log p_{t+h}(x_{t+h}) = \log p_t(x_t) - h\operatorname{tr}(J) + O(h^2)$. Now let $h \to 0$.

Theorem (instantaneous change of variables; Chen et al. 2018). If $x(t)$ solves $\dot x = f(x, t)$ and $p_t$ is the density of $x(t)$, then

The trace of the Jacobian is the divergence $\nabla \cdot f = \sum_i \partial f_i/\partial x_i$: the rate at which the flow expands volume. Expanding volume lowers the density along the path.

Proof (from the continuity equation)

1. Probability mass moves with the flow, so $p_t$ solves the continuity equation $\partial_t p_t + \nabla \cdot (p_t f) = 0$.
2. Along a trajectory, the total derivative is $\frac{d}{dt}p_t(x(t)) = \partial_t p_t + \nabla p_t \cdot f$ (chain rule, with $\dot x = f$).
3. Expand $\nabla \cdot (p_t f) = \nabla p_t \cdot f + p_t\,\nabla \cdot f$ and insert step 1 into step 2: $\frac{d}{dt}p_t(x(t)) = -\nabla p_t \cdot f - p_t\,\nabla \cdot f + \nabla p_t \cdot f = -p_t\,\nabla \cdot f$.
4. Divide by $p_t$: $\frac{d}{dt}\log p_t(x(t)) = -\nabla \cdot f = -\operatorname{tr}(\partial f/\partial x)$. $\blacksquare$

Integrate from the data at $t = 0$ to the base at $t = T$:

One ODE solver call integrates the state $x$ and the accumulated trace together. Training a CNF by maximum likelihood needs gradients through the solver (backpropagation through the solver steps, or the adjoint method of Chen et al. 2018).

Interactive figure: loading…

Hutchinson's estimator (FFJORD). The exact trace needs $D$ backward passes, one per diagonal entry of $\partial f/\partial x$. For any random vector $\varepsilon$ with $\mathbb E[\varepsilon\varepsilon^\top] = I$ (Rademacher $\pm 1$ entries, or Gaussian),

because $\mathbb E[\varepsilon^\top J \varepsilon] = \sum_{i,j} J_{ij}\,\mathbb E[\varepsilon_i \varepsilon_j] = \sum_i J_{ii}$. The product $\varepsilon^\top J$ is one vector-Jacobian product: one backward pass per probe, whatever $D$ is. FFJORD (Grathwohl et al. 2019) uses this estimator and keeps the same $\varepsilon$ for a whole ODE solve, so the dynamics stay deterministic inside the solver. Two facts to remember. The estimate of $\log p$ is unbiased, but its exponential is not an unbiased estimate of $p$ (Jensen's inequality). And the variance of one probe is $2\|J_s\|_F^2$ for Gaussian probes and $2\sum_{i \ne j} (J_s)_{ij}^2$ for Rademacher probes, with $J_s = (J + J^\top)/2$: Rademacher probes add no noise from the diagonal. The calculus lesson proves the variance formulas.

Interactive figure: loading…

Diffusion models and flow matching are continuous flows. The probability-flow ODE of a diffusion model (lesson 09) is $\frac{dx}{dt} = f(x, t) - \tfrac12 g(t)^2 \nabla_x \log p_t(x)$, where $f$ and $g$ are the drift and the diffusion coefficient of the forward SDE (the letter $f$ has that meaning in lessons 07 to 15). It has the same marginals $p_t$ as the diffusion process. With the learned score inserted, its right side is a vector field like the one above, so the formula of this subsection gives the exact log-likelihood of that ODE model, up to solver error and trace noise. Song et al. (2021) report the bits per dimension of diffusion models this way. Flow matching and rectified flow (lesson 10) train the same kind of continuous flow without simulating the ODE; after training, the same formula gives their likelihood.

3.5 What flows cannot do #

• No change of dimension. A flow is a bijection between $\R^D$ and $\R^D$ with a square Jacobian. A flow cannot map data to a smaller latent space, as a VAE does. If $z$ had fewer dimensions than $x$, the samples $f^{-1}(z)$ would lie on a lower-dimensional surface, which has zero volume in $\R^D$: there would be no density $p_X$ at all.
• Topology. A flow is a continuous bijection, and $\mathcal N(0, I)$ is positive everywhere, so $p_X(x) = p_Z(f(x))\,|\det J_f(x)| > 0$ everywhere: the model can never put exactly zero density between two separated modes. To make the density small there, the sampling map $f^{-1}$ must stretch that region very strongly (huge $|\det J_{f^{-1}}|$), which needs very large derivatives and is hard to learn.

Answers to section 1. (1) $O(1000)$: the log-determinant is $\sum_j s_j(x_A)$. The networks cost one forward pass; no determinant is ever computed. (2) MAF evaluates $\log p(x)$ in one pass, because its parameters depend on the observed $x$. IAF samples in one pass, because its parameters depend on the known base variables $u$. Each needs $D$ passes in its other direction.

4. Worked example #

(a) Two coupling layers in 2D (density direction, base $\mathcal N(0, I)$), at $x = (2, 1)$.

• Layer 1 keeps $x_1$, with $s_1(x_1) = x_1/2$ and $t_1(x_1) = -x_1$. So $s_1(2) = 1$, $t_1(2) = -2$, and $y = (2,\ 1 \cdot e^1 - 2) = (2,\ 0.7183)$. Log-determinant: $s_1(2) = 1$.
• Layer 2 keeps $y_2$, with $s_2(y_2) = -y_2/2$ and $t_2(y_2) = 1$. So $s_2(0.7183) = -0.3591$ and $z = (2e^{-0.3591} + 1,\ 0.7183) = (2.3966,\ 0.7183)$. Log-determinant: $-0.3591$.
• Base: $\log \mathcal N(z; 0, I) = -\log 2\pi - \tfrac12(2.3966^2 + 0.7183^2) = -1.8379 - \tfrac12(5.7435 + 0.5159) = -1.8379 - 3.1297 = -4.9676$.
• Result: $\log p_X(x) = -4.9676 + 1 - 0.3591 = -4.3267$.

The layer Jacobians are triangular, $J_1 = \begin{pmatrix} 1 & 0 \\ 0.3591 & 2.7183 \end{pmatrix}$ (lower) and $J_2 = \begin{pmatrix} 0.6983 & -0.6983 \\ 0 & 1 \end{pmatrix}$ (upper), but their product $J_2 J_1 = \begin{pmatrix} 0.4475 & -1.8981 \\ 0.3591 & 2.7183 \end{pmatrix}$ has no zero entry. Still, $\det(J_2 J_1) = 2.7183 \cdot 0.6983 = 1.8981 = e^{0.6409}$, and $0.6409 = 1 - 0.3591$.

Inverse. Undo layer 2 first: $y_2 = z_2 = 0.7183$ and $y_1 = (z_1 - 1)\,e^{y_2/2} = 1.3966 \cdot 1.4321 = 2$. Then undo layer 1: $x_1 = y_1 = 2$ and $x_2 = (y_2 + x_1)\,e^{-x_1/2} = 2.7183 \cdot 0.3679 = 1$.

(b) A one-layer MAF is a Gaussian AR model. Take $\mu_1 = 0$ and $\alpha_1 = \log 2$ (so $x_1 \sim \mathcal N(0, 2^2)$), and $\mu_2(x_1) = x_1^2$, $\alpha_2 = 0$ (so $x_2 \mid x_1 \sim \mathcal N(x_1^2, 1)$, a "banana" shape). At $x = (2, 3)$:

• one pass: $u = \big((2 - 0)/2,\ (3 - 4)/1\big) = (1, -1)$, log-determinant $-\alpha_1 - \alpha_2 = -\log 2$;
• $\log p(x) = \log \mathcal N(u; 0, I) - \log 2 = -2.8379 - 0.6931 = -3.5310$;
• chain-rule check: $\log \mathcal N(2; 0, 4) + \log \mathcal N(3; 4, 1) = -2.1121 - 1.4189 = -3.5310$.

Sampling from $u = (0.5, 0.2)$ takes two passes: first $x_1 = 2 \cdot 0.5 = 1$, then $x_2 = 0.2 + 1^2 = 1.2$.

(c) A linear continuous flow. $f(x) = Ax$ with $A = \begin{pmatrix} -1 & 2 \\ 0 & -0.5 \end{pmatrix}$ and $T = 1$. Here $\operatorname{tr}(\partial f/\partial x) = \operatorname{tr} A = -1.5$ everywhere, so $\log p_0(x(0)) = \log p_1(x(1)) - 1.5$. Check with the discrete formula: the flow map is $x(1) = e^{A}x(0)$, and $\log|\det e^{A}| = \operatorname{tr} A = -1.5$ (Jacobi's formula $\det e^A = e^{\operatorname{tr} A}$). The flow shrinks volume, so the density grows along each path, by the factor $e^{1.5} \approx 4.48$. This is the "worked example" preset of the figure above.

Hutchinson by hand: the four Rademacher probes $\varepsilon = (\pm 1, \pm 1)$ give $\varepsilon^\top A \varepsilon = 0.5, -3.5, -3.5, 0.5$ for $(1, 1), (1, -1), (-1, 1), (-1, -1)$. The mean is $-1.5 = \operatorname{tr} A$, and the variance is $4 = (A_{12} + A_{21})^2$: with Rademacher probes, only the off-diagonal entries add noise.

5. In code #

NumPy and SciPy check the hand computations: the coupling layers with a finite-difference Jacobian, the MAF against the chain rule, Glow's $1 \times 1$ log-determinant, the instantaneous change of variables on a nonlinear 2D flow, and Hutchinson's estimator. For (d), with vector field $f$, we integrate the Jacobian $\Phi(t) = \partial x(t)/\partial x(0)$ of the flow map (it solves $\dot\Phi = (\partial f/\partial x)\,\Phi$) next to $\delta = -\int_0^t \operatorname{tr}(\partial f/\partial x)\,ds$, and compare $\delta$ with $-\log|\det \Phi|$.

import numpy as np
from scipy.integrate import solve_ivp
from scipy.stats import norm

# (a) The two coupling layers of section 4, density direction x -> z
def f(x):
    y = np.array([x[0], x[1] * np.exp(x[0] / 2) - x[0]])        # layer 1 keeps x1
    return np.array([y[0] * np.exp(-y[1] / 2) + 1, y[1]])       # layer 2 keeps y2

x = np.array([2.0, 1.0])
z = f(x)
h = 1e-6                                                         # Jacobian by central differences
J = np.column_stack([(f(x + h * e) - f(x - h * e)) / (2 * h) for e in np.eye(2)])
logdet = np.linalg.slogdet(J)[1]
print(z.round(4), J.round(4).tolist())     # [2.3966 0.7183] [[0.4475, -1.8981], [0.3591, 2.7183]]
print(round(logdet, 4), round(norm.logpdf(z).sum() + logdet, 4))   # 0.6409 -4.3267

# (b) The one-layer MAF of section 4 equals the chain rule p(x1) p(x2 | x1)
x1, x2 = 2.0, 3.0
u = np.array([(x1 - 0.0) / 2.0, (x2 - x1**2) / 1.0])          # mu1 = 0, sigma1 = 2; mu2 = x1^2, sigma2 = 1
print(round(norm.logpdf(u).sum() - np.log(2.0), 4),
      round(norm.logpdf(x1, 0, 2) + norm.logpdf(x2, x1**2, 1), 4))  # -3.531 -3.531

# (c) Glow's 1x1 convolution: the Jacobian is block diagonal with H*W copies of W
c, H, W_ = 3, 4, 5
Wmix = np.random.default_rng(0).normal(size=(c, c))
Jc = np.kron(np.eye(H * W_), Wmix)                               # (60, 60), pixel by pixel
print(round(np.linalg.slogdet(Jc)[1], 6), round(H * W_ * np.linalg.slogdet(Wmix)[1], 6))
# -16.271734 -16.271734

# (d) Instantaneous change of variables for dx/dt = tanh(Ax + b): integrate x, the Jacobian Phi
#     of the flow map, and delta = -int tr(df/dx) dt together
A, b = np.array([[0.5, -2.0], [1.0, -1.0]]), np.array([0.3, -0.2])
def rhs(t, s):
    xs, Phi = s[:2], s[2:6].reshape(2, 2)
    Jf = np.diag(1 - np.tanh(A @ xs + b) ** 2) @ A               # df/dx
    return np.concatenate([np.tanh(A @ xs + b), (Jf @ Phi).ravel(), [-np.trace(Jf)]])
s0 = np.concatenate([[1.0, -0.5], np.eye(2).ravel(), [0.0]])
sol = solve_ivp(rhs, (0.0, 2.0), s0, rtol=1e-10, atol=1e-12)
Phi, delta = sol.y[2:6, -1].reshape(2, 2), sol.y[6, -1]
print(round(delta, 6), round(-np.log(abs(np.linalg.det(Phi))), 6))   # 0.213696 0.213696

# (e) Hutchinson on the 2x2 matrix of section 4: all four Rademacher probes
Alin = np.array([[-1.0, 2.0], [0.0, -0.5]])
eps = np.array([[1, 1], [1, -1], [-1, 1], [-1, -1]])
q = np.einsum("ni,ij,nj->n", eps, Alin, eps)
print(q, q.mean(), q.var())                                      # [ 0.5 -3.5 -3.5  0.5] -1.5 4.0

Check (d) is Liouville's formula $\det \Phi(t) = \exp \int_0^t \operatorname{tr}(\partial f/\partial x)\,ds$: the instantaneous change of variables seen from the Jacobian side.

6. Where it shows up #

• Images and audio. RealNVP and Glow use coupling layers; WaveGlow uses them for speech. On CIFAR-10, Glow reaches 3.35 bits per dimension; strong AR models (PixelCNN++: 2.92) and diffusion models do better (lesson 14 explains how such numbers are compared).
• Variational inference. An IAF makes a VAE's $q(z \mid x)$ flexible: the VAE needs only samples from $q$ and $\log q$ of those samples. Parallel WaveNet distills a slow AR WaveNet into a fast IAF student.
• Simulation-based inference. MAF and neural spline flows are standard conditional density estimators $q(\theta \mid x_{\text{obs}})$ for neural posterior estimation, where a simulator exists but its likelihood is not available.
• Physics sampling. Boltzmann generators (Noé et al. 2019) and flows for lattice field theory sample with an exact density $q$, then reweight by $p/q$ or add a Metropolis correction to remove the model's bias.
• Continuous flows in ML for science. Rectified-flow and flow-matching models are CNFs, and diffusion models have the probability-flow ODE. The held-out log-likelihood from the instantaneous change of variables is a strictly proper score for comparing such models of, for example, ground motion or weather fields. Report it on a fixed data scale: if a model is trained on $x' = x/\sigma$ in $D$ dimensions, then $\log p_X(x) = \log p_{X'}(x/\sigma) - D\log\sigma$.

7. Common mistakes #

• Sign of the log-determinant. In the density direction, $\log p(x) = \log p_Z(f(x)) + \log|\det J_f(x)|$ (add). In the sampling direction, $\log p(x) = \log p_Z(z) - \log|\det J_{f^{-1}}(z)|$ (subtract). Check with $x = 2z$: the density must go down.
• No alternation. Coupling layers with the same split keep $x_A$ unchanged forever.
• Unbounded scales. $e^{s}$ with a raw network output can overflow. Bound it ($\tanh$, a soft clamp) and initialize near the identity.
• MAF sampling in one pass, or IAF density of data in one pass. Each needs $D$ sequential passes in its slow direction.
• Forgetting preprocessing log-determinants. Scaling, logit transforms and dequantization change $\log p$; add their log-determinants before you compare likelihoods.
• Trusting one Hutchinson probe for evaluation, or exponentiating it as an unbiased likelihood. Use the exact trace or many probes for test numbers.
• Expecting a perfect fit of separated modes. The model density is positive everywhere, also between separated modes (section 3.5).

8. Check yourself #

1. A coupling layer gives $s(x_A) = (0.3, -0.1, 0.5)$. What is $\log|\det J|$ of the layer, and of its inverse?
   Answer

   $0.3 - 0.1 + 0.5 = 0.7$. The inverse layer has $-0.7$.
2. A Glow $1 \times 1$ convolution acts on a $16 \times 16 \times 8$ feature map, and $\log|\det W| = 0.2$. What does the layer add to $\log p(x)$ in the density direction?
   Answer

   $16 \cdot 16 \cdot 0.2 = 51.2$ nats: the Jacobian has $HW = 256$ diagonal blocks equal to $W$.
3. A MAF on $D = 100$ dimensions. How many MADE passes does it need to evaluate $\log p$ of a batch of data? To draw a batch of samples?
   Answer

   One pass for the density. 100 sequential passes for sampling, because $x_i$ needs $x_{<i}$.
4. Why is IAF a good VAE posterior but a poor density estimator for data?
   Answer

   A VAE needs samples $z \sim q$ and $\log q(z)$ of those same samples: one pass each, since $u$ is known. The density of an outside point needs $u$ from $x$, which takes $D$ sequential passes.
5. A CNF has $f(x) = Rx$ with $R = \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}$ (a rotation). How does $\log p_t(x(t))$ change along a trajectory?
   Answer

   It stays constant: $\operatorname{tr} R = 0$, so the flow is divergence-free and keeps volume.
6. Hutchinson with Rademacher probes on $J = \begin{pmatrix} 2 & 1 \\ 3 & -1 \end{pmatrix}$. Give the possible values of $\varepsilon^\top J \varepsilon$, the mean and the variance.
   Answer

   $\varepsilon^\top J \varepsilon = 2 - 1 + (1 + 3)\varepsilon_1\varepsilon_2 = 1 + 4\varepsilon_1\varepsilon_2$, so $5$ or $-3$, each with probability $1/2$. Mean $1 = \operatorname{tr} J$; variance $16 = (J_{12} + J_{21})^2$.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. State the change-of-variables formula for $z = f(x)$.

   Show answer

2. $z \sim \mathcal N(0, 1)$ and $x = az + b$. What is $p_X(x)$?

   Show answer

3. Log-likelihood of a flow $f = f_K \circ \dots \circ f_1$ in the density direction?

   Show answer

4. Affine coupling layer: formula and $\log|\det J|$?

   Show answer

5. Why is the Jacobian of a coupling layer triangular?

   Show answer

6. MAF vs IAF: which is fast for what?

   Show answer

7. Log-determinant of a Glow $1 \times 1$ convolution on an $H \times W \times c$ map?

   Show answer

8. State the instantaneous change of variables.

   Show answer

9. What is Hutchinson's trace estimator, and why is it unbiased?

   Show answer

10. Why can a flow not have a lower-dimensional latent space?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Write the log-likelihood of a normalizing flow." — $\log p_Z(f(x)) + \sum_k \log|\det J_{f_k}|$ along the density direction.
• "Coupling or autoregressive flow?" — The cost table: coupling is fast both ways but weaker per layer; MAF is fast for density, IAF for sampling.
• "Derive the log-determinant of a Glow $1 \times 1$ convolution." — Block-diagonal Jacobian with $HW$ copies of $W$: $HW\log|\det W|$; the LU form makes it $O(c)$.
• "How do you get the likelihood of a diffusion or flow-matching model?" — The instantaneous change of variables along the probability-flow ODE, with a Hutchinson or exact trace (for diffusion there is also the ELBO of lesson 07).
• "Why do flows give worse samples than diffusion models?" — The same dimension everywhere, the topology limit, many constrained layers, and maximum likelihood spends capacity on covering all the data. Flow matching and diffusion train an unconstrained network and get invertibility from the ODE.

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution, and a demonstration that trains a small RealNVP on two moons in NumPy:

1. 1D change of variables: the density of $x = \sinh(z)$, checked against a histogram of mapped samples.
2. Affine coupling forward and inverse; compare the log-determinant with a finite-difference Jacobian.
3. A RealNVP stack: log_prob and sample; check that the density integrates to 1 on a grid.
4. Hutchinson's estimator with Gaussian and Rademacher probes; compare the variances with the formulas.
5. The linear ODE $dx/dt = Ax$: check that the change of $\log p_t$ along a path is $-t\operatorname{tr}A$.

11. Watch and read #

What are Normalizing Flows?Ari Seff · 12 min

Change of variables, the log-determinant and coupling layers, with clear animations.

L3 Flow Models -- CS294-158-SP20 Deep Unsupervised Learning -- UC Berkeley -- Spring 2020Pieter Abbeel · Berkeley CS294-158 Deep Unsupervised Learning · 1 h 57 min

A full lecture on flows: 1D flows, change of variables, coupling layers (RealNVP, Glow) and autoregressive flows (MAF, IAF).

Neural Ordinary Differential EquationsYannic Kilcher · 22 min

The paper that introduced continuous normalizing flows and the instantaneous change of variables.

• Papamakarios, Nalisnick, Rezende, Mohamed and Lakshminarayanan, Normalizing Flows for Probabilistic Modeling and Inference (JMLR 2021): §2 (definition and composition), §3.1 (coupling and autoregressive flows, the MAF/IAF cost asymmetry), §3.2 (linear flows, $1 \times 1$ convolutions), §4.1 (continuous-time flows).
• Murphy, Probabilistic Machine Learning: Advanced Topics, free PDF: chapter 23 (§23.2.3 coupling flows, §23.2.4 autoregressive flows, §23.2.6 continuous-time flows).
• Dinh, Sohl-Dickstein and Bengio, Density estimation using Real NVP (ICLR 2017); Kingma and Dhariwal, Glow (2018).
• Papamakarios, Pavlakou and Murray, Masked Autoregressive Flow for Density Estimation (2017); Kingma et al., Improving Variational Inference with Inverse Autoregressive Flow (2016).
• Chen, Rubanova, Bettencourt and Duvenaud, Neural Ordinary Differential Equations (2018); Grathwohl et al., FFJORD (ICLR 2019), §3.1 (the trace estimator with a fixed noise vector).
• Lilian Weng, Flow-based Deep Generative Models: a readable overview.