GANs and Wasserstein GANs: the optimal discriminator, training dynamics and $W_1$

Before this lesson: lesson 01 (maximum likelihood as a KL divergence) and lesson 04 (models with an exact likelihood; a GAN gives that up). From other tracks: the KL divergence and f-divergences, the convex conjugate and critics (information theory), duality and stochastic gradient descent (optimization), and the spectral norm (linear algebra). Time: about 95 minutes, plus the notebook.

---

Why this matters #

A GAN learns to sample without a density. This made GANs the leading image generators from 2014 to about 2021 (DCGAN, StyleGAN, BigGAN). They are still in use. The autoencoder of latent diffusion models (Stable Diffusion) is trained with an adversarial loss, so the latent space of lesson 13 is shaped by a GAN. Few-step diffusion samplers use discriminators (lesson 12). In science, GANs were used for precipitation nowcasting and for seismic ground-motion synthesis before diffusion models.

The ideas of this lesson also go far beyond GANs. "Train a classifier and read off the density ratio" is the basis of neural ratio estimation in physics. The Wasserstein distance and Lipschitz critics come back in optimal transport and flow matching (lesson 10). The questions "what does the discriminator learn?", "why does the gradient vanish?" and "why does WGAN help?" are standard interview questions.

1. Predict first #

2. The picture #

The simplest case: three outcomes. A sample can take three values. The data distribution is $p_{\text{data}} = (0.6, 0.4, 0)$. The generator currently produces $p_g = (0.2, 0.4, 0.4)$. Real and fake samples come in equal numbers. A discriminator sees one sample $x$ and must say "real" or "fake". The best it can do is to compare the two probabilities of $x$:

• outcome 1 is three times more likely under the data, so it says "real" with probability $0.6/(0.6 + 0.2) = 0.75$;
• outcome 2 is equally likely under both, so it says $0.4/(0.4 + 0.4) = 0.5$;
• outcome 3 never occurs in the data, so it says $0$.

In odds, $D/(1 - D) = (3, 1, 0) = p_{\text{data}}/p_g$. A perfect discriminator is a density-ratio estimator. The generator reads this ratio and moves probability from outcome 3 (ratio 0) toward outcome 1 (ratio 3).

The same in 1D. In the figure, the data are the track's two Gaussians and the generator is one Gaussian. The lower panel shows the best discriminator at every point.

Interactive figure: loading…

The generator only sees the slope of $D$ at its own samples. A sample is $x = G_\theta(z)$. The generator changes $\theta$ to raise $D$ at $x$. By the chain rule,

$J_G$ is the Jacobian of the sample with respect to the generator weights. So each sample moves in the direction where $D$ increases. If $D$ is flat at the sample, $\nabla_x D = 0$ and the generator learns nothing from it.

When the supports do not overlap. Real images lie near a thin, low-dimensional set in pixel space. A generator with a low-dimensional input $z$ produces another thin set. Two thin sets in a high-dimensional space almost never overlap. Then a discriminator can separate them perfectly: $D \approx 1$ on the data, $D \approx 0$ on the fakes, and $D$ is flat near both. No gradient.

A distance that sees how far apart. Think of $P$ and $Q$ as two piles of sand. The Wasserstein-1 distance $W_1$ is the least total work (mass times distance moved) needed to turn one pile into the other. If the generator's pile is 0.1 away, $W_1 = 0.1$; if it is 1 away, $W_1 = 1$. The Jensen–Shannon divergence is $\log 2$ in both cases, because it only asks whether the piles overlap.

3. Definitions and main results #

The GAN game #

Notation. Noise $z \sim p_z$ (for example $\mathcal N(0, I_k)$) goes through the generator $G_\theta: \R^k \to \R^d$. The distribution of $G_\theta(z)$ is $p_g$. We can sample from it, but we cannot evaluate it (for $k < d$ it has no density at all). The discriminator outputs a logit $l_\phi(x) \in \R$ and $D_\phi(x) = \sigma(l_\phi(x)) \in (0, 1)$, the probability that $x$ is real, where $\sigma(u) = 1/(1 + e^{-u})$ is the logistic sigmoid. The value function is

The game is $\min_G \max_D V(D, G)$.

$-V$ is the binary cross-entropy of a classifier with label 1 for real and 0 for fake, so the $D$ step is logistic regression. Because $\mathbb E_z[h(G(z))] = \mathbb E_{x \sim p_g}[h(x)]$ for any function $h$, the second term is $\mathbb E_{p_g}[\log(1 - D(x))]$.

The Jensen–Shannon divergence (JSD) of $p$ and $q$ uses their average $m = \tfrac12(p + q)$:

It is symmetric, always finite, between $0$ and $\log 2$, and $0$ only when $p = q$.

Theorem (optimal discriminator; Goodfellow et al. 2014). For a fixed $G$,

The global minimum of $C$ is $-\log 4$, reached only at $p_g = p_{\text{data}}$.

Why this is true, in one sentence. The discriminator can choose its answer separately at every point, and at each point the best answer is the share of real samples there; putting that answer back into $V$ gives two KL divergences to the average density.

Proof.

1. Write $V$ as one integral. $D(x)$ can be chosen separately at every $x$, so we maximize the integrand point by point:

2. For $a, b \ge 0$, not both 0, the function $y \mapsto a\log y + b\log(1 - y)$ is concave on $[0, 1]$ (with $\log 0 = -\infty$). For $a, b > 0$ its derivative $a/y - b/(1 - y)$ is zero at $y^* = a/(a + b)$. For $b = 0$ the maximum is at $y^* = 1$, for $a = 0$ at $y^* = 0$; the same formula. With $a = p_{\text{data}}(x)$ and $b = p_g(x)$ this is $D^*$.
3. Let $m = (p_{\text{data}} + p_g)/2$. Then $D^* = p_{\text{data}}/(2m)$ and $1 - D^* = p_g/(2m)$. Substitute:

4. By the definition of JSD, $C(G) = -\log 4 + 2\,\mathrm{JSD}(p_{\text{data}} \,\|\, p_g)$.
5. $\mathrm{JSD} \ge 0$, with equality only if $p_g = p_{\text{data}}$. There $D^* = \tfrac12$ everywhere and $C = -\log 4$. $\blacksquare$

Assumptions. $D$ can be any function (infinite capacity), both distributions have densities, and the inner maximization is solved for each $G$. In practice none of these holds exactly: $D$ is a network trained for a few steps per generator step. So the theorem describes the target of the game, not what each training step does.

Saturating and non-saturating generator losses #

Write $l = l_\phi(G(z))$ for the logit of a fake sample. Use the softplus function $\operatorname{softplus}(u) = \log(1 + e^u)$ and the identities $\log\sigma(l) = -\operatorname{softplus}(-l)$ and $\log(1 - \sigma(l)) = -\operatorname{softplus}(l)$. These forms stay finite for any $l$.

Generator minimizes	As a function of $l$	Derivative in $l$	At $l = -8$
saturating (minimax): $\log(1 - D(G(z)))$	$-\operatorname{softplus}(l)$	$-\sigma(l)$	$-3.4 \times 10^{-4}$
non-saturating: $-\log D(G(z))$	$\operatorname{softplus}(-l)$	$-\sigma(-l)$	$-0.99966$

Both derivatives are negative, so both losses push $l$ up. But when $D$ confidently rejects the fakes ($l \ll 0$), the saturating gradient is about $e^{l}$: it vanishes exactly when the generator is worst. The non-saturating loss "flips the label instead of the sign" and keeps a gradient near $-1$. The ratio of the two gradients is $\sigma(-l)/\sigma(l) = e^{-l}$, which is $e^8 \approx 2981$ at $l = -8$.

Interactive figure: loading…

The non-saturating game is no longer zero-sum, but it has the same fixed point ($p_g = p_{\text{data}}$, $D = \tfrac12$). With the optimal $D^*$, its expected generator gradient equals $\nabla_\theta\big[\mathrm{KL}(p_g \,\|\, p_{\text{data}}) - 2\,\mathrm{JSD}(p_g \,\|\, p_{\text{data}})\big]$ (Arjovsky and Bottou 2017, Theorem 2.5). The reverse KL $\mathrm{KL}(p_g \,\|\, p_{\text{data}})$ in it is mode-seeking: it punishes fakes where there are no data, but it barely punishes a missing mode (see forward and reverse KL).

Disjoint supports: why JSD gives no gradient #

If $p_{\text{data}}$ and $p_g$ live on disjoint sets, then on the data's set $m = p_{\text{data}}/2$, so $\mathrm{KL}(p_{\text{data}} \,\|\, m) = \mathbb E_{p_{\text{data}}}[\log 2] = \log 2$. The same holds for $p_g$. Hence $\mathrm{JSD} = \log 2$ for every such $p_g$: the loss is constant, and its gradient is zero.

This is the normal case, not a corner case. $p_g$ lives on $G(\R^k)$, a set of dimension at most $k$. If both sets have lower dimension than $\R^d$ and do not line up exactly, their intersection has probability zero under both distributions. Then a perfect discriminator exists that is smooth and constant near both sets (Arjovsky and Bottou 2017, Theorems 2.1–2.2). As $D$ approaches it, the saturating gradient goes to zero, and the non-saturating gradient becomes unstable.

Training dynamics: the Dirac-GAN #

Even when the gradients do not vanish, gradient play on a game is not gradient descent on one function. The cleanest example is the Dirac-GAN (Mescheder, Geiger and Nowozin 2018):

• the data are a point mass at $0$;
• the generator is a point mass at $\theta$ (one number);
• the discriminator is linear, with logit $l(x) = \psi x$ (one number).

The value function is $V(\theta, \psi) = \log\sigma(0) + \log(1 - \sigma(\psi\theta))$. The unique equilibrium is $\theta = 0$ (the generator matches the data), $\psi = 0$ (the discriminator cannot tell). The generator descends in $\theta$, the discriminator ascends in $\psi$. The game vector field is

Step 1: linearize at the equilibrium. At $(0, 0)$, $\sigma(0) = \tfrac12$, so the Jacobian of the field is

Purely imaginary eigenvalues mean rotation: the continuous field circles the equilibrium and never comes closer.

Step 2: simultaneous steps. Both players step from the old point: $(\theta, \psi) \leftarrow (\theta, \psi) + \eta\,(\dot\theta, \dot\psi)$. Near $(0, 0)$ the step matrix is $I + \eta J$, with eigenvalues $1 \pm i\eta/2$ and modulus $\sqrt{1 + \eta^2/4} > 1$. Every step makes the distance to the equilibrium a little larger: the iterates spiral out, for every step size.

Step 3: alternating steps. The generator steps first, and the discriminator then uses the new $\theta$. The linearized step matrix is $\begin{pmatrix} 1 & \eta/2 \\ -\eta/2 & 1 - \eta^2/4 \end{pmatrix}$. Its determinant is exactly 1, and for $\eta < 4$ its eigenvalues lie on the unit circle. The iterates go around a closed curve: they do not diverge, but they never converge.

Step 4: a gradient penalty. The R1 penalty (Mescheder et al. 2018) adds $\tfrac{\gamma}{2}\,\mathbb E_{p_{\text{data}}}\|\nabla_x l(x)\|^2$ to the discriminator loss. Here $\nabla_x l = \psi$, so the discriminator now maximizes $V - \tfrac{\gamma}{2}\psi^2$ and $\dot\psi = -\theta\sigma(\psi\theta) - \gamma\psi$. The Jacobian becomes $\begin{pmatrix} 0 & 1/2 \\ -1/2 & -\gamma \end{pmatrix}$, with eigenvalues $\big(-\gamma \pm \sqrt{\gamma^2 - 1}\big)/2$. Their real part is negative, so small enough steps converge.

Numbers (checked in section 5): from $(\theta, \psi) = (1, 0)$ with $\eta = 0.1$, after 2000 steps the distance to $(0, 0)$ is $3.05$ for simultaneous steps, stays near $1$ for alternating steps, and is below $10^{-10}$ with R1 and $\gamma = 1$.

Interactive figure: loading…

Mode collapse #

Mode collapse means that the generator maps many values of $z$ to one or a few modes of the data: samples look good, but they have low diversity. Partial collapse (the same texture or the same pose again and again) is common.

Why it happens: fix $D$. The generator's best response is to send every $z$ to $\arg\max_x D(x)$, a point mass. The discriminator then learns to reject that point, and the generator jumps to the next maximum: mode hopping. Gradient steps on both players do not clearly solve $\min_G \max_D$; they often behave like $\max_D \min_G$, whose inner problem is exactly this point mass (Goodfellow 2016, §5.1.1). The Dirac-GAN shows the same kind of cycling in one dimension.

Interactive figure: loading…

Remedies: let $D$ see batch statistics (minibatch discrimination, the minibatch standard-deviation feature), unrolled GANs, smoother critics (WGAN-GP, spectral normalization, the R1 penalty), two time-scale learning rates, and class conditioning. Measure coverage with recall (lesson 14), not with the loss.

f-GAN: a variational bound for any f-divergence #

For a convex function $f$ with $f(1) = 0$, the f-divergence is $D_f(P \,\|\, Q) = \mathbb E_Q[f(p/q)]$ (KL is $f(u) = u\log u$). The convex conjugate of $f$ is $f^*(t) = \sup_u\,[\,tu - f(u)\,]$.

Theorem (f-GAN bound; Nguyen, Wainwright and Jordan 2010; Nowozin et al. 2016). Assume $q > 0$ wherever $p > 0$. For every function $T$,

For differentiable $f$, equality holds at $T^*(x) = f'\big(p(x)/q(x)\big)$.

Proof.

1. From the definition of $f^*$: $f^*(t) \ge tu - f(u)$ for all $u, t$, so $f(u) \ge tu - f^*(t)$ (the Fenchel–Young inequality). Equality holds when $t = f'(u)$.
2. Use it at each $x$ with $u = p(x)/q(x)$ and $t = T(x)$, and multiply by $q(x) \ge 0$: $q f(p/q) \ge p\,T - q\,f^*(T)$.
3. Integrate over $x$: $D_f(P \,\|\, Q) \ge \mathbb E_P[T] - \mathbb E_Q[f^*(T)]$ for every $T$. A sup over a critic network gives a lower bound; it is tight if the network can represent $T^*$.
4. Step 1 is an equality at every $x$ when $T(x) = f'(p(x)/q(x))$. $\blacksquare$

KL. For $f(u) = u\log u$, setting the derivative of $tu - u\log u$ to zero gives $t - 1 - \log u = 0$, so $u = e^{t-1}$ and $f^*(t) = e^{t-1}$. The bound is $\mathrm{KL}(P \,\|\, Q) \ge \mathbb E_P[T] - \mathbb E_Q[e^{T-1}]$, with $T^* = 1 + \log(p/q)$.

The GAN. With $f(u) = u\log u - (u + 1)\log(u + 1)$ and $T = \log D$, the bound becomes $V(D, G)$, and $D_f = 2\,\mathrm{JSD} - \log 4$. This $f$ has $f(1) = -\log 4$, not 0; the proof above never uses $f(1) = 0$, so the bound still holds. The message: every such critic estimates a function of the density ratio $p/q$. The details for all common $f$ are in information theory, lesson 08.

Wasserstein-1 and the Kantorovich–Rubinstein dual #

A coupling $\pi \in \Pi(P, Q)$ is a joint distribution of $(x, y)$ whose marginals are $P$ and $Q$: a plan that says how much mass moves from each $x$ to each $y$. The Wasserstein-1 distance is the cost of the cheapest plan:

In 1D, $W_1 = \int |F_P(t) - F_Q(t)|\,dt$ with the cumulative distribution functions $F_P, F_Q$; for point masses at 0 and $\theta$ this is $|\theta|$. A function $f$ is 1-Lipschitz ($\|f\|_L \le 1$) if $|f(x) - f(y)| \le \|x - y\|$ for all $x, y$.

Theorem (Kantorovich–Rubinstein duality). For $P, Q$ with finite first moments,

Proof of the easy half ($\sup \le \inf$).

1. Take any coupling $\pi$ and any 1-Lipschitz $f$. Because $\pi$ has marginals $P$ and $Q$, $\mathbb E_P[f] - \mathbb E_Q[f] = \mathbb E_\pi[f(x) - f(y)]$.
2. The Lipschitz condition gives $f(x) - f(y) \le \|x - y\|$, so this is at most $\mathbb E_\pi\|x - y\|$.
3. Take the sup over $f$ on the left and the inf over $\pi$ on the right. $\blacksquare$

Equality is the Kantorovich–Rubinstein theorem: strong duality of the transport linear program (see duality).

Why 1-Lipschitz. Without a constraint the sup is $+\infty$ for any $P \ne Q$: multiply a good $f$ by 100. The Lipschitz bound says the critic can charge at most $\|x - y\|$ for mass moved from $y$ to $x$, which ties its value to the transport cost. A $K$-Lipschitz bound gives $K \cdot W_1$, which is fine for training.

WGAN (Arjovsky, Chintala and Bottou 2017). The critic $f_w$ (no sigmoid, no log) maximizes $\mathbb E_{\text{data}}[f_w(x)] - \mathbb E_z[f_w(G(z))]$; the generator minimizes $-\mathbb E_z[f_w(G(z))]$. Under mild conditions, $\nabla_\theta W_1(P_{\text{data}}, P_\theta) = -\mathbb E_z[\nabla_\theta f_{\text{opt}}(G_\theta(z))]$ with the optimal critic $f_{\text{opt}}$ (their Theorem 3).

The parallel-lines example #

Let $U \sim \mathcal U[0, 1]$, let $P_0$ be the law of $(0, U)$ and $P_\theta$ the law of $(\theta, U)$: two parallel segments at horizontal distance $|\theta|$.

• $W_1 = |\theta|$. Upper bound: the coupling $(0, u) \mapsto (\theta, u)$ moves every point by $|\theta|$. Lower bound: the 1-Lipschitz critic $f(x) = -\operatorname{sign}(\theta)\,x_1$ gives $\mathbb E_{P_0}[f] - \mathbb E_{P_\theta}[f] = |\theta|$.
• For $\theta \ne 0$ the supports are disjoint: $\mathrm{JSD} = \log 2$, $\mathrm{KL} = +\infty$ (both directions), and the total variation distance is 1. All three are 0 at $\theta = 0$.

Only $W_1$ is continuous in $\theta$. Its derivative $\operatorname{sign}\theta$ points to the answer. The others are flat (or infinite) and then jump at $\theta = 0$.

Interactive figure: loading…

Making the critic 1-Lipschitz #

Method	What it does	Weak points
Weight clipping (WGAN: $c = 0.01$, RMSProp, $n_{\text{critic}} = 5$)	clip every weight to $[-c, c]$; a bounded weight set gives some unknown constant $K$	the critic uses its capacity badly (too simple functions); gradients vanish or explode depending on $c$
Gradient penalty (WGAN-GP)	add $\lambda\,\mathbb E_{\hat x}(\|\nabla_x f(\hat x)\|_2 - 1)^2$ at $\hat x = \alpha x + (1 - \alpha)\tilde x$, $\alpha \sim \mathcal U[0, 1]$, $x$ real, $\tilde x$ fake	soft and local only; needs double backpropagation; invalid with batch norm in the critic
Spectral normalization	divide each weight matrix by its largest singular value (one power-iteration step per update)	the bound $\prod_l \|W_l\|_2$ is loose, so the critic can be far below 1-Lipschitz

• Why interpolates. A differentiable $f$ is 1-Lipschitz exactly when $\|\nabla f\| \le 1$ everywhere, and that cannot be enforced everywhere. The optimal critic has gradient norm 1 along the straight lines between points coupled by the optimal transport plan (Gulrajani et al. 2017, Proposition 1), so the penalty samples points on lines between real and fake samples.
• WGAN-GP defaults: $\lambda = 10$, $n_{\text{critic}} = 5$, Adam with learning rate $10^{-4}$, $\beta_1 = 0$, $\beta_2 = 0.9$. The penalty is per sample and batch norm mixes samples, so use layer norm or no normalization in the critic.
• Double backpropagation. The penalty contains $\nabla_x f$, so its gradient with respect to the critic weights differentiates a gradient (in PyTorch: torch.autograd.grad(..., create_graph=True)).
• Why spectral normalization works. The spectral norm $\|W\|_2$ is the Lipschitz constant of $x \mapsto Wx$. With 1-Lipschitz activations (ReLU, leaky ReLU), $\mathrm{Lip}(f) \le \prod_l \|W_l\|_2 = 1$. It is cheap, and it also stabilizes ordinary discriminators.

Why the R1 penalty and WGAN-GP are different

R1 penalizes $\|\nabla_x l\|^2$ on real data only and pushes it toward 0. Near the equilibrium, where the generator matches the data, a discriminator with zero slope on the data is exactly what we want, so R1 does not move the equilibrium; it only damps the rotation of the game (the $-\gamma$ in the Jacobian above). WGAN-GP pushes the gradient norm toward 1 on interpolates, because the optimal Wasserstein critic has slope 1 along transport lines. StyleGAN2 and many modern GANs use the non-saturating loss with R1.

Answers to section 1. (1) The gradient is $-\sigma(-8) \approx -3.4 \times 10^{-4}$: almost zero. This is why the non-saturating loss is used. (2) $\mathrm{JSD} = \log 2 \approx 0.693$ for both $\theta = 1$ and $\theta = 0.1$; $W_1 = 1$ and $0.1$. Only $W_1$ tells the generator that it got closer. (3) They move away: simultaneous steps multiply the distance by $\sqrt{1 + \eta^2/4} > 1$ per step near the equilibrium. Alternating steps circle, and only a penalty such as R1 makes them converge.

4. Worked example #

(a) Optimal discriminator. $p_{\text{data}} = (0.6, 0.4, 0)$ and $p_g = (0.2, 0.4, 0.4)$.

1. $D^* = p_{\text{data}}/(p_{\text{data}} + p_g)$ on each outcome: $0.6/0.8 = 0.75$, $0.4/0.8 = 0.5$ and $0/0.4 = 0$.
2. $V(D^*, G)$ is $\sum p_{\text{data}}\log D^*$ plus $\sum p_g \log(1 - D^*)$. The term $0 \cdot \log 0$ counts as 0:

3. The average is $m = (0.4, 0.4, 0.2)$. Then $\mathrm{KL}(p_{\text{data}} \,\|\, m) = 0.6\log 1.5 = 0.243279$ and $\mathrm{KL}(p_g \,\|\, m) = 0.2\log 0.5 + 0.4\log 2 = 0.138629$ (the terms with ratio 1 are 0).
4. JSD is the mean of the two KL values: $\mathrm{JSD} = 0.190954$.
5. Check: $-\log 4 = -1.386294$ and $2 \cdot 0.190954 = 0.381908$. The sum is $-1.004386$. ✓

(b) f-GAN bound for KL on the same pair ($P = p_{\text{data}}$, $Q = p_g$).

1. $\mathrm{KL}(P \,\|\, Q) = 0.6\log 3 = 0.659167$ (the second term is $0.4\log 1 = 0$).
2. The optimal critic is $T^* = 1 + \log(p/q)$. On the three outcomes: $1 + \log 3$, $1$ and $-\infty$. On outcome 3 it must be $-\infty$, because $P$ never visits it.
3. A real network outputs a finite value there. Take $T = (1 + \log 3,\ 1,\ 0)$. Then $\mathbb E_P[T] = 0.6 \cdot 2.098612 + 0.4 \cdot 1 = 1.659167$.
4. $\mathbb E_Q[e^{T-1}] = 0.2 \cdot 3 + 0.4 \cdot 1 + 0.4\,e^{-1}$, which is $1.147152$.
5. The bound is $1.659167 - 1.147152 = 0.512016$. It is below KL. The gap is exactly the term $q_3\,e^{T_3 - 1} = 0.4\,e^{-1} = 0.147152$ that the finite value $T_3 = 0$ adds.

(c) A Wasserstein critic for two Gaussians. $P = \mathcal N(\mu_1, \Sigma)$ and $Q = \mathcal N(\mu_2, \Sigma)$ with $\mu_1 = (1, 2)$ and $\mu_2 = (4, 6)$.

1. Translation by $\mu_1 - \mu_2$ is a coupling that moves every point by $\|\mu_1 - \mu_2\| = \|(-3, -4)\| = 5$, so $W_1 \le 5$.
2. The linear critic $f(x) = w^\top x$ with $\|w\| = 1$ is 1-Lipschitz and has value $w^\top(\mu_1 - \mu_2) \le 5$ (Cauchy–Schwarz), with equality at $w = (-0.6, -0.8)$. So $W_1 = 5$.
3. Now use the gradient penalty with $\lambda = 10$ instead of the hard constraint, and let the critic be $w = r\,(-0.6, -0.8)$. Its gradient norm is $r$ everywhere, so the penalized objective is $5r - 10(r - 1)^2$.
4. Set the derivative to zero: $5 - 20(r - 1) = 0$, so $r = 1.25$, and the value is $6.25 - 10 \cdot 0.0625 = 5.625$.
5. In general, for a linear critic and two distributions that differ by a shift of length $W$, $r^* = 1 + W/(2\lambda)$ and the value is $W + W^2/(4\lambda)$. A soft penalty lets the gradient norm go above 1 and overestimates $W_1$ a little. An unconstrained $w = (-3, -4)$ would pay the penalty $10(5 - 1)^2 = 160$.

5. In code #

NumPy and SciPy check the numbers of sections 3 and 4: the optimal discriminator, the logit gradients (against finite differences), the Dirac-GAN, and the exact $W_1$ of the parallel segments (an optimal assignment between equal-size samples is an optimal transport plan).

import numpy as np
from scipy.special import expit as sigmoid, rel_entr
from scipy.spatial.distance import jensenshannon
from scipy.optimize import linear_sum_assignment

# (a) optimal discriminator, V(D*) and JSD for the three-outcome example
p = np.array([0.6, 0.4, 0.0])                       # p_data
q = np.array([0.2, 0.4, 0.4])                       # p_g
D = p / (p + q)
V = rel_entr(p, p + q).sum() + rel_entr(q, p + q).sum()   # sum p log D* + sum q log(1 - D*)
jsd = jensenshannon(p, q) ** 2                      # SciPy returns the square root of JSD
print(D, round(V, 6), round(-np.log(4) + 2 * jsd, 6))
# [0.75 0.5  0.  ] -1.004386 -1.004386

# (b) generator-loss gradients with respect to the logit l, checked by finite differences
softplus = lambda u: np.logaddexp(0.0, u)
sat = lambda l: -softplus(l)                        # log(1 - sigmoid(l))
ns = lambda l: softplus(-l)                         # -log sigmoid(l)
l, h = -8.0, 1e-6
for f, exact in [(sat, -sigmoid(l)), (ns, -sigmoid(-l))]:
    print(f"{exact:.6f} {(f(l + h) - f(l - h)) / (2 * h):.6f}")
# -0.000335 -0.000335
# -0.999665 -0.999665

# (c) Dirac-GAN: simultaneous vs alternating gradient steps from (theta, psi) = (1, 0)
def dirac(eta=0.1, n=2000, alternating=False, gamma=0.0):
    th, ps = 1.0, 0.0
    for _ in range(n):
        g_th = ps * sigmoid(ps * th)                # -dV/dtheta (generator descends V)
        if alternating:
            th = th + eta * g_th
        g_ps = -th * sigmoid(ps * th) - gamma * ps  # dV/dpsi minus the R1 term (discriminator ascends)
        if not alternating:
            th = th + eta * g_th
        ps = ps + eta * g_ps
    return np.hypot(th, ps)
print(round(dirac(), 2), round(dirac(alternating=True), 2), dirac(gamma=1.0) < 1e-10)
# 3.05 0.99 True

# (d) parallel lines: exact W1 between 200 + 200 samples (optimal assignment)
u = (np.arange(200) + 0.5) / 200
for theta in [1.0, 0.1]:
    a = np.stack([np.zeros(200), u], 1)
    b = np.stack([np.full(200, theta), u], 1)
    C = np.linalg.norm(a[:, None] - b[None], axis=2)
    r, c = linear_sum_assignment(C)
    print(theta, round(C[r, c].mean(), 4))
# 1.0 1.0
# 0.1 0.1

The notebook adds the continuous version of (a) by quadrature, $W_1$ of 1D samples by sorting, a WGAN-GP critic trained by gradient ascent, and an optional tiny GAN on the 8 Gaussians in PyTorch.

6. Where it shows up #

• Adversarial losses inside other generators. The autoencoder of latent diffusion is trained with a perceptual loss plus a patch-based adversarial loss (Rombach et al. 2022, §3.1), so the latent space of lesson 13 was shaped by a GAN loss. Adversarial diffusion distillation uses a discriminator to get samplers with 1–4 steps (lesson 12). StyleGAN-type models use the non-saturating loss with the R1 penalty.
• Super-resolution and image-to-image translation. SRGAN, pix2pix and CycleGAN add an adversarial loss to a reconstruction loss. The adversarial part makes the output sharp; the reconstruction part keeps it faithful to the input.
• Weather and climate. DGMR (Ravuri et al., Nature 2021) nowcasts precipitation with a conditional GAN that has a spatial and a temporal discriminator. In an ensemble, mode collapse means an under-dispersed ensemble: the spread is too small, and the CRPS and the rank histogram show it (lesson 14). This is one reason recent ensemble weather models use diffusion.
• Seismology. Florez et al. (BSSA 2022) generate three-component accelerograms with a conditional Wasserstein GAN, conditioned on magnitude, distance and $V_{S30}$. The same task is now also done with (latent) diffusion and flow models.
• Density ratios in science. "Train a classifier, read off $p/q$" also drives neural ratio estimation for likelihood-free inference in particle physics and cosmology.

7. Common mistakes #

• log(sigmoid(l)) in code. It gives $-\infty$ or NaN for large $|l|$. Use the softplus forms or a fused "binary cross-entropy with logits" function.
• Not detaching the fakes in the discriminator step. Then the $D$ loss also sends gradients into $G$: wasted compute, and wrong $G$ updates if those gradients are not zeroed before the $G$ step.
• Reading $C(G) = -\log 4 + 2\,\mathrm{JSD}$ as what training minimizes. It holds only for the optimal $D$, and the non-saturating loss is a different game with the same fixed point.
• Expecting gradient play to converge like gradient descent. A game has a vector field, not a loss surface. Its Jacobian can have imaginary eigenvalues (the Dirac-GAN), and then plain gradient steps rotate or spiral out. A smaller learning rate does not fix this.
• A sigmoid or a log on the WGAN critic. The critic outputs unbounded scores; the losses are plain means.
• Gradient-penalty slips. The penalty uses $\nabla_x f(\hat x)$ (with respect to the inputs, not the weights), and batch norm in the critic breaks the per-sample penalty.
• Using the GAN loss as a quality metric. Discriminator and generator losses oscillate and say little about sample quality or coverage. Use FID, precision and recall (lesson 14).
• Blaming mode collapse only on JSD. GANs that approximate $\mathrm{KL}(p_{\text{data}} \,\|\, p_g)$ collapse too (Goodfellow 2016, §5.1.1); the game dynamics matter as much as the divergence.

8. Check yourself #

1. $p_{\text{data}}$ and $p_g$ have disjoint supports. Give $D^*$, $C(G)$, and the generator gradient of the saturating loss near the samples.
   Answer

   $D^* = 1$ on the data support and $0$ on the generator support. $\mathrm{JSD} = \log 2$, so $C(G) = -\log 4 + 2\log 2 = 0$, the same for every disjoint $p_g$. $D$ is constant near the samples, so $\nabla_x D = 0$ and the generator gets no gradient.
2. The data are three equal modes, $p_{\text{data}} = (\tfrac13, \tfrac13, \tfrac13)$, and the generator collapses to the first: $p_g = (1, 0, 0)$. Compute JSD, $\mathrm{KL}(p_g \,\|\, p_{\text{data}})$ and $\mathrm{KL}(p_{\text{data}} \,\|\, p_g)$. Which one punishes the collapse most?
   Answer

   $m = (\tfrac23, \tfrac16, \tfrac16)$; $\mathrm{KL}(p_{\text{data}} \,\|\, m) = \tfrac13\log\tfrac12 + \tfrac23\log 2$, which is $\tfrac13\log 2 = 0.2310$, $\mathrm{KL}(p_g \,\|\, m) = \log 1.5 = 0.4055$, so $\mathrm{JSD} = 0.3183$. Reverse KL: $\log 3 = 1.0986$ (finite). Forward KL: $+\infty$, because $p_g = 0$ where the data have mass. Forward KL (maximum likelihood) punishes the collapse most; reverse KL and JSD give finite penalties (JSD is at most $\log 2$).
3. In the Dirac-GAN with simultaneous steps, compute the growth factor per step near $(0, 0)$ for $\eta = 0.1$, and estimate the distance after 100 steps from a start at distance $0.01$.
   Answer

   $\sqrt{1 + 0.01/4} = \sqrt{1.0025} = 1.00125$. Near $(0, 0)$ the linearization is accurate, so after 100 steps the distance is about $0.01 \cdot 1.00125^{100} = 0.01 \cdot 1.133 = 0.0113$. Far from $(0, 0)$ the sigmoid factor changes and the growth slows down (the full nonlinear run from distance 1 reaches only 3.05 after 2000 steps).
4. In the KL bound, use the critic $T = T^* + c$ for a constant $c$. Show that the bound equals $\mathrm{KL} + 1 + c - e^{c}$, and conclude where it is tight.
   Answer

   $\mathbb E_P[T^* + c] = 1 + \mathrm{KL} + c$, because $\mathbb E_P[\log(p/q)] = \mathrm{KL}$. And $\mathbb E_Q[e^{T^* + c - 1}] = e^{c}\,\mathbb E_Q[p/q] = e^{c}$. So the bound is $\mathrm{KL} + 1 + c - e^{c}$. Since $e^{c} \ge 1 + c$, it is at most KL, with equality only at $c = 0$.
5. A ReLU critic has three layers with spectral norms 2, 0.5 and 3. Give a Lipschitz bound before and after spectral normalization. Can the true constant be smaller?
   Answer

   Before: $2 \cdot 0.5 \cdot 3 = 3$. After: $1$. Yes: the bound is tight only if the top singular directions of consecutive layers line up and the ReLUs pass them unchanged. Usually neither holds, so the true constant is often much smaller than the product.
6. Why is the WGAN-GP penalty evaluated on random interpolates between real and fake samples, and not only on real data or on a fixed grid?
   Answer

   The optimal critic has unit gradient norm along the straight lines between optimally coupled real and fake points (Gulrajani et al., Proposition 1). Enforcing $\|\nabla f\| \le 1$ everywhere is impossible in high dimension. Lines between real and fake samples are a cheap proxy for the region where the constraint is active (the pairs are random, not optimally coupled), and they move with the generator during training. Real data alone would leave the region between the two distributions unconstrained.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. What is the optimal discriminator $D^*(x)$ of the original GAN game for a fixed generator?

   Show answer

2. What is $C(G) = \max_D V(D, G)$ in terms of a divergence, and its minimum?

   Show answer

3. What are the gradients of the saturating and the non-saturating generator loss with respect to the logit $l$ of a fake?

   Show answer

4. Why does JSD give no gradient when the data and the generator have disjoint supports?

   Show answer

5. What is mode collapse?

   Show answer

6. Why do simultaneous gradient steps not converge in the Dirac-GAN?

   Show answer

7. State the f-GAN lower bound and its optimal critic.

   Show answer

8. State the Kantorovich–Rubinstein dual of $W_1$.

   Show answer

9. What is the WGAN-GP penalty, and where is it evaluated?

   Show answer

10. For two parallel segments at distance $\theta \ne 0$, what are JSD and $W_1$?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "What does the discriminator learn at the optimum?" $D^* = p_{\text{data}}/(p_{\text{data}} + p_g)$, so $D^*/(1 - D^*)$ is the density ratio. The game then minimizes $2\,\mathrm{JSD} - \log 4$.
• "Why the non-saturating loss?" The minimax gradient with respect to the logit is $-\sigma(l) \approx 0$ when $D$ rejects fakes; the non-saturating one is $-\sigma(-l) \approx -1$. Same fixed point.
• "Why is GAN training unstable?" Three reasons: the divergence can be flat (disjoint supports), the game dynamics rotate (imaginary eigenvalues; the Dirac-GAN), and the best response to a fixed $D$ is a point mass (mode collapse).
• "Why does WGAN help?" $W_1$ is continuous in the generator parameters even for disjoint supports (parallel lines), so the critic gives a useful gradient and a loss that tracks progress.
• "How do you enforce the Lipschitz constraint?" Clipping (crude), a gradient penalty on interpolates ($\lambda = 10$, no batch norm), spectral normalization (cheap, loose bound).
• "GAN or diffusion for an ensemble forecast?" A GAN needs one network pass per sample, but it can collapse modes and give an under-dispersed ensemble. Diffusion trains by stable regression and covers modes better, at the cost of many network evaluations per sample.
• "How do you detect mode collapse?" Recall or coverage metrics, nearest training neighbors, counting modes on a toy mixture, class histograms with a pretrained classifier.

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution, and an optional PyTorch demonstration:

1. $D^*$, the value $V(D^*)$ and JSD for discrete distributions; then the same identity for two 1D densities by quadrature.
2. Saturating and non-saturating losses and their gradients in the logit, written in a stable form.
3. The Dirac-GAN: simultaneous, alternating and R1 steps; plot the three trajectories.
4. The f-GAN bound for KL: write the bound for a critic vector and find the critic that makes it equal to KL.
5. $W_1$ of two 1D samples by sorting, and the WGAN-GP optimum of a linear critic, $r^* = 1 + W/(2\lambda)$.
6. Optional (PyTorch): a tiny GAN on the 8 Gaussians; count the covered modes during training.

11. Watch and read #

Understanding GANs (Generative Adversarial Networks)DeepBean · 27 min

The GAN game, the optimal discriminator and the link to the Jensen–Shannon divergence, derived step by step.

L5 GANs -- CS294-158 SP24 Deep Unsupervised Learning -- UC BerkeleyPieter Abbeel · Berkeley CS294-158 Deep Unsupervised Learning · 2 h 32 min

A full lecture on GANs: the original game, training problems, and the main fixes and architectures.

Lecture 6 Implicit Models / GANs part II --- CS294-158-SP20 Deep Unsupervised Learning -- BerkeleyPieter Abbeel · Berkeley CS294-158 Deep Unsupervised Learning · 2 h 9 min

The second GAN lecture: f-GAN, Wasserstein GAN, the gradient penalty and spectral normalization.

• Goodfellow et al., Generative Adversarial Networks (2014), §4.1: Proposition 1 and Theorem 1 ($D^*$ and $C(G)$).
• Goodfellow, NIPS 2016 Tutorial: Generative Adversarial Networks: §3.2 (cost functions, the non-saturating game), §5.1 (non-convergence and mode collapse), §8.2 (gradient descent on $V = xy$).
• Mescheder, Geiger and Nowozin, Which Training Methods for GANs do actually Converge? (ICML 2018): the Dirac-GAN and the R1 penalty.
• Nowozin, Cseke and Tomioka, f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (2016).
• Arjovsky, Chintala and Bottou, Wasserstein GAN (2017): Example 1 (parallel lines) and Theorem 3; Gulrajani et al., Improved Training of Wasserstein GANs (2017): Proposition 1 and Algorithm 1; Miyato et al., Spectral Normalization for Generative Adversarial Networks (2018).
• Murphy, Probabilistic Machine Learning: Advanced Topics, free PDF: chapter 26 (§26.2 density ratios and divergences, §26.3 training challenges).
• Lilian Weng, From GAN to WGAN: a readable overview.