Continuous variables: differential entropy, Gaussians and forward vs reverse KL

Before this lesson: lesson 01 (entropy), lesson 03 (KL divergence, Gibbs' inequality) and lesson 04 (cross-entropy, maximum likelihood is forward KL). Densities and the multivariate Gaussian from probability lesson 05; densities under a change of variables from calculus lesson 10; the determinant as a volume from linear algebra lesson 03 and covariance matrices from linear algebra lesson 07. Time: about 75 minutes, plus the notebook. This lesson is long; take a break after section 3.4.

---

Why this matters #

• VAEs and variational inference. The encoder of a VAE outputs a Gaussian, and the loss contains the KL between two Gaussians, in closed form (section 3.6). VI minimizes the reverse KL to the posterior; this explains why VI posteriors are often too narrow (section 3.7).
• Maximum likelihood and distillation. Training by log-loss minimizes the forward KL from the data to the model (lesson 04). It covers all the data, even at the price of putting mass where there is no data.
• Normalizing flows. A flow maps a simple density through an invertible network. Its log-density contains $\log|\det J|$, the same term as in the change of variables for $h$ (section 3.3).
• Gaussian uncertainty in science. The entropy of a Gaussian posterior, $\frac12\log\det(2\pi e\Sigma)$, measures the remaining uncertainty in Bayesian experimental design and data assimilation. It is meaningful only in fixed units, or as a difference.
• Mutual information. Differential entropy changes with the units, but differences of differential entropies such as the mutual information do not (lesson 07).

In this track $\log$ is the natural logarithm and gives nats; $\log_2$ gives bits. For densities we work in nats, as NumPy, SciPy and PyTorch do.

1. Predict first #

2. The picture #

2.1 From bins to densities #

Let $X$ have a density $p(x)$. Cut the real line into bins of width $\Delta$ and record only the bin that $X$ falls in. This gives a discrete variable $X_\Delta$, and discrete variables have an entropy $H(X_\Delta)$ (lesson 01). What happens when the bins get smaller?

1. A small bin $i$ around a point $x_i$ has probability $P_i \approx p(x_i)\,\Delta$ (height times width).
2. So $H(X_\Delta) = -\sum_i P_i \log P_i \approx -\sum_i p(x_i)\Delta\,\log\big(p(x_i)\Delta\big)$.
3. Split the log of the product: $\log(p(x_i)\Delta) = \log p(x_i) + \log\Delta$. This gives two sums:

4. The first sum is a Riemann sum for $-\int p(x)\log p(x)\,dx$. In the second sum, $\sum_i p(x_i)\Delta \approx 1$ (the total probability). So

The number $h(X)$ is the differential entropy. The term $\log(1/\Delta)$ grows without bound: each halving of $\Delta$ adds one more yes/no question (1 bit) to locate $X$. An exact real number needs infinitely many bits. The differential entropy is what remains after we remove this infinite part. It is not the limit of $H(X_\Delta)$; that limit is $+\infty$.

Interactive figure: loading…

Why $h$ can be negative. A density is not a probability. It can be larger than 1, as long as the area under it is 1. The uniform density on $[0, \frac12]$ equals 2 on its interval, so $-\log p(x) = -\log 2 < 0$ everywhere, and $h = -\log 2 = -1$ bit. A discrete entropy is never negative, because every probability is at most 1.

Why $h$ depends on the units. The bin width $\Delta$ has units. One millimetre is $\frac{1}{1000}$ of a metre, so the same physical bins give $\log(1/\Delta)$ values that differ by $\log 1000$. The remainder $h$ changes by the same amount (section 3.3).

2.2 Two ways to fit one Gaussian #

The second picture is about fitting. A target density $p$ has two bumps. We want the single Gaussian $q = \mathcal{N}(\mu, \sigma^2)$ that is "closest" to $p$. KL is not symmetric (lesson 03), so there are two different questions.

• Forward KL, $\mathrm{KL}(p\,\|\,q) = \mathbb{E}_{x \sim p}\big[\log p(x) - \log q(x)\big]$. The average is over $p$. Where $p$ has mass and $q$ is tiny, $-\log q(x)$ is huge. So $q$ must put mass everywhere $p$ has mass. The fit is wide and covers both bumps: mode covering.
• Reverse KL, $\mathrm{KL}(q\,\|\,p) = \mathbb{E}_{x \sim q}\big[\log q(x) - \log p(x)\big]$. The average is over $q$. Where $q$ has mass and $p$ is tiny, $-\log p(x)$ is huge. So $q$ must avoid every region where $p$ is small, including the gap between the bumps. The fit sits on one bump: mode seeking.

Interactive figure: loading…

3. Definitions and main results #

3.1 Differential entropy #

Let $X$ be a continuous random variable with density $p$ on $\mathbb{R}^d$. Its differential entropy is

with $0\log 0 = 0$, when the integral exists. We write lower-case $h$ to separate it from the discrete $H$. Two examples, step by step:

• Uniform on $[0, a]$. $p(x) = 1/a$ on the interval, so $-\log p(x) = \log a$ for every $x$ that can occur. The average of a constant is the constant: $h = \log a$. It is negative for $a < 1$.
• Gaussian $\mathcal{N}(\mu, \sigma^2)$. The density is $p(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\big(-\frac{(x - \mu)^2}{2\sigma^2}\big)$, so

Take the mean. $\mathbb{E}[(X - \mu)^2] = \sigma^2$, so the second term has mean $\frac12$. With $\frac12 = \frac12\log e$:

The mean $\mu$ does not appear: shifting a density does not change its shape. For $\sigma = 1$, $h = 1.419$ nats $= 2.047$ bits. $h < 0$ exactly when $2\pi e\sigma^2 < 1$, that is, $\sigma < 1/\sqrt{2\pi e} \approx 0.242$.

Theorem 1 (quantization). Let $p$ be continuous and let $p\log p$ be Riemann integrable. Let $X_\Delta$ be $X$ quantized into bins of width $\Delta$. Then $H(X_\Delta) - \log(1/\Delta) \to h(X)$ as $\Delta \to 0$.

Proof of Theorem 1 (Cover and Thomas, Theorem 8.3.1)

1. By the mean value theorem, each bin $[i\Delta, (i + 1)\Delta)$ contains a point $x_i$ with $P_i = \int_{\text{bin } i} p(x)\,dx = p(x_i)\,\Delta$ exactly.
2. Then $H(X_\Delta) = -\sum_i p(x_i)\Delta\log\big(p(x_i)\Delta\big) = -\sum_i p(x_i)\Delta\log p(x_i) - \log\Delta\sum_i p(x_i)\Delta$.
3. $\sum_i p(x_i)\Delta = \sum_i P_i = 1$, so $H(X_\Delta) + \log\Delta = -\sum_i p(x_i)\log p(x_i)\,\Delta$.
4. The right side is a Riemann sum of $-p\log p$, and it tends to $h(X)$ as $\Delta \to 0$. $\blacksquare$

In $d$ dimensions, cubes of side $\Delta$ give $H(X_\Delta) \approx h(X) + d\log(1/\Delta)$.

3.2 The multivariate Gaussian: entropy is a log-volume #

Let $X \sim \mathcal{N}(\mu, \Sigma)$ in $d$ dimensions, with $\Sigma$ positive definite. Its density is $p(x) = \big((2\pi)^d\det\Sigma\big)^{-1/2}\exp\big(-\frac12(x - \mu)^\top\Sigma^{-1}(x - \mu)\big)$, so

The quadratic term has mean $d$. (Proof: $\mathbb{E}[(X - \mu)^\top\Sigma^{-1}(X - \mu)] = \mathbb{E}[\operatorname{tr}(\Sigma^{-1}(X - \mu)(X - \mu)^\top)] = \operatorname{tr}(\Sigma^{-1}\Sigma) = \operatorname{tr}(I_d) = d$.) So

Only $\det\Sigma$ enters. The ellipse (in $d$ dimensions, the ellipsoid) $\{x : (x - \mu)^\top\Sigma^{-1}(x - \mu) \le 1\}$ has volume proportional to $\sqrt{\det\Sigma}$ (linear algebra lesson 03: the determinant scales volume). So the entropy of a Gaussian is the log of the volume of its $1\sigma$ ellipsoid, plus a constant. When two coordinates become almost perfectly correlated, the ellipse flattens to a line, $\det\Sigma \to 0$, and $h \to -\infty$.

Interactive figure: loading…

3.3 Change of variables: the units trap #

Theorem 2 (affine maps). Let $Y = AX + b$, where $A$ is an invertible $d \times d$ matrix and $b$ a vector. Then

Proof.

1. Change of variables (calculus lesson 10): the density of $Y$ is $p_Y(y) = p_X\big(A^{-1}(y - b)\big)/|\det A|$. A box of volume $V$ maps to a box of volume $|\det A|\,V$ with the same probability, so the density divides by $|\det A|$.
2. At $y = Ax + b$ this reads $-\log p_Y(y) = -\log p_X(x) + \log|\det A|$.
3. Take the mean over $X$. The shift $b$ has disappeared: moving a density does not change its shape. $\blacksquare$

Check with Gaussians. $\Sigma_Y = A\Sigma A^\top$ and $\det(A\Sigma A^\top) = (\det A)^2\det\Sigma$, so $\frac12\log\det\Sigma_Y = \frac12\log\det\Sigma + \log|\det A|$. ✓

The units trap. A change of units is a map $Y = cX$. Metres to millimetres is $c = 1000$, so $h$ grows by $\log 1000 = 6.908$ nats $= 9.966$ bits per coordinate, for the same physical uncertainty. So a differential entropy alone has no absolute meaning. Compare entropies only in the same units, or use quantities that do not depend on the units: KL divergence and mutual information.

Nonlinear maps and normalizing flows

For a smooth invertible map $g$ with Jacobian matrix $J_g$, the same three steps give $p_Y(g(x)) = p_X(x)/|\det J_g(x)|$ and

$$h(g(X)) = h(X) + \mathbb{E}\big[\log|\det J_g(X)|\big].$$

A normalizing flow uses the first identity in log form: $\log p_Y(y) = \log p_X(x) - \log|\det J_g(x)|$ with $x = g^{-1}(y)$. Flow architectures are designed so that $\log|\det J_g|$ is cheap (triangular Jacobians). Example: $X \sim \mathcal{N}(0, 1)$ and $g(x) = x^3$: $h(X^3) = h(X) + \mathbb{E}[\log 3X^2] = 1.419 + 1.099 - 1.270 = 1.247$ nats, where $\mathbb{E}[\log X^2] = -(\gamma + \log 2) = -1.270$ ($\gamma = 0.5772$ is Euler's constant). If $A$ is singular, $Y = AX$ lives on a lower-dimensional subspace and has no density on $\mathbb{R}^d$; its differential entropy is $-\infty$ in the limit.

3.4 KL divergence for densities: no units problem #

For densities $p$ and $q$ on $\mathbb{R}^d$, the KL divergence is

It is finite only if $q(x) > 0$ wherever $p(x) > 0$ (up to sets of zero probability). Gibbs' inequality still holds: $\mathrm{KL}(p\,\|\,q) \ge 0$, with equality only when $p = q$ (almost everywhere). The proof of lesson 03 works with integrals in place of sums. Under mild conditions the KL of the quantized versions converges to it: $\mathrm{KL}(p_\Delta\,\|\,q_\Delta) \to \mathrm{KL}(p\,\|\,q)$, because the two $\log(1/\Delta)$ terms cancel.

Theorem 3 (KL does not depend on the coordinates). Let $g$ be a smooth invertible map, and let $p_Y$, $q_Y$ be the densities of $g(X)$ when $X \sim p$ and when $X \sim q$. Then $\mathrm{KL}(p_Y\,\|\,q_Y) = \mathrm{KL}(p\,\|\,q)$.

Proof.

1. Both densities pick up the same Jacobian factor: $p_Y(g(x)) = p(x)/|\det J_g(x)|$ and $q_Y(g(x)) = q(x)/|\det J_g(x)|$.
2. In the ratio the factor cancels: $\frac{p_Y(g(x))}{q_Y(g(x))} = \frac{p(x)}{q(x)}$.
3. The KL is the mean of the log of this ratio under $p$, so it does not change. $\blacksquare$

So measuring in millimetres instead of metres changes every $h$, but no KL divergence and no mutual information. This is why ML objectives use KL and mutual information, not $h$ alone.

3.5 The Gaussian has maximum entropy #

Theorem 4 (maximum entropy). Among all densities on $\mathbb{R}^d$ with mean $\mu$ and covariance $\Sigma$ ($\Sigma$ positive definite), the Gaussian $g = \mathcal{N}(\mu, \Sigma)$ has the largest differential entropy, $h(g) = \frac12\log((2\pi e)^d\det\Sigma)$, and it is the only one.

Why this is true (idea). $\log g(x)$ is a quadratic function of $x$. The average of a quadratic function depends only on the mean and the covariance. So every density with the same mean and covariance has the same "cross-entropy with $g$". Gibbs' inequality then does the rest.

Proof.

1. Let $p$ have mean $\mu$ and covariance $\Sigma$. By Gibbs, $0 \le \mathrm{KL}(p\,\|\,g) = \mathbb{E}_p[\log p] - \mathbb{E}_p[\log g] = -h(p) - \mathbb{E}_p[\log g(X)]$.
2. $\log g(x) = -\frac12\log((2\pi)^d\det\Sigma) - \frac12(x - \mu)^\top\Sigma^{-1}(x - \mu)$.
3. The quadratic term has mean $\operatorname{tr}(\Sigma^{-1}\Sigma) = d$ under $p$, exactly as under $g$ (section 3.2). So $\mathbb{E}_p[\log g(X)] = \mathbb{E}_g[\log g(X)] = -h(g)$.
4. Step 1 becomes $0 \le -h(p) + h(g)$, so $h(p) \le h(g)$. Equality holds only if $\mathrm{KL}(p\,\|\,g) = 0$, that is, $p = g$. $\blacksquare$

Step 4 says more: for every $p$ with the same mean and covariance, $h(g) - h(p) = \mathrm{KL}(p\,\|\,g)$. The entropy gap to the Gaussian is a KL divergence.

Interactive figure: loading…

At unit variance: the two bumps of the figure have $h = 1.1659$ nats; uniform on $[-\sqrt3, \sqrt3]$ has $h = \log\sqrt{12} = 1.2425$ nats; Laplace with scale $b = 1/\sqrt2$ has $h = 1 + \log(2b) = 1.3466$ nats; the Gaussian has $1.4189$ nats. The maximum-entropy principle reads this backwards: if you know only the mean and the covariance of some data, the Gaussian is the model that assumes nothing else. Other constraints give other families: a fixed interval gives the uniform, a fixed mean of $|x|$ gives the Laplace.

3.6 KL between two Gaussians #

Theorem 5 (Gaussian KL). For $p_0 = \mathcal{N}(\mu_0, \Sigma_0)$ and $p_1 = \mathcal{N}(\mu_1, \Sigma_1)$ on $\mathbb{R}^d$,

In one dimension it becomes

Each term has a meaning: the trace compares the spreads, the quadratic term (the Mahalanobis distance of the means, measured in units of $\Sigma_1$) compares the means, and the log-det compares the volumes. Two cases to know by heart:

• Same variance 1: $\mathrm{KL}(\mathcal{N}(\mu, 1)\,\|\,\mathcal{N}(0, 1)) = \mu^2/2$. KL behaves like half a squared distance.
• The VAE term: $\mathrm{KL}\big(\mathcal{N}(\mu, \operatorname{diag}(\sigma^2))\,\|\,\mathcal{N}(0, I)\big) = \frac12\sum_j\big(\sigma_j^2 + \mu_j^2 - 1 - \log\sigma_j^2\big)$.

Proof of Theorem 5

1. $\mathrm{KL} = \mathbb{E}_0[\log p_0(X) - \log p_1(X)]$, where $\mathbb{E}_0$ is the mean under $p_0$ and $\log p_i(x) = -\frac d2\log 2\pi - \frac12\log\det\Sigma_i - \frac12(x - \mu_i)^\top\Sigma_i^{-1}(x - \mu_i)$.
2. One identity does all the work. If $Z$ has mean $m$ and covariance $C$, then $\mathbb{E}[Z^\top MZ] = \mathbb{E}[\operatorname{tr}(MZZ^\top)] = \operatorname{tr}\big(M(C + mm^\top)\big) = \operatorname{tr}(MC) + m^\top Mm$.
3. Under $p_0$, $X - \mu_0$ has mean 0 and covariance $\Sigma_0$, so $\mathbb{E}_0[(X - \mu_0)^\top\Sigma_0^{-1}(X - \mu_0)] = \operatorname{tr}(I_d) = d$.
4. $X - \mu_1$ has mean $\mu_0 - \mu_1$ and covariance $\Sigma_0$, so $\mathbb{E}_0[(X - \mu_1)^\top\Sigma_1^{-1}(X - \mu_1)] = \operatorname{tr}(\Sigma_1^{-1}\Sigma_0) + (\mu_0 - \mu_1)^\top\Sigma_1^{-1}(\mu_0 - \mu_1)$.
5. Subtract the two logs. The $\frac d2\log 2\pi$ terms cancel. The log-dets give $\frac12\log(\det\Sigma_1/\det\Sigma_0)$. The quadratic terms give $\frac12(\text{step 4} - \text{step 3})$. $\blacksquare$

Example. $\mathrm{KL}(\mathcal{N}(0, 1)\,\|\,\mathcal{N}(1, 2^2)) = \log 2 + \frac{1 + 1}{8} - \frac12 = 0.693 + 0.25 - 0.5 = 0.443$ nats. The other direction is $\log\frac12 + \frac{4 + 1}{2} - \frac12 = 1.307$ nats: not symmetric.

3.7 Forward and reverse KL #

Fix a target $p$ and a family of models $q_\theta$ (for example, all Gaussians). Section 2.2 showed the two fits in a picture. Now the precise statements.

Forward KL, $\min_\theta \mathrm{KL}(p\,\|\,q_\theta)$. Write $\mathrm{KL}(p\,\|\,q_\theta) = -h(p) - \mathbb{E}_p[\log q_\theta(X)]$. The first term does not depend on $\theta$. So minimizing the forward KL is maximizing $\mathbb{E}_p[\log q_\theta(X)]$: maximum likelihood (lesson 04). You need samples of $p$ (the data), not its density.

Theorem 6 (forward KL to a Gaussian is moment matching). Let $p$ have mean $m$ and covariance $C$ (positive definite), and let $g = \mathcal{N}(m, C)$. For every Gaussian $q$,

So the best Gaussian is $q = g$: match the mean and the covariance, and there is only one optimum.

Proof.

1. $\log q$ is a quadratic function of $x$, so $\mathbb{E}_p[\log q] = \mathbb{E}_g[\log q]$ (step 3 of Theorem 4).
2. $\mathrm{KL}(p\,\|\,q) = -h(p) - \mathbb{E}_g[\log q] = \big[h(g) - h(p)\big] + \big[-h(g) - \mathbb{E}_g[\log q]\big]$.
3. The second bracket is $\mathrm{KL}(g\,\|\,q)$. The first is $\mathrm{KL}(p\,\|\,g)$ (end of Theorem 4). The first does not depend on $q$; the second is $\ge 0$, with equality only for $q = g$. $\blacksquare$

For the two-bump target $p = 0.25\,\mathcal{N}(-3, 1) + 0.75\,\mathcal{N}(3, 1)$: the mean is $m = 0.25 \cdot (-3) + 0.75 \cdot 3 = 1.5$, and $\mathbb{E}[X^2] = 1 + 9 = 10$ (each bump has $\mathbb{E}[X^2] = 1 + 3^2$), so the variance is $10 - 1.5^2 = 7.75$. The forward fit is $q = \mathcal{N}(1.5, 7.75)$, a standard deviation of $2.78$. It covers both bumps and puts its peak at $x = 1.5$, where $p$ is small.

Reverse KL, $\min_\theta \mathrm{KL}(q_\theta\,\|\,p)$. Now $\mathrm{KL}(q_\theta\,\|\,p) = \mathbb{E}_{q_\theta}[\log q_\theta(X) - \log p(X)]$. Two consequences:

• Mode seeking and local optima. A $q_\theta$ on one bump avoids the gap, so it is a local optimum. There is one local optimum near each bump, and the start decides which one gradient descent finds. Why is the value close to $-\log w_k$, where $w_k$ is the weight of the bump? Where $q_\theta$ has mass, $p(x) \approx w_k\,\mathcal{N}(x; \mu_k, 1)$, because the other bump is far away. If $q_\theta$ equals that component, $\log\frac{q_\theta}{p} \approx \log\frac{1}{w_k}$ everywhere on its mass. For the example, $-\log 0.75 = 0.288$ and $-\log 0.25 = 1.386$ nats; the exact optima are $0.285$ and $1.378$.
• $p$ is needed only up to a constant. Suppose $p = \tilde p/Z$, where $\tilde p$ is known and the normalizing constant $Z$ is not. Then $\mathrm{KL}(q\,\|\,p) = \mathbb{E}_q[\log q - \log\tilde p] + \log Z$, and $\log Z$ does not depend on $\theta$. You need samples of $q$ (you choose $q$, so this is easy) and the unnormalized $\tilde p$. A Bayesian posterior $p(z \mid x) = p(x, z)/p(x)$ is exactly this case: $p(x, z)$ is known, $p(x)$ is not. This is why variational inference uses the reverse KL. The quantity $\log Z - \mathrm{KL}(q\,\|\,p)$ is the ELBO (lesson 09).

Mean-field fits are too narrow. A mean-field (factorized) family assumes independent coordinates: $q(z) = \prod_i q_i(z_i)$. Take a correlated Gaussian target and a factorized Gaussian $q$.

Theorem 7 (factorized Gaussian fits). Let $p = \mathcal{N}(0, \Sigma)$ and $q = \mathcal{N}(0, \operatorname{diag}(s_1^2, \dots, s_d^2))$.

• The forward KL is smallest for $s_i^2 = \Sigma_{ii}$: the marginal variances.
• The reverse KL is smallest for $s_i^2 = 1/(\Sigma^{-1})_{ii}$, which is never larger than $\Sigma_{ii}$.

Proof. Use Theorem 5 with $D = \operatorname{diag}(s_1^2, \dots, s_d^2)$ and zero means.

1. Forward: $\mathrm{KL}(p\,\|\,q) = \frac12\big[\sum_i \Sigma_{ii}/s_i^2 - d + \sum_i\log s_i^2 - \log\det\Sigma\big]$. The derivative with respect to $s_i^2$ is $\frac12\big[-\Sigma_{ii}/s_i^4 + 1/s_i^2\big]$. It is zero at $s_i^2 = \Sigma_{ii}$.
2. Reverse: $\mathrm{KL}(q\,\|\,p) = \frac12\big[\sum_i (\Sigma^{-1})_{ii}\,s_i^2 - d + \log\det\Sigma - \sum_i\log s_i^2\big]$. The derivative with respect to $s_i^2$ is $\frac12\big[(\Sigma^{-1})_{ii} - 1/s_i^2\big]$. It is zero at $s_i^2 = 1/(\Sigma^{-1})_{ii}$.
3. Each term has the form $cu - \log u$ with $c > 0$: in step 1 with $u = 1/s_i^2$, in step 2 with $u = s_i^2$. This function is convex in $u > 0$, so the stationary points are the minima.
4. $1/(\Sigma^{-1})_{ii}$ is the conditional variance $\operatorname{Var}(X_i \mid X_{\text{rest}})$ of a Gaussian, and conditioning never increases a Gaussian variance. So $1/(\Sigma^{-1})_{ii} \le \Sigma_{ii}$, with equality only when $X_i$ is uncorrelated with the other coordinates. $\blacksquare$

For $\Sigma = \begin{pmatrix} 1 & 0.9 \\ 0.9 & 1 \end{pmatrix}$, the forward fit has variances 1 and the reverse fit has variances $1 - 0.9^2 = 0.19$: a standard deviation of $0.44$ instead of 1. The reverse fit sits inside the narrow ellipse and reports far too little uncertainty. Mean-field VI does this in general (Bishop, PRML §10.1.2).

Interactive figure: loading…

	forward $\mathrm{KL}(p \Vert q)$	reverse $\mathrm{KL}(q \Vert p)$
large penalty where	$p > 0$ and $q \approx 0$	$q > 0$ and $p \approx 0$
behaviour	mode covering, often too wide	mode seeking, often too narrow
needs	samples of $p$	samples of $q$; $p$ up to a constant
Gaussian $q$	moment matching, one optimum	for a multimodal $p$: one local optimum per mode
used in	maximum likelihood, cross-entropy training, distillation	variational inference, ELBO, VAEs

Answers to section 1. (1) $h = \log a$, which is negative when $a < 1$ m. In millimetres, $X_{\text{mm}} = 1000X$, so $h$ grows by $\log 1000 = 6.91$ nats ($9.97$ bits) for the same physical uncertainty. (2) The forward KL gives the wide $q = \mathcal{N}(1.5, 2.78^2)$ that covers both bumps. The reverse KL gives a $q$ close to $\mathcal{N}(3, 1)$ or $\mathcal{N}(-3, 1)$, depending on the start. (3) Too small. A factorized $q$ fitted by reverse KL has variances $1/(\Sigma^{-1})_{ii}$, the conditional variances, which are smaller than the marginal variances $\Sigma_{ii}$.

4. Worked example #

(a) Units. $X$ is uniform on $[0, 0.5]$ m.

• In metres: $h = \log 0.5 = -0.693$ nats $= -1$ bit.
• In millimetres, $X$ is uniform on $[0, 500]$: $h = \log 500 = 6.215$ nats.
• The difference is $6.215 - (-0.693) = 6.908 = \log 1000$, as Theorem 2 says with $A = 1000$.

(b) A 2-D Gaussian and a change of units. $X \sim \mathcal{N}(0, \Sigma)$ with $\Sigma = \begin{pmatrix} 1 & 0.8 \\ 0.8 & 1 \end{pmatrix}$.

• $\det\Sigma = 1 \cdot 1 - 0.8^2 = 0.36$.
• $h(X) = \frac12\log\big((2\pi e)^2 \cdot 0.36\big) = \log(2\pi e) + \frac12\log 0.36 = 2.838 - 0.511 = 2.327$ nats.
• $Y = 10X$: $A = 10I$, $\det A = 100$, so $h(Y) = 2.327 + \log 100 = 2.327 + 4.605 = 6.932$ nats.
• A 1-D $\mathcal{N}(0, 0.2^2)$: $h = \frac12\log(2\pi e) + \log 0.2 = 1.419 - 1.609 = -0.190$ nats. Negative, because $0.2 < 0.242$.

(c) Gaussian KL in 2-D. $p_0 = \mathcal{N}(0, I)$ and $p_1 = \mathcal{N}\big((1, 0), \operatorname{diag}(2, 0.5)\big)$.

• $\Sigma_1^{-1} = \operatorname{diag}(0.5, 2)$, so the trace term is $\operatorname{tr}(\Sigma_1^{-1}I) = 0.5 + 2 = 2.5$.
• Mean term: $(1, 0)\,\Sigma_1^{-1}\,(1, 0)^\top = 0.5$.
• Log-det term: $\log\frac{\det\Sigma_1}{\det\Sigma_0} = \log\frac{1}{1} = 0$.
• $\mathrm{KL}(p_0\,\|\,p_1) = \frac12(2.5 + 0.5 - 2 + 0) = 0.5$ nats.
• The other direction: the trace term is $\operatorname{tr}(\Sigma_1) = 2.5$, the mean term is $1^2 \cdot 1 = 1$, the log-det term is $\log\frac{1}{1} = 0$, so $\mathrm{KL}(p_1\,\|\,p_0) = \frac12(2.5 + 1 - 2) = 0.75$ nats. The asymmetry comes from the mean term: the same shift is measured in units of $\Sigma_1$ in one direction and of $\Sigma_0$ in the other.

(d) Forward vs reverse on the two bumps $p = 0.25\,\mathcal{N}(-3, 1) + 0.75\,\mathcal{N}(3, 1)$, by numerical integration (section 5):

$q$	$\mathrm{KL}(p \Vert q)$	$\mathrm{KL}(q \Vert p)$
$\mathcal{N}(1.5, 2.78^2)$, the forward fit	0.465	0.935
$\mathcal{N}(3, 1)$	3.941	0.286
$\mathcal{N}(-3, 1)$	12.941	1.379

Each fit wins its own objective by a wide margin. Theorem 6 checks out: $\mathrm{KL}(p\,\|\,\mathcal{N}(3, 1)) = 0.465 + \mathrm{KL}(\mathcal{N}(1.5, 7.75)\,\|\,\mathcal{N}(3, 1))$, and the 1-D formula gives $\log\frac{1}{2.784} + \frac{7.75 + 2.25}{2} - \frac12 = -1.024 + 5 - 0.5 = 3.476$; $0.465 + 3.476 = 3.941$ ✓.

(e) Mean-field. $\Sigma = \begin{pmatrix} 1 & 0.9 \\ 0.9 & 1 \end{pmatrix}$, $\det\Sigma = 0.19$, $\Sigma^{-1} = \frac{1}{0.19}\begin{pmatrix} 1 & -0.9 \\ -0.9 & 1 \end{pmatrix}$.

• $(\Sigma^{-1})_{11} = 1/0.19 = 5.263$, so the reverse fit has $s_1^2 = s_2^2 = 0.19$.
• The forward fit has $s_1^2 = s_2^2 = \Sigma_{11} = 1$.
• The forward KL of the forward fit is $\frac12[2 - 2 + \log 1 - \log 0.19] = 0.830$ nats; the reverse KL of the reverse fit is $\frac12[2 - 2 + \log 0.19 - \log 0.19^2] = 0.830$ nats. (In this symmetric 2-D case the two numbers agree; in general they do not.)

5. In code #

SciPy has the entropies of standard distributions built in (.entropy(), in nats), and scipy.integrate.quad computes KL integrals in one dimension.

import numpy as np
from scipy.stats import norm, multivariate_normal, uniform, laplace, entropy
from scipy.integrate import quad

# (a) Differential entropy (nats): negative values and units
print(uniform(scale=0.5).entropy(), uniform(scale=500).entropy())   # -0.6931 6.2146
print(norm(scale=0.2).entropy())                                     # -0.1905

# (b) Quantize N(0, 0.2^2) into bins of width d: H(X_d) - log(1/d) -> h(X)
for d in [0.1, 0.01, 0.001]:
    P = np.diff(norm(scale=0.2).cdf(np.arange(-2, 2 + d / 2, d)))
    print(d, entropy(P), entropy(P) + np.log(d))
# 0.1 2.1224 -0.1802 | 0.01 4.4148 -0.1904 | 0.001 6.7173 -0.1905

# (c) A 2-D Gaussian and a change of units Y = 10 X
S = np.array([[1.0, 0.8], [0.8, 1.0]])
print(multivariate_normal(cov=S).entropy(), multivariate_normal(cov=100 * S).entropy())  # 2.3271 6.9322

# (d) Maximum entropy at unit variance: uniform < Laplace < Gaussian
print(uniform(scale=np.sqrt(12)).entropy(), laplace(scale=1 / np.sqrt(2)).entropy(), norm.entropy())
# 1.2425 1.3466 1.4189

# (e) KL between Gaussians: closed form vs numerical integration
def gauss_kl(m0, S0, m1, S1):
    d, S1i, dm = len(m0), np.linalg.inv(S1), m1 - m0
    return 0.5 * (np.trace(S1i @ S0) + dm @ S1i @ dm - d + np.log(np.linalg.det(S1) / np.linalg.det(S0)))
print(gauss_kl(np.zeros(1), np.eye(1), np.ones(1), 4 * np.eye(1)),
      quad(lambda x: norm.pdf(x) * (norm.logpdf(x) - norm.logpdf(x, 1, 2)), -20, 20)[0])   # 0.4431 0.4431

# (f) Forward vs reverse KL on p = 0.25 N(-3, 1) + 0.75 N(3, 1)
p = lambda x: 0.25 * norm.pdf(x, -3, 1) + 0.75 * norm.pdf(x, 3, 1)
for mu, s in [(1.5, np.sqrt(7.75)), (3.0, 1.0), (-3.0, 1.0)]:
    fwd = quad(lambda x: p(x) * np.log(p(x) / norm.pdf(x, mu, s)), -30, 30, limit=400)[0]
    rev = quad(lambda x: norm.pdf(x, mu, s) * (norm.logpdf(x, mu, s) - np.log(p(x))), -30, 30, limit=400)[0]
    print(mu, round(fwd, 3), round(rev, 3))
# 1.5 0.465 0.935 | 3.0 3.941 0.286 | -3.0 12.941 1.379

# (g) Mean-field fits: forward keeps Sigma_ii, reverse uses 1/(Sigma^-1)_ii
S = np.array([[1.0, 0.9], [0.9, 1.0]])
print(np.diag(S), 1 / np.diag(np.linalg.inv(S)))   # [1. 1.] [0.19 0.19]

Numerics: in the tails a density underflows to 0 and np.log(0) gives -inf. Work with log-densities (norm.logpdf, scipy.special.logsumexp for mixtures) whenever a KL integrand reaches far into the tails.

6. Where it shows up #

• VAEs. The encoder outputs $q_\phi(z \mid x) = \mathcal{N}(\mu, \operatorname{diag}(\sigma^2))$, and the KL to the prior $\mathcal{N}(0, I)$ is the closed form of section 3.6. The whole objective is a reverse-KL fit to the posterior (lesson 09).
• Variational inference in science. Mean-field VI for Bayesian neural networks, inverse problems and data assimilation is fast, but its variances are often too small (Theorem 7). Check the spread against MCMC or an ensemble before you trust an uncertainty band.
• Maximum likelihood and distillation. Fitting by log-loss and distilling a teacher on soft labels minimize forward KLs: the student must cover everything the teacher allows. The KL penalty of RLHF, $\mathrm{KL}(\pi_\theta\,\|\,\pi_{\text{ref}})$, is a reverse KL, estimated on samples of the policy.
• Normalizing flows. The training loss of a flow is $-\log p_X(g^{-1}(y)) - \log|\det J_{g^{-1}}(y)|$, the change of variables of section 3.3 applied to every data point.
• Bayesian experimental design. For a Gaussian posterior, the information gained by an experiment is $h(\text{prior}) - h(\text{posterior}) = \frac12\log\frac{\det\Sigma_{\text{prior}}}{\det\Sigma_{\text{post}}}$, a difference of entropies, so the units cancel. This is the D-optimal design criterion.
• Scientific data and units. A Gaussian fit to a forecast ensemble has entropy $\frac12\log\det(2\pi e\Sigma)$. Precipitation in mm and in m differ by $\log 1000$ per dimension. Report differences, KL divergences or mutual information instead.

7. Common mistakes #

• Treating $h$ like $H$. $h$ can be negative and changes with the units: $h(cX) = h(X) + \log|c|$. A discrete $H$ is $\ge 0$ and does not change under relabelling.
• Reading $h$ as the limit of $H$. $H(X_\Delta) \to \infty$ as $\Delta \to 0$. $h$ is the finite part after removing $\log(1/\Delta)$.
• "A negative log-likelihood below 0 is a bug." For densities it is not: a density can exceed 1. Only a discrete (or dequantized) likelihood bounds a code length (lesson 04).
• Swapping the directions. Forward $\mathrm{KL}(p\,\|\,q)$: average under the data, mode covering, maximum likelihood. Reverse $\mathrm{KL}(q\,\|\,p)$: average under the model, mode seeking, VI.
• Gaussian KL slips. Mixing $\sigma$ and $\sigma^2$, forgetting the $\frac12$ or the $-d$, or putting $\Sigma_0^{-1}$ instead of $\Sigma_1^{-1}$ in the trace.
• Trusting mean-field variances. A factorized reverse-KL fit gives conditional variances, which are too small when the target has correlations.

8. Check yourself #

1. Give the differential entropy of the uniform density on $[0, \frac18]$ in bits. $X \sim \mathcal{N}(0, \sigma^2)$ and $Y = 3X - 5$: what is $h(Y) - h(X)$?
   Answer

   $\log_2\frac18 = -3$ bits. $h(Y) - h(X) = \log 3 = 1.099$ nats; the shift $-5$ does not matter.
2. A normalizing flow on images scaled to $[0, 1)$ reports a log-density of $+8000$ nats per image. Is a positive log-density (negative NLL) a bug?
   Answer

   No. A density on $[0, 1)^D$ can be much larger than 1, so $\log p$ can be positive. To compare with a code length, convert to bits per dimension on the 8-bit scale ([lesson 04](/information-theory/04-cross-entropy-loss/)).
3. A VAE encoder outputs $\mu = (1, 0)$ and $\sigma = (0.5, 0.5)$. Compute the KL to $\mathcal{N}(0, I)$.
   Answer

   $\frac12\sum_j(\sigma_j^2 + \mu_j^2 - 1 - \log\sigma_j^2) = \frac12\big[(0.25 + 1 - 1 + 1.386) + (0.25 + 0 - 1 + 1.386)\big] = \frac12(1.636 + 0.636) = 1.136$ nats.
4. The target is $p = 0.5\,\mathcal{N}(-5, 1) + 0.5\,\mathcal{N}(5, 1)$. Give the forward-KL Gaussian fit, and the approximate value of the reverse KL at a reverse-KL local optimum.
   Answer

   Forward: mean 0 and variance $1 + 25 - 0 = 26$, so $q = \mathcal{N}(0, 26)$. Reverse: $q \approx \mathcal{N}(\pm 5, 1)$ with $\mathrm{KL}(q\,\|\,p) \approx -\log 0.5 = \log 2 = 0.693$ nats (numerically $0.6931$).
5. A posterior is $\mathcal{N}(0, \Sigma)$ with unit variances and correlation $0.6$. Mean-field VI fits $q = \mathcal{N}(0, \operatorname{diag}(s_1^2, s_2^2))$. What variances does it find?
   Answer

   $s_i^2 = 1/(\Sigma^{-1})_{ii} = 1 - 0.6^2 = 0.64$, a standard deviation of $0.8$ instead of 1.
6. Why does the Gaussian have the largest entropy among densities with a given covariance, in one sentence?
   Answer

   For any $p$ with the same mean and covariance, $h(g) - h(p) = \mathrm{KL}(p\,\|\,g) \ge 0$, because $\log g$ is quadratic and so its mean under $p$ equals its mean under $g$.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Define the differential entropy of a continuous random variable with density $p$.

   Show answer

2. What is the differential entropy of $\mathcal{N}(\mu, \sigma^2)$, and when is it negative?

   Show answer

3. What is the entropy of a $d$-dimensional Gaussian $\mathcal{N}(\mu, \Sigma)$?

   Show answer

4. How does $h$ change under $Y = AX + b$ with $A$ invertible?

   Show answer

5. Relate the entropy of $X$ quantized into bins of width $\Delta$ to $h(X)$.

   Show answer

6. Why does the KL divergence between densities not depend on the units?

   Show answer

7. Which density has the largest differential entropy for a given mean and covariance?

   Show answer

8. Give $\mathrm{KL}(\mathcal{N}(\mu_0, \sigma_0^2)\,\|\,\mathcal{N}(\mu_1, \sigma_1^2))$.

   Show answer

9. What Gaussian minimizes the forward KL $\mathrm{KL}(p\,\|\,q)$ to a target $p$, and why is it often too wide?

   Show answer

10. Why does variational inference minimize the reverse KL, and what is the side effect?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Entropy of a $d$-dimensional Gaussian? Why can it be negative?" $\frac12\log((2\pi e)^d\det\Sigma)$. A density can be larger than 1 when $\det\Sigma$ is small, so $-\log p$ can be negative on average.
• "What happens to differential entropy if you rescale the data?" It shifts by $\log|\det A|$; so units matter. KL and mutual information do not change.
• "Derive the KL between two Gaussians." Use $\mathbb{E}[Z^\top MZ] = \operatorname{tr}(MC) + m^\top Mm$ for both log-densities (Theorem 5). Know the 1-D formula and the VAE term $\frac12\sum_j(\sigma_j^2 + \mu_j^2 - 1 - \log\sigma_j^2)$.
• "Forward vs reverse KL?" Forward: mode covering, needs data samples, equals maximum likelihood, moment matching for Gaussians. Reverse: mode seeking, needs the target only up to a constant, used by VI; underestimates variance.
• "Why do VAEs or mean-field VI give overconfident posteriors?" Reverse KL avoids regions where the posterior is small; a factorized $q$ on a correlated posterior gets the conditional variances.

10. Practice #

The notebook for this lesson runs in Colab. It has five exercises, each with a check cell and a folded solution:

1. gaussian_entropy(cov) from the log-determinant, checked against scipy.stats.multivariate_normal(...).entropy() and against $h(AX) = h(X) + \log|\det A|$.
2. The quantization experiment: quantized_entropy(sigma, delta) and the limit $H(X_\Delta) - \log(1/\Delta) \to h$.
3. gaussian_kl(mu0, cov0, mu1, cov1) from Theorem 5, checked against a Monte Carlo estimate.
4. Forward fit (moment matching) and reverse fit (a local optimizer from two starts) of one Gaussian to the two-bump target.
5. Mean-field reverse-KL fit of a correlated Gaussian by coordinate ascent; compare its variances with $\Sigma_{ii}$.

11. Watch and read #

KL Divergence: Forward vs Reverse, Mass-Covering vs Mode-Seekingbrevfeed · 4 min

A short animation of the two fits of section 2.2.

Fantastic KL Divergence and How to (Actually) Compute ItJia-Bin Huang · 12 min

Forward and reverse KL, and how to estimate KL from samples when there is no closed form.

Gaussian Models and the Perils of Differential EntropyDeep Foundations · 15 min

Why differential entropy can be negative and depends on the coordinates, with Gaussian examples.

Chapter 10 Differential Entropy - Section 10.6 Maximum Differential Entropy DistributionsInformation Theory by Prof. Raymond W. Yeung, CUHK · 24 min

A careful lecture proof of the maximum-entropy results of section 3.5.

• Kevin Murphy, Probabilistic Machine Learning: An Introduction, free PDF: §6.1.6 (differential entropy and the Gaussian), §6.2.3 (KL between two Gaussians) and §6.2.6 (forward vs reverse KL).
• Christopher Bishop, Pattern Recognition and Machine Learning, free PDF: §1.6 (differential entropy, maximum entropy) and §10.1.2 (Figures 10.2 and 10.3: factorized approximations and the two KL directions).
• Tim Vieira, "KL-divergence as an objective function": which direction needs samples and which needs the normalizer.
• David MacKay, Information Theory, Inference, and Learning Algorithms, free PDF: chapter 33 (variational free-energy minimization, the reverse-KL view of approximate inference).