Jensen's inequality and the KL divergence

Before this lesson: lesson 01 (surprisal and entropy) and lesson 02 (conditional entropy). Convex functions from optimization lesson 02; the expectation of a function of a random variable from the probability track. Time: about 60 minutes, plus the notebook.

---

Why this matters #

KL divergence is inside many training objectives. The cross-entropy loss of every classifier and language model is the entropy of the data plus a KL term (lesson 04). RLHF (reinforcement learning from human feedback) adds a penalty $\beta\,\mathrm{KL}(\pi_\theta\,\|\,\pi_{\text{ref}})$ so that the tuned language model stays close to the reference model. Knowledge distillation trains a small student to match a large teacher by minimizing a KL. The VAE loss has a KL term that pulls the encoder towards the prior (lesson 09). TRPO limits each policy update to a small KL ball (a trust region). Monitoring systems detect data drift by comparing the feature distributions of two time windows with KL or a symmetric version of it.

Jensen's inequality is the tool behind the proofs. It shows that KL is never negative, that the uniform distribution has the largest entropy, and that the ELBO of variational inference is a lower bound on the log-likelihood.

In this track $\log$ is the natural logarithm and gives nats; $\log_2$ gives bits, and $1\text{ nat} = 1/\ln 2 \approx 1.443$ bits. NumPy, SciPy and PyTorch use nats.

1. Predict first #

2. The picture #

Jensen with two points: the chord #

A function $f$ is convex if every chord lies on or above its graph. Take two points $a$ and $b$ and a weight $\lambda \in [0, 1]$. Let $X$ be a random variable that equals $a$ with probability $\lambda$ and $b$ with probability $1 - \lambda$. Then

• $\mathbb{E}[X] = \lambda a + (1 - \lambda) b$ is a point between $a$ and $b$;
• $\mathbb{E}[f(X)] = \lambda f(a) + (1 - \lambda) f(b)$ is the height of the chord above that point;
• $f(\mathbb{E}[X])$ is the height of the graph above the same point.

The chord is above the graph, so $f(\mathbb{E}[X]) \le \mathbb{E}[f(X)]$. For two points, this inequality is exactly the definition of convexity.

Many points: the centre of mass #

Now let $X$ take values $x_1, \dots, x_n$ with probabilities $p_1, \dots, p_n$. Put a mass $p_i$ on the graph at the point $(x_i, f(x_i))$. The centre of mass of these masses is $\big(\sum_i p_i x_i,\ \sum_i p_i f(x_i)\big) = \big(\mathbb{E}[X],\ \mathbb{E}[f(X)]\big)$. It lies inside the polygon spanned by the points. For a convex $f$ this polygon lies on or above the graph, so the centre of mass does too: $\mathbb{E}[f(X)] \ge f(\mathbb{E}[X])$. This is Jensen's inequality.

Interactive figure: loading…

Concave functions turn the inequality around #

A function is concave if $-f$ is convex: every chord lies on or below the graph. The logarithm is concave, so

Example. $X$ is 1 or 4, each with probability $\frac12$. Then $\mathbb{E}[X] = 2.5$ and $\log \mathbb{E}[X] = \log 2.5 = 0.916$, but $\mathbb{E}[\log X] = \frac12(\log 1 + \log 4) = \log 2 = 0.693$. Take the exponential of both sides: the geometric mean $\sqrt{1 \cdot 4} = 2$ is at most the arithmetic mean $2.5$. This is the AM–GM inequality, and it is Jensen for $\log$.

KL divergence: the extra surprise of a wrong model #

Lesson 01 measured the surprisal of an outcome $x$ as $-\log_2 p(x)$ bits. Now the data come from $p$, but you predict with a model $q$. Each outcome surprises you by $-\log_2 q(x)$.

A fair coin, $p = (0.5, 0.5)$, and a model that believes $q = (0.8, 0.2)$:

outcome	$p(x)$	$q(x)$	surprise under $q$: $-\log_2 q(x)$	surprise under $p$: $-\log_2 p(x)$
heads	0.5	0.8	0.322 bits	1 bit
tails	0.5	0.2	2.322 bits	1 bit

The coin is fair, so each row happens half of the time.

1. Your average surprise with the model is $\frac12(0.322 + 2.322) = 1.322$ bits. This average is the cross-entropy $H(p, q)$ (lesson 04 studies it).
2. The best possible average is the entropy $H(p) = 1$ bit. You reach it only with the right model $q = p$.
3. The difference, $1.322 - 1 = 0.322$ bits, is the Kullback–Leibler (KL) divergence $\mathrm{KL}(p\,\|\,q)$: the price of the wrong belief, per coin flip.

Look at the rows separately. On heads, the model is less surprised than the truth would be ($0.322 < 1$). On tails it is more surprised ($2.322 > 1$). The losses on tails are larger than the gains on heads, so the average is positive. This is the content of Gibbs' inequality below.

Interactive figure: loading…

3. Definitions and main results #

Convex and concave functions #

A function $f$ on an interval $I$ is convex if

for all $a, b \in I$ and all $\lambda \in [0, 1]$.

It is strictly convex if the inequality is strict whenever $a \ne b$ and $0 < \lambda < 1$. It is (strictly) concave if $-f$ is (strictly) convex. A twice differentiable $f$ is convex when $f''(x) \ge 0$ on $I$, and strictly convex when $f''(x) > 0$ (optimization lesson 02).

• Strictly convex: $x^2$, $e^x$, $1/x$ for $x > 0$, $t \log t$ for $t \ge 0$.
• Strictly concave: $\log x$ and $\sqrt{x}$ for $x > 0$.
• Convex but not strictly: $|x|$, and every linear function $ax + b$ (it is also concave).

Jensen's inequality #

Theorem 1 (Jensen). Let $f$ be convex on an interval $I$, and let $X$ take values in $I$ with $\mathbb{E}[X]$ and $\mathbb{E}[f(X)]$ finite. Then

If $f$ is strictly convex, equality holds if and only if $X$ is constant (with probability 1). For a concave $f$ the inequality is reversed: $f(\mathbb{E}[X]) \ge \mathbb{E}[f(X)]$.

Why this is true. A convex function lies above each of its tangent lines. A line is easy to average: the average of a line is the line at the average. So the average of $f$ is above the average of the line, which is the value of $f$ at the mean.

Proof.

1. Let $\mu = \mathbb{E}[X]$. If $\mu$ is an endpoint of $I$, say the left one, then $X - \mu \ge 0$ and $\mathbb{E}[X - \mu] = 0$. A non-negative variable with mean 0 is 0 with probability 1, so $X = \mu$ and both sides equal $f(\mu)$. From now on, $\mu$ is an interior point of $I$.
2. A convex $f$ has a supporting line at every interior point: a slope $c$ with $f(x) \ge f(\mu) + c\,(x - \mu)$ for all $x \in I$. If $f$ is differentiable, $c = f'(\mu)$ and the line is the tangent line.
3. Put $x = X$ and take the expectation of both sides. Expectation is linear and keeps inequalities, so $\mathbb{E}[f(X)] \ge f(\mu) + c\,(\mathbb{E}[X] - \mu) = f(\mu)$.
4. Equality case. The difference $D = f(X) - f(\mu) - c\,(X - \mu)$ is $\ge 0$. Equality in step 3 means $\mathbb{E}[D] = 0$, so $D = 0$ with probability 1: $X$ only takes values where the graph touches the supporting line.
5. A strictly convex $f$ touches a line below it at one point only. (If it touched at $\mu$ and at $x' \ne \mu$, then at the midpoint of $\mu$ and $x'$ the graph would lie strictly below the chord, which is a piece of the line. That contradicts $f \ge$ line.) So $X = \mu$ with probability 1.
6. For a concave $f$, apply steps 1–5 to the convex function $-f$. $\blacksquare$

For a convex $f$ that is not strictly convex, equality can hold with a random $X$: when $f$ is linear on the values of $X$. For example $f(x) = |x|$ and $X \in \{1, 3\}$: $|\mathbb{E}[X]| = 2 = \mathbb{E}[|X|]$.

How large is the gap for the logarithm? #

The Jensen gap for $\log$ is $\log \mathbb{E}[X] - \mathbb{E}[\log X] \ge 0$. Let $\mu = \mathbb{E}[X]$ and $\sigma^2 = \operatorname{Var}(X)$. When $X$ stays close to $\mu$,

Derivation.

1. Second-order Taylor expansion of $\log$ at $\mu$: $\log x \approx \log \mu + \frac{x - \mu}{\mu} - \frac{(x - \mu)^2}{2\mu^2}$.
2. Put $x = X$ and take expectations. $\mathbb{E}[X - \mu] = 0$ and $\mathbb{E}[(X - \mu)^2] = \sigma^2$, so $\mathbb{E}[\log X] \approx \log \mu - \frac{\sigma^2}{2\mu^2}$.
3. Subtract from $\log \mathbb{E}[X] = \log \mu$.

For $X \in \{0.9, 1.1\}$ with equal probabilities, the exact gap is $0.00503$ and the approximation gives $0.005$. For $X \in \{1, 4\}$ (a large spread) the exact gap is $0.223$ and the approximation gives $0.18$: the Taylor expansion is only good for a small spread. The lesson to remember: the gap grows with the relative spread $\sigma/\mu$. Lesson 09 uses this to explain why the IWAE bound, which averages several samples inside the log, is tighter than the ELBO.

Preview: why the ELBO is a lower bound

In a latent-variable model $p(x, z)$, the evidence $p(x) = \sum_z p(x, z)$ is often too expensive to compute. Choose any distribution $q(z)$ that is positive wherever $p(x, z) > 0$, and define the weight $w(z) = p(x, z)/q(z)$. Then $p(x) = \sum_z q(z)\,w(z) = \mathbb{E}_{z \sim q}[w(z)]$. Jensen for the concave $\log$ gives

$$\log p(x) = \log \mathbb{E}_q[w] \ \ge\ \mathbb{E}_q[\log w] = \mathbb{E}_q\Big[\log \frac{p(x, z)}{q(z)}\Big].$$

The right side is the evidence lower bound (ELBO). Lesson 09 shows that the gap is exactly a KL divergence, $\mathrm{KL}(q(z)\,\|\,p(z \mid x))$.

The KL divergence #

Let $p$ and $q$ be distributions on the same finite set. The Kullback–Leibler divergence (also called relative entropy) of $q$ from $p$ is

Read it as: the average, under the truth $p$, of the extra surprise $-\log q(x) - (-\log p(x))$. The first argument is the distribution we average under. In machine learning it is usually the data distribution, and the second argument is the model.

Conventions for zeros (they are the limits of $t \log t$ and $t \log(t/s)$):

• $0 \log \frac{0}{q} = 0$ for every $q \ge 0$: outcomes that never happen cost nothing.
• $p \log \frac{p}{0} = +\infty$ for $p > 0$: an outcome that happens but has model probability 0 makes the KL infinite.

So $\mathrm{KL}(p\,\|\,q) < \infty$ exactly when $q(x) > 0$ for every $x$ with $p(x) > 0$ (the support of $p$ lies inside the support of $q$). For densities, replace the sum by an integral; the results below stay the same (lesson 06).

Gibbs' inequality: KL is never negative #

Theorem 2 (Gibbs' inequality). $\mathrm{KL}(p\,\|\,q) \ge 0$, with equality if and only if $p = q$.

Proof.

1. Let $S = \{x : p(x) > 0\}$ be the support of $p$. If $q(x) = 0$ for some $x \in S$, the KL is $+\infty$ and there is nothing to prove. Otherwise, the terms outside $S$ are 0, and $-\mathrm{KL}(p\,\|\,q) = \sum_{x \in S} p(x) \log \frac{q(x)}{p(x)}$.
2. This is $\mathbb{E}\big[\log \frac{q(X)}{p(X)}\big]$ with $X \sim p$. Jensen for the concave $\log$ moves the log outside: $-\mathrm{KL}(p\,\|\,q) \le \log \sum_{x \in S} p(x) \frac{q(x)}{p(x)}$, and the right side simplifies to $\log \sum_{x \in S} q(x)$.
3. The probabilities $q(x)$ over the part $S$ add up to at most 1, so the right side is $\le \log 1 = 0$. Hence $\mathrm{KL}(p\,\|\,q) \ge 0$.
4. Equality needs both steps to be tight. Step 2 is tight only if $q(x)/p(x) = c$ is the same number for all $x \in S$ ($\log$ is strictly concave). Step 3 is tight only if $\sum_{x \in S} q(x) = 1$. Then $1 = \sum_{S} q = c \sum_{S} p = c$, so $q = p$ on $S$, and $q = 0$ outside $S$. That is, $q = p$. $\blacksquare$

Second proof: the log-sum inequality (and the convexity of KL)

Theorem 3 (log-sum inequality). For $a_i \ge 0$ and $b_i > 0$, with $A = \sum_i a_i$ and $B = \sum_i b_i$,

$$\sum_i a_i \log \frac{a_i}{b_i} \ \ge\ A \log \frac{A}{B},$$

with equality if and only if $a_i / b_i$ is the same for all $i$.

Proof.

1. Let $g(t) = t \log t$ for $t \ge 0$. Then $g''(t) = 1/t > 0$, so $g$ is strictly convex.
2. Define weights $w_i = b_i / B$ (positive, sum 1) and ratios $t_i = a_i / b_i$.
3. Rewrite the left side: $\sum_i a_i \log \frac{a_i}{b_i} = \sum_i b_i\, t_i \log t_i = B \sum_i w_i\, g(t_i)$.
4. Jensen with the weights $w$: $\sum_i w_i\, g(t_i) \ge g\big(\sum_i w_i t_i\big) = g(A/B)$, because $\sum_i w_i t_i = \sum_i a_i / B = A / B$.
5. Multiply by $B$: the left side is $\ge B \cdot \frac{A}{B} \log \frac{A}{B} = A \log \frac{A}{B}$.
6. Equality holds when Jensen is tight for the strictly convex $g$: all $t_i$ are equal. $\blacksquare$

Gibbs from log-sum. Put $a = p$ and $b = q$ (restrict to the $x$ with $q(x) > 0$). Then $A = B = 1$ and the right side is $1 \cdot \log 1 = 0$.

KL is jointly convex. For two pairs $(p_1, q_1)$, $(p_2, q_2)$ and $\lambda \in [0, 1]$,

$$\begin{aligned} &\mathrm{KL}\big(\lambda p_1 + (1 - \lambda) p_2 \,\big\|\, \lambda q_1 + (1 - \lambda) q_2\big) \\ &\qquad \le \lambda\, \mathrm{KL}(p_1\,\|\,q_1) + (1 - \lambda)\, \mathrm{KL}(p_2\,\|\,q_2). \end{aligned}$$

Proof: fix one outcome $x$ and apply the log-sum inequality to the two terms $a_1 = \lambda p_1(x)$, $a_2 = (1 - \lambda) p_2(x)$, $b_1 = \lambda q_1(x)$, $b_2 = (1 - \lambda) q_2(x)$. The $\lambda$ inside the logs cancels. Then add over $x$. Convexity is why minimizing a KL over a convex set of distributions has no bad local minima.

Two consequences for entropy #

Gibbs' inequality proves two facts that lessons 01 and 02 stated.

Corollary 1 (maximum entropy). For $p$ on $K$ values, $H(p) \le \log K$, with equality only for the uniform distribution $u = (\frac1K, \dots, \frac1K)$.

Proof. Put $u(x) = 1/K$ into the definition:

Gibbs gives $\log K - H(p) \ge 0$, with equality only if $p = u$. $\blacksquare$

So the entropy gap to the maximum is a KL divergence to the uniform distribution. For $p = (\frac12, \frac14, \frac18, \frac18)$: $H(p) = 1.75$ bits, $\log_2 4 = 2$ bits, and $\mathrm{KL}(p\,\|\,u) = 0.25$ bits.

Corollary 2 (conditioning reduces entropy on average). $H(Y \mid X) \le H(Y)$, with equality if and only if $X$ and $Y$ are independent.

Proof.

1. Write both entropies as averages over the joint distribution $p(x, y)$: $H(Y) = \sum_{x, y} p(x, y) \log \frac{1}{p(y)}$ and $H(Y \mid X) = \sum_{x, y} p(x, y) \log \frac{1}{p(y \mid x)}$.
2. Subtract: $H(Y) - H(Y \mid X) = \sum_{x, y} p(x, y) \log \frac{p(y \mid x)}{p(y)}$.
3. Multiply the top and bottom of the fraction by $p(x)$: $\frac{p(y \mid x)}{p(y)} = \frac{p(x, y)}{p(x)\,p(y)}$.
4. So $H(Y) - H(Y \mid X) = \mathrm{KL}\big(p(x, y)\,\|\,p(x)\,p(y)\big) \ge 0$ by Gibbs. Equality holds only if $p(x, y) = p(x)\,p(y)$ for all $x, y$: independence. $\blacksquare$

The difference $H(Y) - H(Y \mid X)$ is the mutual information $I(X;Y)$ (lesson 07). Step 4 shows that it is the KL divergence from the joint distribution to the product of the marginals.

KL is not a metric #

A metric (distance) $d$ must be symmetric, $d(p, q) = d(q, p)$, and must satisfy the triangle inequality, $d(p, r) \le d(p, q) + d(q, r)$. KL fails both.

• Not symmetric. For the coin, $\mathrm{KL}(p\,\|\,q) = 0.223$ nats ($0.322$ bits) but $\mathrm{KL}(q\,\|\,p) = 0.8 \log 1.6 + 0.2 \log 0.4 = 0.376 - 0.183 = 0.193$ nats ($0.278$ bits). The two arguments play different roles: the first one decides where we average.
• No triangle inequality. Write $\mathrm{Ber}(a)$ for the coin with $P(\text{heads}) = a$. Then $\mathrm{KL}(\mathrm{Ber}(0.9)\,\|\,\mathrm{Ber}(0.1)) = 1.758$ nats, but the two steps through the fair coin give $\mathrm{KL}(\mathrm{Ber}(0.9)\,\|\,\mathrm{Ber}(0.5)) + \mathrm{KL}(\mathrm{Ber}(0.5)\,\|\,\mathrm{Ber}(0.1)) = 0.368 + 0.511 = 0.879$ nats. The direct route is twice as long as the detour.

This is why the name is "divergence", not "distance". For two close distributions, KL behaves like half a squared distance (see the folded block below), and squared distances also break the triangle inequality: $2^2 > 1^2 + 1^2$.

Bernoulli KL. For two coins, $\mathrm{KL}(\mathrm{Ber}(a)\,\|\,\mathrm{Ber}(b))$ has two terms, one for heads and one for tails:

Interactive figure: loading…

Near $p$, KL is half a squared distance

Let $q = p + \delta$ with a small change $\delta$ ($\sum_x \delta(x) = 0$, and $p(x) > 0$). A second-order Taylor expansion of $\log(1 + t) \approx t - t^2/2$ gives

$$\begin{aligned} \mathrm{KL}(p\,\|\,p + \delta) &= -\sum_x p(x) \log\Big(1 + \frac{\delta(x)}{p(x)}\Big) \\ &\approx -\sum_x \delta(x) + \frac12 \sum_x \frac{\delta(x)^2}{p(x)} = \frac12 \sum_x \frac{\delta(x)^2}{p(x)}. \end{aligned}$$

This is half a weighted squared distance (the weights $1/p(x)$ are the Fisher information of the categorical distribution). The reverse KL has the same second-order term, so for close distributions the asymmetry disappears; it shows up only at third order. For $p = (0.5, 0.5)$ and $q = (0.52, 0.48)$: both KLs are $0.000800$ nats, and the formula gives $\frac12(0.0008 + 0.0008) = 0.0008$. Natural-gradient methods and TRPO use this local metric.

Answers to section 1. (1) No. $\mathrm{KL}(p\,\|\,q) = 0.322$ bits for the fair coin modelled as $(0.8, 0.2)$, but $\mathrm{KL}(q\,\|\,p) = 0.278$ bits for the reverse situation. (2) Infinite: the term $0.01 \log \frac{0.01}{0}$ is $+\infty$. (3) Yes. A term is negative wherever $q(x) > p(x)$; only the total is $\ge 0$.

4. Worked example #

(a) Jensen for the log. $X$ is 1 or 4 with probability $\frac12$ each.

• $\mathbb{E}[X] = \frac12 \cdot 1 + \frac12 \cdot 4 = 2.5$, so $\log \mathbb{E}[X] = \log 2.5 = 0.916$.
• $\mathbb{E}[\log X] = \frac12 \log 1 + \frac12 \log 4$ $= 0 + \log 2 = 0.693$.
• Gap: $0.916 - 0.693 = 0.223 = \log \frac{2.5}{2} = \log 1.25$. It is positive, as Jensen says.
• Variance: $\mathbb{E}[X^2] - 2.5^2 = \frac12(1 + 16) - 6.25 = 2.25$. The Taylor estimate $\sigma^2/(2\mu^2) = 2.25/12.5 = 0.18$ is in the right range, but not exact, because the spread is large.

(b) KL on three outcomes, term by term. $p = (0.6, 0.3, 0.1)$ and $q = (0.3, 0.4, 0.3)$, in bits.

$x$	$p(x)$	$q(x)$	$p(x)/q(x)$	$p(x)\log_2\frac{p(x)}{q(x)}$
1	0.6	0.3	2	$0.6 \cdot 1 = 0.600$
2	0.3	0.4	0.75	$0.3 \cdot (-0.415) = -0.125$
3	0.1	0.3	0.333	$0.1 \cdot (-1.585) = -0.158$

$\mathrm{KL}(p\,\|\,q) = 0.600 - 0.125 - 0.158 = 0.317$ bits. Two of the three terms are negative.

The reverse direction averages under $q$: $\mathrm{KL}(q\,\|\,p) = 0.3 \log_2 \frac{0.3}{0.6} + 0.4 \log_2 \frac{0.4}{0.3} + 0.3 \log_2 \frac{0.3}{0.1} = -0.300 + 0.166 + 0.475 = 0.342$ bits. The largest term now comes from outcome 3, where $q$ puts three times the mass of $p$. In nats, multiply by $\ln 2$: $0.220$ and $0.237$ nats.

(c) Distance to the uniform distribution. $p = (\frac12, \frac14, \frac18, \frac18)$ on $K = 4$ values. Surprisals: 1, 2, 3, 3 bits, so $H(p) = \frac12 \cdot 1 + \frac14 \cdot 2 + \frac18 \cdot 3 + \frac18 \cdot 3 = 1.75$ bits. Then $\mathrm{KL}(p\,\|\,u) = \log_2 4 - 1.75 = 0.25$ bits. Direct check: $\frac12 \log_2 \frac{1/2}{1/4} + \frac14 \log_2 1 + 2 \cdot \frac18 \log_2 \frac{1/8}{1/4} = 0.5 + 0 - 0.25 = 0.25$ bits.

(d) The triangle inequality fails. In nats,

• $\mathrm{KL}(\mathrm{Ber}(0.9)\,\|\,\mathrm{Ber}(0.5))$ $= 0.9 \log 1.8 + 0.1 \log 0.2$ $= 0.529 - 0.161 = 0.368$;
• $\mathrm{KL}(\mathrm{Ber}(0.5)\,\|\,\mathrm{Ber}(0.1))$ $= 0.5 \log 5 + 0.5 \log \frac{5}{9}$ $= 0.805 - 0.294 = 0.511$;
• $\mathrm{KL}(\mathrm{Ber}(0.9)\,\|\,\mathrm{Ber}(0.1))$ $= 0.9 \log 9 + 0.1 \log \frac19$ $= 0.8 \log 9 = 1.758$, which is larger than $0.368 + 0.511$.

5. In code #

SciPy has the KL divergence built in: scipy.stats.entropy(p, q) is $\mathrm{KL}(p\,\|\,q)$, and scipy.special.rel_entr(p, q) returns the single terms $p \log(p/q)$ with the zero conventions.

import numpy as np
from scipy.stats import entropy
from scipy.special import rel_entr

# (a) Jensen for log: E[log X] <= log E[X]
x, w = np.array([1.0, 4.0]), np.array([0.5, 0.5])
print(np.log(w @ x), w @ np.log(x))          # 0.9163 0.6931

# (b) KL term by term, in bits, and the reverse
p, q = np.array([0.6, 0.3, 0.1]), np.array([0.3, 0.4, 0.3])
print(rel_entr(p, q) / np.log(2))            # [ 0.6    -0.1245 -0.1585]
print(entropy(p, q, base=2), entropy(q, p, base=2))   # 0.3170 0.3415

# The coin: not symmetric
print(entropy([0.5, 0.5], [0.8, 0.2]), entropy([0.8, 0.2], [0.5, 0.5]))   # 0.2231 0.1927 nats

# (c) KL to the uniform distribution = log K - H(p)
p4 = np.array([1/2, 1/4, 1/8, 1/8])
print(entropy(p4, np.full(4, 1/4), base=2), 2 - entropy(p4, base=2))   # 0.25 0.25

# Zeros: q = 0 where p > 0 gives inf; p = 0 terms give 0
print(entropy([0.99, 0.01], [1.0, 0.0]), entropy([1.0, 0.0], [0.99, 0.01]))   # inf 0.01005

entropy(p, q) normalizes both inputs so that they sum to 1. Pass probabilities, not counts with different totals, if you want to see a mistake instead of a silent fix.

6. Where it shows up #

• RLHF and fine-tuning. PPO-based RLHF maximizes reward minus $\beta\,\mathrm{KL}(\pi_\theta\,\|\,\pi_{\text{ref}})$, estimated on samples from the tuned policy. Without this term the policy drifts into text that fools the reward model. DPO is derived from the same KL-regularized objective.
• Knowledge distillation. A student $q$ is trained on $\mathrm{KL}(p_{\text{teacher}}\,\|\,q)$, the KL from the teacher's softened output distribution (lesson 04).
• VAEs and variational inference. The VAE loss contains $\mathrm{KL}(q_\phi(z \mid x)\,\|\,p(z))$, and the gap of the ELBO is a KL (lesson 09). Which direction of KL you minimize changes the fit (lesson 06).
• Trust regions. TRPO constrains $\mathbb{E}_s\big[\mathrm{KL}(\pi_{\text{old}}(\cdot \mid s)\,\|\,\pi_\theta(\cdot \mid s))\big] \le \delta$. Near the old policy, KL is half a squared distance with the Fisher information as the metric.
• Data drift and monitoring. The population stability index (PSI) used in credit scoring and model monitoring is $\sum_x (p(x) - q(x)) \log \frac{p(x)}{q(x)}$ $= \mathrm{KL}(p\,\|\,q) + \mathrm{KL}(q\,\|\,p)$, a symmetric version of KL between the binned feature distributions of two time windows.
• ML for science. Model selection with AIC estimates the expected KL from the true data distribution to a fitted model. In climate and weather work, KL compares a forecast distribution with an observed or reference one; it needs the support condition, so forecasts must never give probability 0 to an event that can occur.

7. Common mistakes #

• Wrong direction of Jensen for the log. $\mathbb{E}[\log X] \le \log \mathbb{E}[X]$, because $\log$ is concave. If unsure, draw the chord, or check with $X \in \{1, 4\}$.
• "Every term of KL is non-negative." Single terms $p(x) \log \frac{p(x)}{q(x)}$ are negative wherever $q(x) > p(x)$. Only the sum is $\ge 0$.
• Swapping the arguments. $\mathrm{KL}(p\,\|\,q)$ averages under $p$. In general $\mathrm{KL}(p\,\|\,q) \ne \mathrm{KL}(q\,\|\,p)$, so always say which one you mean.
• Calling KL a distance. It is not symmetric and it breaks the triangle inequality. If you need a true metric, use the total variation distance, the Hellinger distance or the square root of the Jensen–Shannon divergence (lesson 08).
• Forgetting the zeros. A model probability of exactly 0 for a possible outcome gives an infinite KL (and an infinite loss). In code, p * np.log(p / q) gives nan for $p = 0$; use scipy.special.rel_entr, which applies $0 \log \frac{0}{q} = 0$.
• Wrong equality condition in Jensen. For a strictly convex $f$, equality holds only if $X$ is constant. For a convex $f$ that is linear on the range of $X$ (such as $|x|$ on $x > 0$), equality can hold with a random $X$.

8. Check yourself #

1. Use Jensen to show $\mathbb{E}[1/X] \ge 1/\mathbb{E}[X]$ for a positive random variable $X$. Check it for $X$ uniform on $\{1, 3\}$.
   Answer

   $f(x) = 1/x$ has $f''(x) = 2/x^3 > 0$ on $x > 0$, so it is strictly convex and Jensen gives $1/\mathbb{E}[X] \le \mathbb{E}[1/X]$, with equality only for a constant $X$. Example: $\mathbb{E}[1/X] = \frac12(1 + \frac13) = \frac23$ and $1/\mathbb{E}[X] = \frac12$; indeed $\frac23 \ge \frac12$.
2. Compute $\mathrm{KL}(\mathrm{Ber}(0.5)\,\|\,\mathrm{Ber}(0.9))$ and $\mathrm{KL}(\mathrm{Ber}(0.9)\,\|\,\mathrm{Ber}(0.5))$ in nats.
   Answer

   $0.5 \log \frac{0.5}{0.9} + 0.5 \log \frac{0.5}{0.1} = \frac12 \log \frac{25}{9} = \log \frac53 = 0.511$ nats. The other way: $0.9 \log 1.8 + 0.1 \log 0.2 = 0.529 - 0.161 = 0.368$ nats. Not equal.
3. In the proof of Gibbs' inequality, why do we sum $q(x)$ only over the support $S$ of $p$, and why is that sum at most 1?
   Answer

   Terms with $p(x) = 0$ contribute 0 by the convention $0 \log \frac{0}{q} = 0$, so the KL is a sum over $S$ only, and Jensen is applied to $X \sim p$, which lives on $S$. $q$ may put some mass outside $S$, so $\sum_{x \in S} q(x) \le \sum_x q(x) = 1$.
4. A model predicts $q = (0.5, 0.5, 0)$ for data with $p = (0.5, 0.25, 0.25)$. What is $\mathrm{KL}(p\,\|\,q)$? What is $\mathrm{KL}(q\,\|\,p)$ in bits?
   Answer

   $\mathrm{KL}(p\,\|\,q) = +\infty$, because outcome 3 has $p = 0.25 > 0$ and $q = 0$. The reverse is finite: $0.5 \log_2 \frac{0.5}{0.5} + 0.5 \log_2 \frac{0.5}{0.25} + 0 = 0.5$ bits.
5. Show that $H(X, Y) \le H(X) + H(Y)$, using a result of this lesson.
   Answer

   By the chain rule of lesson 02, $H(X, Y) = H(X) + H(Y \mid X)$. Corollary 2 gives $H(Y \mid X) \le H(Y)$. So $H(X, Y) \le H(X) + H(Y)$, with equality only for independent $X$ and $Y$. Equivalently, $H(X) + H(Y) - H(X, Y) = \mathrm{KL}(p(x, y)\,\|\,p(x)p(y)) \ge 0$.
6. Two distributions are very close: $p = (0.5, 0.5)$ and $q = (0.52, 0.48)$. Estimate $\mathrm{KL}(p\,\|\,q)$ without a calculator.
   Answer

   Use $\mathrm{KL} \approx \frac12 \sum_x \delta(x)^2 / p(x)$ with $\delta = (0.02, -0.02)$: $\frac12 (0.0004/0.5 + 0.0004/0.5) = 0.0008$ nats. The exact value is $0.000800$ nats.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. State Jensen's inequality for a convex function $f$, with the equality condition.

   Show answer

2. Which way does Jensen's inequality go for $\log$, and why?

   Show answer

3. About how large is the Jensen gap $\log \mathbb{E}[X] - \mathbb{E}[\log X]$ when $X$ has mean $\mu$ and small variance $\sigma^2$?

   Show answer

4. Define the KL divergence $\mathrm{KL}(p\,\|\,q)$ of two discrete distributions.

   Show answer

5. State Gibbs' inequality.

   Show answer

6. When is $\mathrm{KL}(p\,\|\,q)$ infinite?

   Show answer

7. What is $\mathrm{KL}(p\,\|\,u)$ for the uniform distribution $u$ on $K$ values?

   Show answer

8. Write the KL divergence between two coins, $\mathrm{KL}(\mathrm{Ber}(a)\,\|\,\mathrm{Ber}(b))$.

   Show answer

9. Compute $\mathrm{KL}(p\,\|\,q)$ in bits for a fair coin $p$ and the model $q = (0.8, 0.2)$.

   Show answer

10. Why is the KL divergence not a metric?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Prove that the KL divergence is non-negative." Write $-\mathrm{KL}(p\,\|\,q) = \mathbb{E}_p[\log(q/p)]$, apply Jensen for the concave log: $\le \log \mathbb{E}_p[q/p] = \log \sum_{x: p(x) > 0} q(x) \le 0$. Equality only for $p = q$.
• "Is KL a distance?" No: not symmetric, no triangle inequality. Give the coin numbers. Near $p$ it behaves like half a squared distance with the Fisher metric.
• "What happens if the model assigns probability 0 to a real outcome?" The KL (and the cross-entropy loss) is infinite. This is why models use a softmax, which is never exactly 0, and why probabilities are clipped or smoothed.
• "Why is the ELBO a lower bound?" Jensen for the concave log: $\log \mathbb{E}_q[p(x, z)/q(z)]$ $\ge \mathbb{E}_q[\log(p(x, z)/q(z))]$.
• "Which distribution on $K$ outcomes has the largest entropy?" The uniform one, because $\log K - H(p) = \mathrm{KL}(p\,\|\,u) \ge 0$.

10. Practice #

The notebook for this lesson runs in Colab. It has five exercises, each with a check cell and a folded solution:

1. jensen_gap(f, values, probs), and a check of the Taylor estimate $\sigma^2/(2\mu^2)$ for $\log$.
2. kl_divergence(p, q, base) with the right handling of zeros, checked against scipy.special.rel_entr.
3. Random pairs of distributions: check $\mathrm{KL} \ge 0$, and find the pair with the largest asymmetry.
4. Verify $\log K - H(p) = \mathrm{KL}(p\,\|\,u)$ and $H(Y) - H(Y \mid X) = \mathrm{KL}(p(x,y)\,\|\,p(x)p(y))$ on random tables.
5. The log-sum inequality on random positive vectors, and its equality case.

11. Watch and read #

Jensen's InequalityMutual Information · 7 min

The chord and centre-of-mass picture of section 2, with the equality condition.

Intuitively Understanding the KL DivergenceAdian Liusie · 5 min

KL as the average log-ratio of two distributions, built up from the likelihood ratio.

A Short Introduction to Entropy, Cross-Entropy and KL-DivergenceAurélien Géron · 11 min

Entropy, cross-entropy and KL as message lengths; the same decomposition as our coin example.

The Key Equation Behind ProbabilityArtem Kirsanov · 26 min

Surprise, entropy, cross-entropy and KL divergence in one story, with clear animations.

• David MacKay, Information Theory, Inference, and Learning Algorithms, free PDF: §2.6 (Gibbs' inequality) and §2.7 (Jensen's inequality, with the centre-of-mass picture); exercise 2.14 asks you to prove Jensen.
• Kevin Murphy, Probabilistic Machine Learning: An Introduction, free PDF: §6.2 (KL divergence: definition, non-negativity, KL and maximum likelihood).
• Yury Polyanskiy and Yihong Wu, Information Theory: From Coding to Learning, free draft: chapter 2 (divergence, its convexity and the log-sum inequality), for a rigorous treatment.