Cross-entropy: the loss of classifiers and language models

Before this lesson: lesson 01 (entropy), lesson 02 (conditional entropy) and lesson 03 (KL divergence, Gibbs' inequality). Maximum likelihood from the probability track; the softmax gradient also appears in calculus lesson 06. Logistic regression (ML fundamentals track, in preparation) can be read in parallel. Time: about 70 minutes, plus the notebook. This is a central lesson; take a break after section 3.4.

---

Why this matters #

You meet cross-entropy every day in machine learning.

• Classifiers (images, text, tabular data) are trained with softmax cross-entropy.
• Language models are trained to predict the next token with cross-entropy. Papers report the result as a loss in nats, as perplexity, or as bits per byte.
• Image and audio generative models report bits per dimension, which is a cross-entropy.
• Label smoothing and knowledge distillation change the target of the cross-entropy.
• Scaling laws for language models have an irreducible term: the entropy part of the loss.

When a loss is nan, when two perplexities cannot be compared, or when a training loss is "too good to be true", the explanation is usually in this lesson.

As in the whole track, $\log$ is the natural log (nats) and $\log_2$ gives bits; PyTorch and NumPy losses are in nats.

1. Predict first #

2. The picture #

Cross-entropy: average surprise under your model #

Recall the coin of lesson 03. The coin is fair, $p = (0.5, 0.5)$, but the model says $q = (0.8, 0.2)$. Heads surprises the model by $-\log_2 0.8 = 0.322$ bits and tails by $-\log_2 0.2 = 2.322$ bits. Each side comes up half of the time, so the model's average surprise is

This is the cross-entropy. It has two parts:

The entropy $H(p) = 1$ bit is there because the coin is random. The KL part, $0.322$ bits, is there because the model is wrong.

Even a perfect model is surprised by a fair coin: 1 bit per flip. Only the second part depends on the model.

Interactive figure: loading…

One example at a time: the log-loss #

A classifier sees an input $x$ and outputs probabilities $q_\theta(y \mid x)$ for every class $y$. The true label is one class, $y^\star$. The log-loss (or negative log-likelihood, NLL) of this example is the surprise of the model at the true label:

• The model gives the true class probability 1: $\ell = 0$.
• Probability 0.5: $\ell = \log 2 = 0.693$ nats (1 bit).
• Probability 0.01: $\ell = \log 100 = 4.61$ nats.
• Probability $\to 0$: $\ell \to \infty$. A confident mistake has no upper limit on its cost.

Interactive figure: loading…

3. Definitions and main results #

3.1 Cross-entropy #

The cross-entropy of a distribution $q$ relative to a distribution $p$ is

Read it as: the data come from $p$, you predict with $q$, and you pay $-\log q(x)$ for each outcome. The first argument is the data distribution, the second is the model. As for KL, $H(p, q) = \infty$ if $q(x) = 0$ for an outcome with $p(x) > 0$, and outcomes with $p(x) = 0$ cost nothing.

Theorem 1 (cross-entropy = entropy + KL).

So $H(p, q) \ge H(p)$, with equality if and only if $q = p$.

Proof.

1. For every $x$ with $p(x) > 0$: $-\log q(x) = -\log p(x) + \log \frac{p(x)}{q(x)}$ (add and subtract $\log p(x)$).
2. Take the average under $p$. The first term gives $H(p)$, the second gives $\mathrm{KL}(p\,\|\,q)$.
3. Gibbs' inequality (lesson 03) says $\mathrm{KL}(p\,\|\,q) \ge 0$, with equality only for $q = p$. $\blacksquare$

Log-loss is a strictly proper scoring rule. A scoring rule gives a loss $S(q, y)$ to a predicted distribution $q$ when the outcome is $y$. It is strictly proper if the expected loss $\mathbb{E}_{y \sim p}[S(q, y)]$ is smallest only at $q = p$. For the log-loss, the expected loss is $H(p, q)$, and Theorem 1 shows that its only minimizer is $q = p$. So a model trained with log-loss has a reason to report its true beliefs, not more confident or less confident ones.

3.2 Minimizing the log-loss is maximum likelihood #

Take a data set $x_1, \dots, x_N$ (discrete values) and a model $q_\theta$. The empirical distribution $\hat p$ puts mass $1/N$ on each data point: $\hat p(x) = \frac{1}{N}\#\{i : x_i = x\}$.

Theorem 2 (log-loss = NLL = forward KL).

So the three problems "minimize the average log-loss", "maximize the likelihood $\prod_i q_\theta(x_i)$" and "minimize $\mathrm{KL}(\hat p\,\|\,q_\theta)$" have the same solutions.

Proof.

1. Group equal data points. The value $x$ appears $N \hat p(x)$ times, so the sum over the data points equals a sum over the values: $\sum_i -\log q_\theta(x_i) = \sum_x N\hat p(x)\,(-\log q_\theta(x))$. Divide by $N$: the average log-loss is $\sum_x \hat p(x)(-\log q_\theta(x)) = H(\hat p, q_\theta)$.
2. Theorem 1 with $p = \hat p$ gives $H(\hat p, q_\theta) = H(\hat p) + \mathrm{KL}(\hat p\,\|\,q_\theta)$.
3. $H(\hat p)$ depends on the data only, not on $\theta$. So the minimizers of the average log-loss and of $\mathrm{KL}(\hat p\,\|\,q_\theta)$ are the same.
4. $\frac1N \sum_i -\log q_\theta(x_i) = -\frac1N \log \prod_i q_\theta(x_i)$, and $-\frac1N \log$ is decreasing. So minimizing the average log-loss is maximizing the likelihood. $\blacksquare$

As $N \to \infty$, the average log-loss tends to $H(p_{\text{data}}, q_\theta) = H(p_{\text{data}}) + \mathrm{KL}(p_{\text{data}}\,\|\,q_\theta)$. The KL has the data first and the model second: this is the forward KL. The other direction, $\mathrm{KL}(q_\theta\,\|\,p)$ (the reverse KL), is what variational inference minimizes, and it behaves very differently (lesson 06).

3.3 Classification: softmax and its gradient #

A classifier with $K$ classes computes a vector of logits $z = (z_1, \dots, z_K)$ (real numbers) and turns them into probabilities with the softmax

Write the true label as a one-hot vector $y$: $y_k = 1$ for the true class $k = c$, and 0 for the others. The loss is the cross-entropy between $y$ and $q$:

The last form is how libraries compute it. The function $\operatorname{LSE}(z) = \log \sum_j e^{z_j}$ (log-sum-exp) is evaluated as $m + \log \sum_j e^{z_j - m}$ with $m = \max_j z_j$, so no exponential overflows.

Theorem 3 (gradient of softmax cross-entropy). $\dfrac{\partial \ell}{\partial z_k} = q_k - y_k$, that is, $\nabla_z \ell = q - y$: prediction minus target.

So the gradient on the true logit is $q_c - 1 < 0$ (raise it), and on every other logit it is $q_k > 0$ (lower it). The entries add up to $\sum_k q_k - \sum_k y_k = 1 - 1 = 0$.

Proof of Theorem 3

1. $\ell(z) = -z_c + \operatorname{LSE}(z)$.
2. The derivative of the first term: $\frac{\partial}{\partial z_k}(-z_c) = -1$ if $k = c$ and 0 otherwise, that is, $-y_k$.
3. The derivative of log-sum-exp: $\frac{\partial}{\partial z_k} \log \sum_j e^{z_j} = \frac{e^{z_k}}{\sum_j e^{z_j}} = q_k$ (chain rule: the derivative of $\log s$ is $1/s$, and $\partial s / \partial z_k = e^{z_k}$).
4. Add: $\frac{\partial \ell}{\partial z_k} = q_k - y_k$. $\blacksquare$

The same proof works for any target distribution $t$ (not only one-hot) with $\ell = H(t, q)$: $\nabla_z \ell = q - t$, because $\sum_k t_k = 1$. Calculus lesson 06 derives the batch version and the Hessian.

Binary case. With two classes, a model often outputs one logit $z$ and $q = \sigma(z) = 1/(1 + e^{-z})$, the probability of class 1. The loss with label $y \in \{0, 1\}$ is the binary cross-entropy (BCE)

and its gradient is $\dfrac{\partial \ell}{\partial z} = \sigma(z) - y$.

This is softmax cross-entropy with the two logits $(z, 0)$, so the gradient is again prediction minus target.

Interactive figure: loading…

3.4 The floor: the entropy of the labels #

A classifier is trained on pairs $(x, y)$ from a data distribution $p(x, y)$. Its expected loss is $\mathbb{E}_{(x, y) \sim p}\big[-\log q_\theta(y \mid x)\big]$.

Theorem 4 (the floor).

Proof.

1. Write the expectation as an outer average over $x$ and an inner average over $y$ given $x$. The inner average is $\sum_y p(y \mid x)\big(-\log q_\theta(y \mid x)\big)$, which is the cross-entropy $H\big(p(\cdot \mid x), q_\theta(\cdot \mid x)\big)$.
2. Apply Theorem 1 inside, for each $x$: $H(p(\cdot \mid x)) + \mathrm{KL}(p(\cdot \mid x)\,\|\,q_\theta(\cdot \mid x))$.
3. The average over $x$ of $H(p(\cdot \mid x))$ is the conditional entropy $H(Y \mid X)$ (lesson 02). $\blacksquare$

Only the KL part depends on the model. No model can get below $H(Y \mid X)$, the uncertainty of the label that remains when you know the input.

Example: label noise. The clean label is a function of $x$, but each label is flipped with probability 0.1. For every $x$, $p(\cdot \mid x)$ is a coin with probabilities $(0.9, 0.1)$, so $H(Y \mid X) = H_b(0.1) = 0.469$ bits $= 0.325$ nats. The best model predicts 0.9 for the clean class. A training loss far below 0.325 nats means the model has memorized the flipped labels; its test loss cannot go below 0.325 nats.

For a language model, the floor is the entropy of the next token given the context. Natural text is not deterministic, so this floor is positive. The irreducible term in the scaling laws of language models plays this role (LLMs track, in preparation).

3.5 Soft targets: label smoothing and distillation #

The target does not have to be one-hot. For a target distribution $t$, the loss is $H(t, q)$, and by Theorem 1

The minimum over $q$ is $H(t)$, reached at $q = t$.

Label smoothing. Replace the one-hot target by $t = (1 - \varepsilon)\,y + \frac{\varepsilon}{K}$ (every class gets $\varepsilon/K$ extra). With $\varepsilon = 0.1$ and $K = 3$: $t = (0.933, 0.033, 0.033)$. The model is never pushed towards $q_c = 1$, so its logits stay finite and its predictions are less overconfident. Label smoothing was used in Inception-v3 and in the original Transformer, both with $\varepsilon = 0.1$.

Knowledge distillation. A large teacher network with logits $z^{\text{T}}$ gives soft targets $t = \operatorname{softmax}(z^{\text{T}} / T)$, where $T \ge 1$ is a temperature. A higher $T$ gives a flatter distribution and shows which wrong classes the teacher finds similar ("dark knowledge"). For teacher logits $(3, 1, 0.2)$: $T = 1$ gives $(0.836, 0.113, 0.051)$, and $T = 5$ gives $(0.446, 0.299, 0.255)$. The student is trained on $\mathrm{KL}(t\,\|\,q_T)$, where $q_T$ is its own softmax at the same temperature, usually mixed with the normal loss on the true labels. Hinton et al. multiply this term by $T^2$, because its gradient shrinks like $1/T^2$.

Cross-entropy or KL as the loss? $H(t, q)$ and $\mathrm{KL}(t\,\|\,q)$ differ by $H(t)$, which does not depend on the student. So they have the same gradient and the same minimizer. The difference is only the reported number: the KL is 0 at the optimum, the cross-entropy is $H(t)$.

Interactive figure: loading…

3.6 Why not squared error or accuracy? #

Accuracy (or the 0–1 loss) is piecewise constant in the logits: its gradient is 0 almost everywhere, so gradient descent cannot use it. It is also not strictly proper: every $q$ with the right most likely class gets the same score, so it does not reward good probabilities.

Squared error on probabilities (the Brier score, $\sum_k (q_k - y_k)^2$) is a strictly proper scoring rule too, so the issue is not its minimizer. The issue is the gradient through the sigmoid or softmax. For two classes with true label 1, $\frac{d}{dz}(1 - \sigma(z))^2 = -2\sigma(z)(1 - \sigma(z))^2$. For a confident mistake, $z = -5$: $\sigma(z) = 0.0067$, the squared-error gradient is $-0.013$, but the log-loss gradient is $\sigma(z) - 1 = -0.993$. The log cancels the flat tail of the sigmoid, so the log-loss learns fast from exactly the examples that are most wrong. And the log-loss is the likelihood, so all of maximum-likelihood theory applies.

Calibration. A model is calibrated if, among the inputs where it says "70%", about 70% are correct. Because the log-loss is proper, a model that minimizes the expected log-loss is calibrated. Large networks still end up overconfident on test data, because they overfit the training loss. Temperature scaling (dividing the logits by one number $T$ fitted on validation data) fixes most of it (ML fundamentals track, in preparation).

3.7 Units: nats, bits, perplexity, bits per dimension #

These are all the same quantity, an average NLL, in different units. Let $\bar\ell$ be the average NLL per token (or per symbol) in nats.

unit	formula	meaning
bits per token	$\bar\ell / \ln 2$	the same loss with $\log_2$
perplexity	$\mathrm{PPL} = e^{\bar\ell} = 2^{\bar\ell / \ln 2}$	effective number of equally likely choices
bits per dimension	$\text{NLL}_{\text{nats}} / (D \ln 2)$	NLL of one image (or sound) divided by its $D$ numbers
bits per byte	$\bar\ell / (\ln 2 \cdot \text{bytes per token})$	does not depend on the tokenizer

• Perplexity is the geometric mean of $1/q(x_t)$ over the tokens: $e^{\bar\ell} = \big(\prod_{t=1}^n \frac{1}{q(x_t)}\big)^{1/n}$. A model that is uniform over $V$ tokens has $\bar\ell = \log V$ and $\mathrm{PPL} = V$: it "chooses among $V$ tokens". By Theorem 1, the perplexity is at least $e^{H}$, where $H$ is the entropy per token of the text.
• Bits per dimension (bpd): an image has $D = \text{height} \times \text{width} \times \text{channels}$ numbers. Dividing by $D$ lets you compare images of different sizes.
• Bits per byte: a tokenizer with longer tokens has fewer tokens per text, so its loss per token is larger. Dividing the total bits by the number of raw bytes removes this effect.

Interactive figure: loading…

Continuous models of discrete data: dequantization

Images are integers, $x \in \{0, \dots, 255\}^D$, but flows and diffusion models use densities $p$ on $\mathbb{R}^D$. A density can put arbitrarily tall spikes on the integer points, so the likelihood of the raw integers can be made arbitrarily large and means nothing. The standard fix is uniform dequantization: train and evaluate on $x + u$ with $u \sim \mathrm{Uniform}[0, 1)^D$.

Theorem 5 (dequantization bound, Theis et al. 2016, §3.1). Define the discrete model $P(x) = \int_{[0,1)^D} p(x + u)\,du$ for integer points $x$. Then $\sum_x P(x) \le 1$, and

$$\mathbb{E}_{u}\big[\log p(x + u)\big] \le \log P(x).$$

Proof.

1. The unit cubes $x + [0, 1)^D$ do not overlap, so $\sum_x P(x)$ is the mass of $p$ on their union, which is at most 1.
2. The cube has volume 1, so $P(x) = \mathbb{E}_u[p(x + u)]$. Jensen for the concave $\log$ (lesson 03): $\mathbb{E}_u[\log p(x + u)] \le \log \mathbb{E}_u[p(x + u)] = \log P(x)$. $\blacksquare$

So the continuous NLL on dequantized data is an upper bound on $-\log P(x)$, the code length of a real discrete model: the reported bits per dimension are conservative.

Rescaling. Models often work on $y = (x + u)/256 \in [0, 1)^D$. The density changes by the Jacobian: $p_y(y) = 256^D\,p(x + u)$. So $-\log_2 p(x + u) = -\log_2 p_y(y) + 8D$: add 8 bits per dimension to an NLL measured on the $[0, 1)$ scale. For data scaled by $1/a$, add $\log_2 a$ bits per dimension.

Answers to section 1. (1) $-\log 0.01 = 4.61$ nats. (2) $2^3 = 8$. (3) No. The floor is $H(Y \mid X) = H_b(0.1) = 0.325$ nats per example (section 3.4), even for a perfect model.

4. Worked example #

(a) Softmax cross-entropy from logits. Logits $z = (2, 1, 0)$, true class 1.

1. Exponentials: $e^2 = 7.389$, $e^1 = 2.718$, $e^0 = 1$. Sum: $11.107$.
2. Softmax: $q = (7.389, 2.718, 1)/11.107$ $= (0.665, 0.245, 0.090)$.
3. Loss: $-\log 0.665 = 0.408$ nats $= 0.588$ bits. With log-sum-exp: $-z_1 + \log 11.107 = -2 + 2.408 = 0.408$. ✓
4. Gradient: $q - y = (0.665 - 1, 0.245, 0.090)$ $= (-0.335, 0.245, 0.090)$. The entries add up to 0.
5. If the true class were class 3, the loss would be $-\log 0.090 = 2.408$ nats: the same logits, a confident mistake, six times the loss.

(b) Label smoothing. Same logits, $\varepsilon = 0.1$, $K = 3$: $t = (0.933, 0.033, 0.033)$.

1. Loss: $H(t, q) = -(0.933 \log 0.665 + 0.033 \log 0.245 + 0.033 \log 0.090)$ $= 0.933 \cdot 0.408 + 0.033 \cdot 1.408 + 0.033 \cdot 2.408 = 0.380 + 0.047 + 0.080 = 0.508$ nats.
2. Entropy of the target: $H(t) = -0.933 \log 0.933 - 2 \cdot 0.033 \log 0.033 = 0.064 + 0.227 = 0.291$ nats.
3. KL part: $0.508 - 0.291 = 0.216$ nats. So $0.508 = 0.291 + 0.216$, the split of Theorem 1.
4. Gradient: $q - t = (0.665 - 0.933,\ 0.245 - 0.033,\ 0.090 - 0.033)$ $= (-0.268, 0.211, 0.057)$.

(c) Distillation. Teacher logits $(3, 1, 0.2)$, student logits $(2, 1, 0)$.

• $T = 1$: $t = (0.836, 0.113, 0.051)$ and $q = (0.665, 0.245, 0.090)$. $H(t, q) = 0.622 = H(t)\ 0.548 + \mathrm{KL}\ 0.075$ nats.
• $T = 5$: $t = (0.446, 0.299, 0.255)$ and $q_5 = \operatorname{softmax}((2, 1, 0)/5)$ $= (0.402, 0.329, 0.269)$. $\mathrm{KL}(t\,\|\,q_5) = 0.0042$ nats; times $T^2 = 25$ gives $0.104$.

(d) Units. A language model has $\bar\ell = 2.5$ nats per token.

• Bits per token: $2.5 / 0.693 = 3.61$. Perplexity: $e^{2.5} = 12.2$.
• Model A: 3.0 nats per token, 4.0 bytes per token. Model B: 2.0 nats per token, 2.5 bytes per token. Perplexity: A $e^3 = 20.1$, B $e^2 = 7.39$. Bits per byte: A $3.0/(4.0 \cdot 0.693) = 1.08$, B $2.0/(2.5 \cdot 0.693) = 1.15$. B has the lower perplexity, but A predicts the text better.
• An image model has an NLL of 25 000 nats on one $64 \times 64 \times 3$ image, so $D = 12\,288$ and $\text{bpd} = 25\,000/(12\,288 \cdot 0.693) = 2.94$.

5. In code #

NumPy and SciPy check the numbers above. The notebook builds these functions step by step.

import numpy as np
from scipy.special import logsumexp, softmax
from scipy.stats import entropy

# (a) softmax cross-entropy from logits, the stable way
z, c = np.array([2.0, 1.0, 0.0]), 0                # true class 1 is index 0
loss = logsumexp(z) - z[c]
q = softmax(z)
y = np.eye(3)[c]
print(q, loss, loss / np.log(2))                    # [0.6652 0.2447 0.09  ] 0.4076 0.5881
print(q - y, np.isclose((q - y).sum(), 0))          # [-0.3348  0.2447  0.09  ] True

# H(p, q) = H(p) + KL(p || q) for the coin, in bits
p, qc = np.array([0.5, 0.5]), np.array([0.8, 0.2])
print(-(p * np.log2(qc)).sum(), entropy(p, base=2) + entropy(p, qc, base=2))   # 1.3219 1.3219

# (b) label smoothing: loss = H(t) + KL(t || q)
t = 0.9 * y + 0.1 / 3
print(-(t * np.log(q)).sum(), entropy(t), entropy(t, q))   # 0.5076 0.2911 0.2165

# (d) units
nll = 2.5
print(nll / np.log(2), np.exp(nll))                          # 3.6067 12.1825
print(np.log(50257), np.exp(np.log(50257)))                  # 10.8249 50257.0 (uniform model)
print(25_000 / (64 * 64 * 3 * np.log(2)))                    # 2.9352 bits per dimension

In PyTorch, F.cross_entropy takes logits (not probabilities) and class indices or soft targets. F.kl_div(input, target) computes $\mathrm{KL}(\text{target}\,\|\,\cdot)$ and expects input as log-probabilities: the model goes first, the target second, the opposite of the math notation.

import torch, torch.nn.functional as F

z = torch.tensor([[2.0, 1.0, 0.0]])
print(F.cross_entropy(z, torch.tensor([0])).item())                       # 0.4076
print(F.cross_entropy(z, torch.tensor([0]), label_smoothing=0.1).item())  # 0.5076
p, q = torch.tensor([[0.5, 0.5]]), torch.tensor([[0.8, 0.2]])
print(F.kl_div(q.log(), p, reduction="batchmean").item())                 # 0.2231 = KL(p || q)

Never compute torch.log(torch.softmax(z)): for large logits the softmax underflows to 0 and the log gives -inf. Use F.log_softmax or F.cross_entropy, which use log-sum-exp.

6. Where it shows up #

• Image and text classifiers. Softmax cross-entropy on one-hot labels, sometimes with label smoothing (deep learning lesson 06). Binary and multi-label problems use the sigmoid with BCE on each output.
• Language models. Pretraining minimizes the average next-token cross-entropy. By the chain rule (lesson 02), the sum of per-token conditional losses is the NLL of the whole sequence. Papers report the loss in nats, perplexity, or bits per byte (LLMs track, in preparation).
• Scaling laws. The fitted loss curve $L(N, D) = E + A/N^\alpha + B/D^\beta$ has an irreducible term $E$: the entropy part of the cross-entropy, which more parameters or data cannot remove.
• Distillation. DistilBERT and many small language and vision models are trained on soft teacher outputs with a KL term at a temperature.
• Generative models. Autoregressive image models, flows and diffusion models report bits per dimension on dequantized data (generative models track).
• ML for science. The log score $-\log q(y)$ evaluates probabilistic forecasts (precipitation categories, earthquake magnitudes, extreme-event warnings). Because it is strictly proper, a forecaster cannot improve the expected score by hedging or by exaggerating its confidence.

7. Common mistakes #

• The arguments in the wrong order. $H(p, q)$ averages under the data $p$ and takes the log of the model $q$. $H(q, p)$ is a different number.
• log(softmax(z)) in code. It underflows to $\log 0 = -\infty$. Use log_softmax or a loss that takes logits.
• F.kl_div with probabilities or swapped arguments. It expects input = log-probabilities of the model and target = probabilities of the target. Wrong inputs give a wrong number and no error.
• Gradient with respect to the probability. $\partial \ell / \partial q_c = -1/q_c$, but the gradient that training uses is with respect to the logits: $q - y$.
• Averaging perplexities. Perplexity is the exponential of the average loss. The average of per-document perplexities is larger: for losses 1 and 3 nats, $e^{2} = 7.39$, but $\frac12(e^1 + e^3) = 11.4$ (Jensen, $e^x$ is convex).
• Comparing perplexities across tokenizers. Use bits per byte.
• Expecting a loss of 0. With noisy labels, label smoothing or ambiguous inputs the floor is $H(Y \mid X) > 0$ or $H(t) > 0$. A training loss below the floor means memorization.
• Bits per dimension without the $+8$ after rescaling 8-bit data to $[0, 1)$, or on raw integers without dequantization.

8. Check yourself #

1. A binary classifier outputs $\sigma(z) = 0.8$ for an example with label $y = 0$. Compute the loss in nats and the gradient with respect to $z$.
   Answer

   $\ell = -\log(1 - 0.8) = -\log 0.2 = 1.609$ nats. $\partial \ell / \partial z = \sigma(z) - y = 0.8$: positive, so gradient descent lowers $z$.
2. Show that the gradient of $H(t, q)$ with respect to the logits is the same as the gradient of $\mathrm{KL}(t\,\|\,q)$, for any fixed target $t$.
   Answer

   $H(t, q) = H(t) + \mathrm{KL}(t\,\|\,q)$ (Theorem 1), and $H(t)$ does not depend on the logits. So both have the gradient $q - t$.
3. The labels of a data set are flipped with probability 0.2. Your training loss is 0.30 nats and your test loss is 0.70 nats. What do you conclude?
   Answer

   The floor is $H_b(0.2) = 0.500$ nats. The training loss of 0.30 is below the floor, so the model has memorized some flipped labels. The test loss of 0.70 is 0.20 nats above the floor: that is the KL part, the room for improvement on new data.
4. A language model is uniform over a vocabulary of $V = 32\,000$ tokens. Give its loss in nats and in bits per token, and its perplexity.
   Answer

   Each token has $q = 1/V$, so $\bar\ell = \log 32\,000 = 10.37$ nats $= \log_2 32\,000 = 14.97$ bits per token, and $\mathrm{PPL} = e^{10.37} = 32\,000$.
5. A flow on $32 \times 32 \times 3$ images, with data scaled to $[0, 1)$, reports $\log p_y(y) = +8000$ nats per image. What are the bits per dimension on the 8-bit scale? Is the positive log-density a bug?
   Answer

   $D = 3072$. On the $[0, 1)$ scale, $-8000/(3072 \cdot 0.693) = -3.757$ bits per dimension; add 8: $4.24$ bits per dimension. Not a bug: a density on $[0, 1)^D$ can be larger than 1, so its log can be positive. Only the corrected number bounds a code length.
6. Label smoothing with $\varepsilon = 0.1$ on $K = 10$ classes. What is the target for the true class and for each other class? What is the gradient on a wrong logit when the model predicts $q_k = 0$ for it?
   Answer

   True class: $0.9 + 0.01 = 0.91$; each other class: $0.01$. Gradient $q_k - t_k = 0 - 0.01 = -0.01$: negative, so gradient descent *raises* that logit a little. The model is pushed away from total confidence.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Define the cross-entropy $H(p, q)$.

   Show answer

2. How do cross-entropy, entropy and KL divergence relate?

   Show answer

3. Why is minimizing the average cross-entropy on data the same as maximum likelihood?

   Show answer

4. What is the gradient of softmax cross-entropy with respect to the logits?

   Show answer

5. What is the lowest expected loss a classifier can reach?

   Show answer

6. Compute the loss floor for binary labels flipped with probability 0.1.

   Show answer

7. What is the label-smoothing target for the true class and the other classes?

   Show answer

8. What does a knowledge-distillation loss minimize?

   Show answer

9. How is perplexity computed from the average loss $\bar\ell$ in nats?

   Show answer

10. How do you compute bits per dimension from an NLL in nats?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Why cross-entropy and not MSE for classification?" It is the NLL of a categorical model, so training is maximum likelihood. With a softmax, its gradient is $q - y$, which stays large for confident mistakes; squared error through a sigmoid has a gradient that vanishes there.
• "Derive the gradient of softmax cross-entropy." $\ell = \operatorname{LSE}(z) - z_c$, and $\partial \operatorname{LSE} / \partial z_k = q_k$, so $\nabla_z \ell = q - y$.
• "What is perplexity? What is it for a uniform model over 50k tokens?" The exponential of the average NLL per token in nats, the effective number of choices; 50k.
• "Your training loss is much lower than you expect from the label noise. What is happening?" The loss cannot go below $H(Y \mid X)$ on new data; the model is memorizing noisy labels. Use early stopping, regularization or label smoothing.
• "Cross-entropy or KL for distillation?" They differ by the entropy of the teacher's targets, which is a constant, so the gradients are the same.
• "How do you compute cross-entropy without overflow?" From the logits with log-sum-exp: $\operatorname{LSE}(z) - z_c$ with the max subtracted first.

10. Practice #

The notebook for this lesson runs in Colab. It has five exercises, each with a check cell and a folded solution:

1. cross_entropy(p, q, base), and a check of $H(p, q) = H(p) + \mathrm{KL}(p\,\|\,q)$.
2. A numerically stable softmax cross-entropy from logits (log-sum-exp), checked against sklearn.metrics.log_loss, also for logits of size 1000.
3. A gradient check of $q - y$ by finite differences.
4. Logistic regression by gradient descent on a toy set whose labels are flipped with probability 0.1; plot the loss against the floor $H_b(0.1) = 0.325$ nats.
5. perplexity(nll_nats) and bits_per_dim(nll_nats, D), with the uniform-model checks.

11. Watch and read #

But what is cross-entropy? | Compression is Intelligence Part 23Blue1Brown · 34 min

Cross-entropy as the cost of coding with the wrong distribution, and why it is the loss of language models.

Neural Networks Part 6: Cross EntropyStatQuest with Josh Starmer · 10 min

The softmax cross-entropy loss on a small network, computed by hand.

Neural Networks Part 7: Cross Entropy Derivatives and BackpropagationStatQuest with Josh Starmer · 22 min

The derivative of the loss with respect to the logits, step by step; compare with Theorem 3.

• Kevin Murphy, Probabilistic Machine Learning: An Introduction, free PDF: §6.1.2 (cross-entropy), §6.1.5 (perplexity) and §6.2.5 (KL and maximum likelihood).
• Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network (2015): soft targets, temperature and the $T^2$ factor.
• Müller, Kornblith and Hinton, When Does Label Smoothing Help? (2019): what label smoothing does to the representations and to calibration.
• Guo, Pleiss, Sun and Weinberger, On Calibration of Modern Neural Networks (2017): overconfidence and temperature scaling.
• Theis, van den Oord and Bethge, A note on the evaluation of generative models (2016), §3.1: the dequantization bound and bits per dimension.