From scores to probabilities: sigmoid, softmax and log-sum-exp

Before this lesson: lesson 04 (exp, log and the log rules), lesson 02 ($\sum$, max and argmax) and lesson 03 (arrays, axis, broadcasting). Time: about 55 minutes, plus the notebook.

---

Why this matters #

Almost every classifier ends with these functions. Logistic regression is $\sigma(w^\top x + b)$. An image classifier or a language model ends with a softmax over its classes or its vocabulary, and is trained with the cross-entropy loss $\operatorname{LSE}(z) - z_y$. Attention in a transformer turns similarity scores into weights with a softmax. When a chatbot "samples with temperature 0.7", it divides the logits by 0.7 before the softmax. The stable formulas of this lesson are inside torch.nn.CrossEntropyLoss, torch.logsumexp and scipy.special.logsumexp; when you write your own loss, you need them too.

1. Predict first #

2. The picture #

One score, one probability: the sigmoid. The sigmoid (also called the logistic function) is

Read the formula from the inside. For large positive $z$, $e^{-z}$ is close to 0, so $\sigma(z)$ is close to 1. For large negative $z$, $e^{-z}$ is huge, so $\sigma(z)$ is close to 0. At $z = 0$, $\sigma(0) = 1/(1 + 1) = 1/2$. The graph is an S-shaped curve from 0 to 1. In a model, the score is often $z = wx + b$: the weight $w$ sets how steep the S is, and the curve crosses $1/2$ where $wx + b = 0$, at $x = -b/w$.

Interactive figure: loading…

Many scores, one probability vector: the softmax. For $n$ classes with logits $z = (z_1, \dots, z_n)$, the softmax is

Two steps: exponentiate (every $e^{z_k}$ is positive, and a larger $z_k$ gives a larger $e^{z_k}$), then normalize (divide by the sum, so the entries add up to 1). A temperature $T > 0$ divides the logits first: $\operatorname{softmax}(z / T)$.

Interactive figure: loading…

3. Definitions and main results #

3.1 The sigmoid and the logit #

Properties of $\sigma(z) = 1/(1 + e^{-z})$.

1. Range $(0, 1)$. $e^{-z} > 0$, so the denominator is larger than 1: $0 < \sigma(z) < 1$. The values 0 and 1 are never reached; they are limits as $z \to -\infty$ and $z \to +\infty$.
2. Increasing. When $z$ grows, $e^{-z}$ shrinks, the denominator shrinks, and $\sigma(z)$ grows.
3. Symmetry: $\sigma(-z) = 1 - \sigma(z)$. Proof. $1 - \sigma(z) = \dfrac{1 + e^{-z} - 1}{1 + e^{-z}} = \dfrac{e^{-z}}{1 + e^{-z}}$. Multiply the numerator and the denominator by $e^{z}$: $\dfrac{1}{e^{z} + 1} = \sigma(-z)$. $\blacksquare$
4. Another form: $\sigma(z) = \dfrac{e^{z}}{1 + e^{z}}$ (multiply the numerator and the denominator by $e^z$).

The inverse of the sigmoid is the logit. For a probability $p \in (0, 1)$, the odds are $p / (1 - p)$ (for example, $p = 0.8$ gives odds 4, "4 to 1"). The logit is the log of the odds:

Why it inverts $\sigma$. Let $p = \sigma(z) = 1/(1 + e^{-z})$.

1. Take the reciprocal: $1/p = 1 + e^{-z}$, so $e^{-z} = (1 - p)/p$.
2. Take the log: $-z = \log\frac{1 - p}{p}$, so $z = \log\frac{p}{1 - p}$. $\blacksquare$

So the score $z$ of a sigmoid model is exactly the log-odds of its probability. Example: $\operatorname{logit}(0.8) = \log 4 \approx 1.386$, and $\sigma(1.386) \approx 0.8$.

Naming. "Logit" is a function of a probability. In deep learning, logits (plural) means the raw scores that go into a sigmoid or a softmax. For a sigmoid the two meanings agree; for a softmax, the logits are log-probabilities up to one shared constant (section 3.5).

Slope of the sigmoid. The slope (derivative) of $\sigma$ at $z$ is

We use this formula here without proof; lesson 07 checks it. Since $s(1 - s) = \tfrac14 - (s - \tfrac12)^2$, the slope is at most $1/4$, reached at $z = 0$. For large $|z|$ the slope is tiny: $\sigma'(5) \approx 0.0066$, $\sigma'(10) \approx 0.000045$. This flatness is called saturation; it is one cause of vanishing gradients in deep networks. For $p(x) = \sigma(wx + b)$ the slope at the centre $x = -b/w$ is $w \cdot \tfrac14 = w/4$ (the chain rule, lesson 07).

Logistic regression in one paragraph. For inputs $x \in \R^n$, logistic regression models $P(y = 1 \mid x) = \sigma(w^\top x + b)$. It predicts class 1 when this probability is at least $1/2$, that is, when $w^\top x + b \ge 0$. So the boundary between the classes is the flat set $w^\top x + b = 0$ (a line in $\R^2$, a plane in $\R^3$), and $w^\top x + b$ is the log-odds of class 1.

3.2 The softmax #

Properties. For any $z \in \R^n$, $p = \operatorname{softmax}(z)$ satisfies:

1. $p_k > 0$ for every $k$ (an exponential is positive, and so is a sum of them);
2. $\sum_k p_k = \sum_k e^{z_k} / \sum_j e^{z_j} = 1$;
3. order is kept: $z_i > z_j$ implies $p_i > p_j$ (because $e^z$ is increasing and all $p_k$ share one denominator). So $\arg\max_k p_k = \arg\max_k z_k$.

Why the exponential? The simpler idea "divide by the sum", $z_k / \sum_j z_j$, fails:

• with $z = (-1, 2)$ it gives $(-1, 2)$: a negative "probability" and one larger than 1;
• with $z = (1, -1)$ the sum is 0: division by zero;
• it changes when you add the same number to every score.

The exponential makes every score positive, keeps the order, and turns differences of scores into ratios of probabilities: $p_i / p_j = e^{z_i - z_j}$.

Theorem (shift invariance). For every real number $c$, $\operatorname{softmax}(z + c\mathbf{1}) = \operatorname{softmax}(z)$, where $z + c\mathbf 1 = (z_1 + c, \dots, z_n + c)$.

Proof.

1. By the rule $e^{a + b} = e^a e^b$ (lesson 04): $e^{z_k + c} = e^c e^{z_k}$.
2. So $\operatorname{softmax}(z + c\mathbf 1)_k = \dfrac{e^c e^{z_k}}{\sum_j e^c e^{z_j}} = \dfrac{e^c e^{z_k}}{e^c \sum_j e^{z_j}}$.
3. Cancel $e^c \ne 0$: the result is $\operatorname{softmax}(z)_k$. $\blacksquare$

So only the differences between logits matter. This is the reason for the max trick of section 3.6.

Two classes give the sigmoid. With logits $(z, 0)$:

By shift invariance, $\operatorname{softmax}(z_1, z_2)_1 = \operatorname{softmax}(z_1 - z_2, 0)_1 = \sigma(z_1 - z_2)$. So a binary classifier needs only one score: the difference of the two logits.

3.3 Temperature #

For $T > 0$, $p^{(T)} = \operatorname{softmax}(z / T)$. Then $p^{(T)}_i / p^{(T)}_j = e^{(z_i - z_j)/T}$.

• $T = 1$: the ordinary softmax.
• $T \to 0$: if the largest logit $z_k$ is unique, $p^{(T)} \to$ the one-hot vector at $k$. Reason: $p^{(T)}_k = 1 / \sum_j e^{(z_j - z_k)/T}$. For $j \ne k$, $z_j - z_k < 0$, so $(z_j - z_k)/T \to -\infty$ and $e^{(z_j - z_k)/T} \to 0$. The sum tends to $e^0 = 1$, so $p^{(T)}_k \to 1$. If several logits tie for the maximum, the probability is split equally between them.
• $T \to \infty$: $z / T \to 0$, and $\operatorname{softmax}(0, \dots, 0) = (1/n, \dots, 1/n)$, the uniform vector.

Example with $z = (1, 2, 3)$: $T = 0.5$ gives $(0.016, 0.117, 0.867)$, $T = 1$ gives $(0.090, 0.245, 0.665)$, $T = 10$ gives $(0.301, 0.332, 0.367)$.

So the softmax is a smooth version of the argmax (as a one-hot vector); a better name would be "soft argmax". Language models sample the next token from $\operatorname{softmax}(z / T)$: small $T$ gives safe, repetitive text, large $T$ more varied text, and $T \to 0$ is greedy decoding (always the most likely token).

3.4 Log-sum-exp: a smooth maximum #

The log-sum-exp of $z \in \R^n$ is

It is the log of the softmax normalizer. Its name says what it does, from the inside out.

Theorem (LSE is close to the max). Let $m = \max_j z_j$. Then

Proof of the bounds

1. One of the terms of $\sum_j e^{z_j}$ is $e^m$, and the other terms are positive. So $\sum_j e^{z_j} \ge e^m$.
2. Every term is at most $e^m$, because $z_j \le m$ and $e^z$ is increasing. So $\sum_j e^{z_j} \le n\,e^m$.
3. Take logs (the log is increasing, so it keeps $\le$): $m \le \operatorname{LSE}(z) \le \log(n e^m) = m + \log n$. $\blacksquare$

The upper bound is an equality exactly when all $z_j$ are equal. The lower bound is strict when $n \ge 2$, because then the other terms are $> 0$.

Examples: $\operatorname{LSE}(1, 2, 3) \approx 3.408$, which is in $[3, 3 + \log 3] \approx [3, 4.099]$. $\operatorname{LSE}(5, 5) = \log(2e^5) = 5 + \log 2 \approx 5.693$: the upper bound, since the two numbers are equal.

With a temperature, $T \operatorname{LSE}(z / T)$ lies in $[m, m + T\log n]$, so it tends to the max as $T \to 0$. For two numbers $z = (x, 0)$, $\operatorname{LSE}(x, 0) = \log(1 + e^x)$ is the softplus function, a smooth version of $\max(x, 0)$ (the ReLU).

Interactive figure: loading…

Two more facts. First, $\operatorname{LSE}(z + c\mathbf 1) = \operatorname{LSE}(z) + c$ (take $e^c$ out of the sum, as in the proof of shift invariance). Second, the log-softmax: take the log of the softmax and use the quotient rule of lesson 04,

3.5 Cross-entropy from logits #

A classifier with logits $z$ and true class $y$ is trained with the cross-entropy loss, the negative log-probability of the true class:

The right-hand side never forms a probability, so it never takes $\log 0$. For $z = (1, 2, 3)$: $y = 3$ gives $3.408 - 3 = 0.408$ (a confident, correct prediction, small loss); $y = 1$ gives $3.408 - 1 = 2.408$ (a large loss). The /information-theory/ track explains why this loss is a cross-entropy.

3.6 Numerical stability: the max trick #

The problem. A float64 number is at most about $1.8 \times 10^{308}$, so $e^z$ overflows to inf for $z > 709.78$ ($= \log(1.8 \times 10^{308})$). In float32 (common on GPUs) the limit is $z > 88.72$. In the other direction, $e^z$ underflows to 0 for $z$ below about $-745$ in float64.

• Naive softmax of $(1000, 1001, 1002)$: every $e^{z_k}$ is inf, and inf / inf is nan.
• Naive softmax of $(-1000, -1001, -1002)$: every $e^{z_k}$ is 0, and 0 / 0 is nan.

The fix. Let $m = \max_j z_j$ and use shift invariance with $c = -m$:

Why it is safe.

1. Every $z_j - m \le 0$, so every term $e^{z_j - m} \le 1$: no overflow.
2. The term with $z_j = m$ is $e^0 = 1$, so the sum is at least 1: no division by 0, and the log of the sum is $\ge 0$.
3. Terms with $z_j - m < -745$ underflow to 0. This is harmless: their true probability is below $10^{-323}$ anyway.

Log-softmax in one step. Compute $z_k - \operatorname{LSE}(z)$ with the stable LSE. Never compute np.log(softmax(z)) in two steps: for $z = (0, 1000)$, even the stable softmax gives $(0, 1)$ in float64 (the true first entry $e^{-1000}$ underflows), and $\log 0 = -\infty$, while the true value $z_1 - \operatorname{LSE}(z)$ is $-1000$.

Interactive figure: loading…

Stable sigmoid. $\sigma(z) = 1/(1 + e^{-z})$ computes $e^{1000}$ for $z = -1000$. The result happens to be right (0), but NumPy warns about the overflow, and $\log \sigma(z)$ becomes $\log 0$. Use $1/(1 + e^{-z})$ for $z \ge 0$ and $e^{z}/(1 + e^{z})$ for $z < 0$: the exponent is then never positive. Libraries do this: scipy.special.expit, torch.sigmoid, and for losses torch.nn.functional.logsigmoid and BCEWithLogitsLoss.

In practice use the library functions: scipy.special.softmax, log_softmax, logsumexp; torch.softmax, torch.log_softmax, torch.logsumexp; and torch.nn.CrossEntropyLoss, which takes logits (not probabilities) for this reason.

Accuracy, the gradient of LSE, and multi-label outputs

• Gradient of LSE. The partial derivative of $\operatorname{LSE}$ with respect to $z_k$ is $\operatorname{softmax}(z)_k$ (the /calculus/ track derives it). So the gradient of the cross-entropy $\operatorname{LSE}(z) - z_y$ is $\operatorname{softmax}(z) - e_y$, where $e_y$ is the one-hot vector of the true class: "predicted minus true". The figure above shows the 2-class case: the slope of $\log(1 + e^x)$ is $\sigma(x)$.
• Accuracy. Blanchard, Higham and Higham (2021) analyse the rounding errors of the shifted formulas and show that they are accurate; they also discuss log1p for the case where one term dominates: $\operatorname{LSE}(z) = m + \operatorname{log1p}\big(\sum_{j \ne k} e^{z_j - m}\big)$.
• Sigmoid or softmax? Softmax for one class out of $n$ (the probabilities compete and sum to 1). Independent sigmoids for multi-label problems, where an image can contain a cat and a dog: each class gets its own yes/no probability, and they need not sum to 1.

Answers to section 1. (1) The same: $(101, 102, 103) = (1, 2, 3) + 100\cdot\mathbf 1$, and the softmax is shift invariant. Both give $(0.090, 0.245, 0.665)$. (2) nan in every entry: each $e^{z_k}$ is inf, and inf / inf is nan. Subtracting the max first gives the correct $(0.090, 0.245, 0.665)$. (3) $\sigma(-2) = 1 - \sigma(2) \approx 0.119$.

4. Worked example #

Take the logits $z = (1, 2, 3)$ of a 3-class classifier.

(a) Softmax, naive order.

1. Exponentials: $e^1 \approx 2.718$, $e^2 \approx 7.389$, $e^3 \approx 20.086$.
2. Sum: $s = 2.718 + 7.389 + 20.086 = 30.193$.
3. Divide: $p = (2.718, 7.389, 20.086)/30.193 \approx (0.090, 0.245, 0.665)$. The entries add up to 1.

(b) Softmax with the max trick. $m = 3$, so $z - m = (-2, -1, 0)$.

1. Exponentials: $e^{-2} \approx 0.1353$, $e^{-1} \approx 0.3679$, $e^0 = 1$.
2. Sum: $s' = 1.5032$ (at least 1, as promised).
3. Divide: $p \approx (0.0900, 0.2447, 0.6652)$: the same as in (a), because $s = e^3 s'$.

(c) Log-sum-exp, both ways. Naive: $\log 30.193 \approx 3.4076$. Stable: $m + \log s' = 3 + \log 1.5032 = 3 + 0.4076 = 3.4076$. The bounds hold: $3 \le 3.4076 \le 3 + \log 3 \approx 4.0986$.

(d) Log-softmax and cross-entropy. $\log p = z - \operatorname{LSE}(z) = (1, 2, 3) - 3.4076 = (-2.4076, -1.4076, -0.4076)$. Check: $e^{-0.4076} \approx 0.665$ ✓. If the true class is $y = 3$, the loss is $0.4076$; if it is $y = 1$, the loss is $2.4076$.

(e) Temperature. $T = 0.5$: $z / T = (2, 4, 6)$, and the same steps give $(0.016, 0.117, 0.867)$: sharper.

(f) Two classes. For logits $(2, 0)$: $\operatorname{softmax}(2, 0)_1 = e^2/(e^2 + 1) = 7.389/8.389 \approx 0.881 = \sigma(2)$. And $\operatorname{logit}(0.881) = \log(0.881/0.119) \approx 2$: back to the score.

5. In code #

NumPy checks the worked example; SciPy has the same functions. The notebook for this lesson has these cells and the exercises.

import numpy as np
from scipy.special import softmax as sp_softmax, logsumexp, expit

def softmax(z):
    z = np.asarray(z, dtype=float)
    e = np.exp(z - z.max())          # the max trick: every exponent <= 0
    return e / e.sum()

def lse(z):
    z = np.asarray(z, dtype=float)
    m = z.max()
    return m + np.log(np.sum(np.exp(z - m)))

z = np.array([1.0, 2.0, 3.0])
print(softmax(z).round(4), softmax(z + 100).round(4))   # [0.09   0.2447 0.6652] [0.09   0.2447 0.6652]
print(round(lse(z), 4), round(logsumexp(z), 4))         # 3.4076 3.4076
print((z - lse(z)).round(4))                            # [-2.4076 -1.4076 -0.4076]

big = np.array([1000.0, 1001.0, 1002.0])
with np.errstate(over="ignore", invalid="ignore"):
    print(np.exp(big) / np.exp(big).sum())              # [nan nan nan]
print(softmax(big).round(4), sp_softmax(big).round(4))  # [0.09   0.2447 0.6652] [0.09   0.2447 0.6652]

print(round(expit(2), 4), round(expit(-2), 4))          # 0.8808 0.1192
print(round(np.log(0.8 / 0.2), 4))                      # 1.3863  (logit of 0.8)
print(round(softmax([2.0, 0.0])[0], 4))                 # 0.8808  (2-class softmax = sigmoid)

6. Where it shows up #

• Logistic regression and every binary classifier: $P(y = 1 \mid x) = \sigma(w^\top x + b)$; the loss is the binary cross-entropy $-y\log\sigma(z) - (1 - y)\log(1 - \sigma(z))$, computed stably from the logit $z$ by BCEWithLogitsLoss.
• The output layer of classifiers. Image classifiers (1000 ImageNet classes) and language models (a softmax over a vocabulary of tens of thousands of tokens) end with a softmax and are trained with the cross-entropy $\operatorname{LSE}(z) - z_y$. The ML Fundamentals track and the Deep Learning track use it in every classifier.
• Attention. A transformer computes weights $\operatorname{softmax}(q^\top k_j / \sqrt{d})$ over the keys $k_j$; the $\sqrt d$ acts like a temperature (Linear Algebra lesson 15). Fast attention kernels (FlashAttention) keep a running max and a running sum of exponentials (the "online softmax"): the max trick, done block by block.
• Sampling from language models: temperature, and greedy decoding as $T \to 0$.
• Smooth maxima. Softplus $\log(1 + e^x)$ is a smooth ReLU; LSE pooling and "soft-min" losses replace a hard max by a differentiable one.
• Other places: the Gumbel-softmax trick, mixture models (weights from a softmax), the Boltzmann distribution $p_k \propto e^{-E_k / (k_B T)}$ of statistical physics (the same formula, with energies as negative logits and a physical temperature), and the softmax policies of reinforcement learning.

7. Common mistakes #

• Normalizing raw scores by their sum. $z_k / \sum_j z_j$ fails for negative scores and for a zero sum. Use the softmax.
• Naive exponentials. np.exp(z) / np.exp(z).sum() returns nan for large logits (overflow) and for very negative logits (underflow to 0/0). Subtract the max first.
• log(softmax(z)) in two steps. A probability that rounds to 0 gives -inf. Compute $z - \operatorname{LSE}(z)$.
• Passing probabilities to a loss that expects logits. torch.nn.CrossEntropyLoss applies the log-softmax itself; giving it softmax outputs applies the softmax twice and trains badly.
• Softmax over the wrong axis. For a batch of logits with shape (batch, classes), use axis=-1 (or axis=1), so each row sums to 1. With axis=0 each column sums to 1, which is wrong.
• Softmax for multi-label outputs. If several labels can be true at once, use one sigmoid per label.
• Thinking LSE equals the max. It is always larger (for $n \ge 2$), by at most $\log n$.
• Confusing "logit" and "logits". The logit is $\log\frac{p}{1-p}$; "logits" are the raw scores of a model.

8. Check yourself #

1. Compute $\sigma(0)$, $\sigma(\log 3)$ and $\operatorname{logit}(0.5)$.
   Answer

   $\sigma(0) = 1/2$. $\sigma(\log 3) = 1/(1 + e^{-\log 3}) = 1/(1 + 1/3) = 3/4$. $\operatorname{logit}(0.5) = \log(0.5/0.5) = \log 1 = 0$.
2. Compute $\operatorname{softmax}(0, \log 2, \log 3)$ by hand.
   Answer

   The exponentials are $1, 2, 3$; their sum is 6. So the softmax is $(1/6, 2/6, 3/6) = (1/6, 1/3, 1/2)$.
3. Show that $\operatorname{softmax}(z)_i / \operatorname{softmax}(z)_j = e^{z_i - z_j}$, and use it to explain shift invariance in one sentence.
   Answer

   The denominators cancel: $\dfrac{e^{z_i}/s}{e^{z_j}/s} = e^{z_i}/e^{z_j} = e^{z_i - z_j}$. All ratios depend only on differences of logits, and adding $c$ to every logit does not change any difference; since the entries also sum to 1, the whole vector stays the same.
4. Give the bounds for $\operatorname{LSE}(z)$ when $z = (0, 0, 0, 0)$, and the exact value.
   Answer

   $m = 0$ and $n = 4$, so $0 \le \operatorname{LSE} \le \log 4$. All entries are equal, so the upper bound is exact: $\log(4e^0) = \log 4 \approx 1.386$.
5. A float32 model produces the logits $(90, 85, 80)$. What happens with naive softmax? What does the stable version give?
   Answer

   In float32, $e^{90}$ overflows (the limit is about 88.72), so the first exponential is `inf` and the result contains `nan`. Stable: $z - m = (0, -5, -10)$, exponentials $(1, 0.0067, 0.0000454)$, sum $1.00678$, so $p \approx (0.9933, 0.0067, 0.0000451)$.
6. Logits $z = (2, 0)$ and true class 2. Compute the cross-entropy loss.
   Answer

   $\operatorname{LSE}(2, 0) = \log(e^2 + 1) \approx 2.1269$. The loss is $\operatorname{LSE}(z) - z_2 = 2.1269 - 0 = 2.1269$. Equivalently $-\log(1 - \sigma(2)) = -\log 0.1192 \approx 2.127$.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Write the sigmoid, and give $\sigma(0)$ and $\sigma(-z)$.

   Show answer

2. What is the inverse of the sigmoid?

   Show answer

3. Compute the logit of $p = 0.8$.

   Show answer

4. Write the softmax of $z \in \R^n$.

   Show answer

5. Why does adding the same constant $c$ to every logit not change the softmax?

   Show answer

6. What does a 2-class softmax with logits $(z, 0)$ give for class 1?

   Show answer

7. State the bounds of the log-sum-exp of $z \in \R^n$.

   Show answer

8. How do you compute log-softmax and LSE without overflow?

   Show answer

9. What does $\operatorname{softmax}(z/T)$ approach as $T \to 0$, and as $T \to \infty$?

   Show answer

10. What is the slope of the sigmoid, and where is it largest?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Why subtract the max in softmax?" — Shift invariance makes it exact; afterwards every exponent is $\le 0$ (no overflow) and the denominator is $\ge 1$ (no division by 0).
• "Why does CrossEntropyLoss take logits, not probabilities?" — It computes $\operatorname{LSE}(z) - z_y$ in one stable step; computing a softmax and then a log can give $\log 0$. Its gradient is $\operatorname{softmax}(z) - e_y$.
• "Sigmoid or softmax for the output layer?" — Softmax when exactly one class is true (multi-class); one sigmoid per class when several can be true (multi-label). For two classes they are the same model.
• "What does temperature do in sampling?" — Divides the logits by $T$; $T < 1$ sharpens the distribution toward the argmax, $T > 1$ flattens it toward uniform.
• "What is LSE, and why is it called a soft maximum?" — $\log\sum_j e^{z_j}$, within $\log n$ above the max, smooth everywhere, and its gradient is the softmax.

10. Practice #

The notebook for this lesson runs in Colab or in your browser. It has five exercises, each with a check cell and a folded solution:

1. A stable sigmoid that gives no overflow warning for $z = \pm 1000$; compare with scipy.special.expit.
2. A stable softmax(z, axis=-1) for 2-D input; test shift invariance and logits near 1000 against scipy.special.softmax.
3. logsumexp and log_softmax; check the bounds and compare with scipy.special.logsumexp.
4. The mean cross-entropy loss for a batch of logits and labels, with broadcasting and fancy indexing from lesson 03.
5. Temperature sampling with a seeded random generator: compare the empirical frequencies for $T = 0.5, 1, 2$.

11. Watch and read #

Neural Networks Part 5: ArgMax and SoftMaxStatQuest with Josh Starmer · 14 min

Softmax as a smooth replacement of argmax for the output layer of a network.

Odds and Log(Odds), Clearly Explained!!!StatQuest with Josh Starmer · 12 min

Odds and the log of the odds, which is the logit of this lesson.

StatQuest: Logistic RegressionStatQuest with Josh Starmer · 9 min

The sigmoid curve as a model of a probability, and how logistic regression uses it.

Transformers, the tech behind LLMs | Deep Learning Chapter 53Blue1Brown · Deep learning, chapter 5 · 4 min (from 22:22)

The section "Softmax with temperature": how a language model turns logits into next-token probabilities.

• Goodfellow, Bengio and Courville, Deep Learning, §4.1 Overflow and underflow: the max trick for softmax, in two pages.
• Zhang, Lipton, Li and Smola, Dive into Deep Learning, §4.1 Softmax regression: softmax, cross-entropy and its gradient, with code.
• Andrew Ng and Tengyu Ma, CS229 lecture notes, §2.1 (logistic regression) and §2.3 (multi-class classification with softmax).
• Blanchard, Higham and Higham (2021), Accurately computing the log-sum-exp and softmax functions, IMA Journal of Numerical Analysis, doi:10.1093/imanum/draa038; free preprint at MIMS EPrints 2765: the error analysis behind the max trick (advanced).