Mixed precision: formats, loss scaling and accumulation

Before this lesson: lesson 1 (tensor cores) and lesson 2 (memory-bound and compute-bound ops); floating-point numbers, $\varepsilon$ and the unit roundoff from linear algebra 13; backpropagation (deep learning 3) and Adam (optimization 6). Time: about 60 minutes, plus the notebook.

---

Why this matters #

Fewer bits per number buy two things.

• Arithmetic. The dense peaks of an H100 SXM (2022) are about 67 TFLOP/s in fp32 (CUDA cores), 495 in tf32, 989 in bf16 and 1,979 in fp8 (tensor cores). NVIDIA's product page lists twice these numbers "with sparsity"; we always use the dense numbers.
• Bytes. A 16-bit tensor is half the bytes of an fp32 tensor. Memory-bound ops (lesson 2) run about twice as fast, and stored activations take half the memory.

The price is precision and range. Every large model today is trained in mixed precision: bf16 with fp32 master weights is the default on A100 and H100 GPUs, and some models (DeepSeek-V3, 2024) run their matmuls in fp8. The same failures (long sums, small updates, tiny values) appear in scientific code that moves to GPUs.

1. Predict first #

2. The picture #

A float is a grid that gets coarser. A binary floating-point number is $x = \pm(1.f_1 f_2 \dots f_{p-1})_2 \times 2^e$. It has $p$ significant bits (the leading 1 is not stored) and an exponent $e$. The exponent bits set the range (how large and how small a number can be). The mantissa bits set the precision (how fine the grid is). Linear algebra 13 introduced these ideas. Here is the short review we need:

• machine epsilon $\varepsilon = 2^{1-p}$ is the gap between 1 and the next float;
• the unit roundoff $u = \varepsilon/2 = 2^{-p}$ bounds the relative error of one rounding: $\mathrm{fl}(x) = x(1 + \delta)$ with $|\delta| \le u$ (if there is no overflow or underflow);
• round to nearest, ties to even: a value exactly halfway between two floats goes to the one whose last bit is 0.

A toy format. Take $p = 3$. Between 1 and 2 the format has 1, 1.25, 1.5, 1.75. Between 2 and 4 it has 2, 2.5, 3, 3.5. Between 8 and 16 it has 8, 10, 12, 14. Each interval $[2^e, 2^{e+1})$ holds $2^{p-1} = 4$ evenly spaced numbers, so the gap doubles at every power of two.

Now count with this format: start at 0 and add 1 again and again. Up to 8 every integer is representable. At 8 the gap is 2, so $8 + 1 = 9$ lies exactly halfway between 8 and 10. Ties go to the even neighbour: $8 = 1.00_2 \times 2^3$ ends in 0, and $10 = 1.01_2 \times 2^3$ ends in 1. So $\mathrm{fl}(8 + 1) = 8$. The counter is stuck at $8 = 2^p$. Every later addition of 1 is lost.

Interactive figure: loading…

Two different failures. So a float can fail in two ways.

• Range. A number that is too large overflows to inf. A number that is too small underflows to 0.
• Precision. A number added to a much larger one can vanish, because it is smaller than half the gap at the larger number.

A training step meets both failures in three places: tiny gradients (range), weight updates that are tiny compared with the weights (precision), and long sums in matmuls, norms and losses (precision). The mixed-precision recipe has one fix for each.

Interactive figure: loading…

3. Definitions and main results #

The formats #

format	bits s/e/m	$p$	$\varepsilon$	largest	smallest normal	typical use
fp32	1/8/23	24	$2^{-23}$	$3.4 \cdot 10^{38}$	$2^{-126}$	master weights, optimizer states, accumulators, softmax, norms, loss
tf32	1/8/10	11	$2^{-10}$	$3.4 \cdot 10^{38}$	$2^{-126}$	tensor-core input mode for "fp32" matmuls (A100 and later)
bf16	1/8/7	8	$2^{-7}$	$3.4 \cdot 10^{38}$	$2^{-126}$	weights, activations, gradients in training
fp16	1/5/10	11	$2^{-10}$	65504	$2^{-14}$	inference; training only with loss scaling
fp8 E4M3	1/4/3	4	$2^{-3}$	448 (no inf)	$2^{-6}$	FP8 weights and activations
fp8 E5M2	1/5/2	3	$2^{-2}$	57344	$2^{-14}$	FP8 gradients

"s/e/m" counts the sign, exponent and mantissa bits, and $p$ = mantissa bits + 1. In decimals: $\varepsilon$ is about $1.2 \cdot 10^{-7}$ (fp32), $9.8 \cdot 10^{-4}$ (tf32, fp16), $7.8 \cdot 10^{-3}$ (bf16), 0.125 (E4M3) and 0.25 (E5M2). The smallest normal number is about $1.2 \cdot 10^{-38}$ for the formats with 8 exponent bits, $6.1 \cdot 10^{-5}$ for fp16 and E5M2, and 0.016 for E4M3.

Read the table in two directions:

• Same exponent bits, same range. fp32, tf32 and bf16 have 8 exponent bits, so they have the same range. bf16 is exactly the top 16 bits of an fp32 number: the same range, but only 8 significant bits (2–3 decimal digits).
• Same mantissa bits, same precision. fp16 and tf32 have 10 mantissa bits ($p = 11$). fp16 is more precise than bf16, but its range ends at 65504.

Below the smallest normal number come subnormal numbers, with fewer significant bits. The smallest positive fp16 number is $2^{-24} \approx 6 \times 10^{-8}$. A value of at most $2^{-25}$ rounds to 0 in fp16.

tf32 is not a storage format. With tf32 enabled, the tensor cores round fp32 inputs to 10 mantissa bits, multiply, and accumulate in fp32. The tensors in memory stay fp32.

Theorem (stagnation of a low-precision sum) #

Let the format have $p$ significant bits and round to nearest. Let $S > 0$ be a normal float with $2^e \le S < 2^{e+1}$, and let $\mathrm{ulp}(S) = 2^{e-p+1}$ be the gap above $S$ (the unit in the last place). If $0 \le x < \mathrm{ulp}(S)/2$, then $\mathrm{fl}(S + x) = S$.

Why this is true, in one sentence. The exact sum $S + x$ lies closer to $S$ than to the next float, so rounding returns $S$.

Proof.

1. The floats in $[2^e, 2^{e+1}]$ are $2^e + k \cdot 2^{e-p+1}$ for $k = 0, \dots, 2^{p-1}$. They are evenly spaced with gap $\mathrm{ulp}(S)$.
2. So the next float above $S$ is $S + \mathrm{ulp}(S)$.
3. $S \le S + x < S + \mathrm{ulp}(S)/2$: the exact sum is closer to $S$ than to $S + \mathrm{ulp}(S)$.
4. Round to nearest returns $S$, and $x$ is lost. $\blacksquare$

At a tie, $x = \mathrm{ulp}(S)/2$, the rounding keeps the neighbour whose last bit is 0. So $x$ is also lost when $S$ has an even last bit (as in $8 + 1$ above).

Consequence: where a sum of equal terms stops. For $2^e \le S < 2^{e+1}$ we have $2^{-p} S < \mathrm{ulp}(S) \le 2^{1-p} S$. Take terms of size $c > 0$:

• the sum cannot stop before $\mathrm{ulp}(S) \ge 2c$, that is before $S \ge 2^p c$;
• it must stop once $2^{-p} S \ge 2c$, that is at about $S = 2^{p+1} c$.

So a running sum of terms of size $c$ stops between about $2^p c$ and $2^{p+1} c$, after about $2^p$ terms: 256 terms in bf16, 2048 in fp16, $1.7 \times 10^7$ in fp32.

Three fixes for long sums #

1. fp32 accumulation. A tensor core computes $D = AB + C$ with $A, B$ in 16 (or 8) bits and $C, D$ in fp32. The product of two bf16 significands (8 bits each) has at most 16 bits, so it is exact in fp32. Only the additions round, at fp32 precision ($u = 2^{-24}$), and the output is rounded to bf16 once. The accumulator is a register, so this costs nothing. A bf16 matmul with inner dimension 4096 therefore does not stagnate: stagnation in fp32 needs about $2^{24}$ terms.
2. Pairwise summation. Sum each half recursively, then add the two results. Each input passes through only $\lceil \log_2 n \rceil$ additions instead of up to $n - 1$. GPU reductions are trees anyway, so they are close to pairwise.
3. Kahan summation. Keep a second variable $c$ for the lost part. For each $x_i$: $y = x_i - c$, $t = S + y$, $c = (t - S) - y$, $S = t$. Here $t - S$ is the part of $y$ that really entered the sum, so $c$ is minus the part that was lost, and the next step adds it back. It costs 4 operations instead of 1. Use it when the accumulator must stay in low precision (for example, bf16 weights without an fp32 copy).

Error bounds for the three methods

With $u$ the unit roundoff of the accumulator and to first order in $u$ (Higham, Accuracy and Stability of Numerical Algorithms, chapter 4), the error of a computed sum of $x_1, \dots, x_n$ is at most

• $(n - 1)\, u \sum_i |x_i|$ for the left-to-right (naive) sum;
• $\lceil \log_2 n \rceil\, u \sum_i |x_i|$ for pairwise summation;
• $(2u + O(n u^2)) \sum_i |x_i|$ for Kahan summation.

With bf16's $u = 2^{-8}$, the naive bound exceeds the sum itself once $n > 257$, which matches the stagnation after about $2^p$ terms. The problem returns one level down: DeepSeek-V3 (2024) found that the FP8 matmuls of the H800 GPU keep only about 14 bits in the accumulator. Their kernels therefore copy the partial sums to fp32 registers every 128 products.

The mixed-precision recipe #

The recipe of Micikevicius et al. (2018), as used by PyTorch's automatic mixed precision (AMP) and by Megatron-LM, has five rules.

1. fp32 master weights and optimizer states. Apply the theorem with $S = w$ and $x = \Delta w$: an update is lost when $|\Delta w|$ is below $\mathrm{ulp}(w)/2$, that is, when $|\Delta w| / |w|$ is below about $2^{-p}$. This is about $4 \times 10^{-3}$ in bf16 and $5 \times 10^{-4}$ in fp16. Adam's relative updates are often $10^{-3}$ to $10^{-5}$, so many of them would be lost.
2. 16-bit weights for the forward and backward pass. Matmuls and convolutions run on tensor cores with fp32 accumulation.
3. Range-sensitive ops and long sums in fp32. PyTorch's CUDA autocast runs, among others, softmax, log_softmax, layer_norm, group_norm, cross_entropy, mse_loss, exp, log, pow, sum, cumsum and norm in fp32. The reasons: $e^z$ overflows fp16 for $z > \ln 65504 \approx 11.1$, and norm statistics, softmax denominators and losses are long sums. Elementwise ops (add, GELU) stay in 16 bits: they are memory-bound, and each element is rounded only once.
4. Gradients. Activation gradients are 16-bit. Weight gradients are 16-bit when the step uses 16-bit weights (Megatron-LM); with PyTorch autocast and fp32 parameters they are fp32. The sum over micro-batches and the sum across GPUs often run in fp32 (Megatron's fp32 "main grad", FSDP's reduce_dtype).
5. The update runs in fp32 on the master weights. The next 16-bit copy is cast from them.

Interactive figure: loading…

Loss scaling for fp16 #

fp16 has 3 more mantissa bits than bf16, but only 5 exponent bits: its range ends near $6 \times 10^{-8}$. Micikevicius et al. (section 3.2 and figure 3) show a detector network in which many activation gradients are below $2^{-24}$ and would become 0 in fp16, while the top of the fp16 range stays unused.

The fix: multiply the loss by $S = 2^k$.

1. Compute the loss $\ell$ in fp32 and multiply it by $S$. A power of 2 only changes the exponent, so this is exact.
2. Run the backward pass on $S\ell$. By the chain rule, $\partial (S\ell)/\partial \theta = S \, \partial \ell / \partial \theta$: every gradient is multiplied by $S$, so the small ones move up into the fp16 range.
3. Right after backward, unscale in fp32: $g = g_S / S$. Do this before gradient clipping, weight decay or the update, because all of them expect the true gradient.

Dynamic loss scaling. A fixed $S$ is hard to choose: too small and gradients underflow, too large and some overflow to inf. PyTorch's torch.amp.GradScaler adapts $S$:

• start at $S = 2^{16}$;
• if any gradient is inf or NaN: skip the step (no update) and halve $S$;
• after 2000 clean steps in a row: double $S$.

So $S$ stays near the largest safe value. A skipped step costs one batch, which is cheap.

Interactive figure: loading…

Why bf16 needs no loss scaling. bf16 has the 8 exponent bits of fp32, so gradients do not underflow: $4 \cdot 10^{-9}$ is stored as about $3.99 \cdot 10^{-9}$. Its weakness is precision, and rules 1–5 handle precision. (The PyTorch documentation warns about the other direction: many models pretrained in bf16 produce values above 65504, so they overflow in fp16.)

FP8 training: scale factors #

E4M3 covers only about $2^{-9}$ to 448, and E5M2 about $2^{-16}$ to 57344. Real tensors do not fit these ranges by chance, so every FP8 tensor gets a scale factor. A tensor $x$ is stored as $q = \mathrm{fp8}(x \sigma)$ with, for example, $\sigma = 448 / \mathrm{amax}(x)$, where $\mathrm{amax}(x) = \max_i |x_i|$. A matmul of two scaled tensors accumulates in higher precision and divides the result by $\sigma_A \sigma_B$.

• The 2022 recipe (Micikevicius et al., "FP8 Formats for Deep Learning"): E4M3 for weights and activations (they need precision: 3 mantissa bits), E5M2 for gradients (they spread over many orders of magnitude: they need range), one scale per tensor.
• Delayed vs current scaling. Computing amax needs a pass over the tensor before the cast. Delayed scaling (NVIDIA Transformer Engine) takes the scale from the amax history of recent steps instead; current scaling computes it from the tensor itself.
• Fine-grained scales. DeepSeek-V3 (2024) uses E4M3 for all tensors, with one scale per $1 \times 128$ tile of activations and per $128 \times 128$ block of weights.
• No inf in E4M3. Its top codes are used for numbers and NaN. Depending on the library, an overflow gives NaN or saturates to 448 (PyTorch's cast saturates). Either way, you will not see an inf.

The master weights, the optimizer and the reductions stay in fp32, as in bf16 training.

Answers to section 1. (1) About 0.003. The bf16 sum freezes at 32 (section 4), and $32 / 10{,}000 = 0.0032$. (2) Only fp16. In fp16, $4 \cdot 10^{-9}$ is below $2^{-25} \approx 3 \cdot 10^{-8}$ and rounds to 0. bf16 stores it as $3.99 \cdot 10^{-9}$.

4. Worked example #

(a) The mean of 10,000 values of 0.1 in bf16. First, 0.1 itself is rounded: the stored input is $0.10009765625$.

1. In $[8, 16)$, $\mathrm{ulp} = 2^{3-7} = 0.0625$. The exact sum $S + 0.1001$ lies between $S + 0.0625$ and $S + 0.125$, closer to $S + 0.125$. So almost every addition contributes 0.125: 25% too much.
2. In $[16, 32)$, $\mathrm{ulp} = 2^{4-7} = 0.125$. Again each addition contributes 0.125.
3. At 32, $\mathrm{ulp} = 2^{5-7} = 0.25$ and $\mathrm{ulp}/2 = 0.125 > 0.1001$. By the theorem, every later addition is lost. The sum freezes at 32 after 276 additions.
4. Mean: $32 / 10{,}000 = 0.0032$.

For comparison: with an fp32 accumulator the sum is 1000.98 and the mean 0.1001 (the 0.1% error comes only from storing 0.1 in bf16). Pairwise summation in bf16 gives 1000. Kahan in bf16 gives 1004: the result itself must be a bf16 number, and the bf16 gap at 1000 is 4.

(b) An update that vanishes. A weight $w = 0.02$ is stored in bf16 as $0.02001953125$. It lies in $[2^{-6}, 2^{-5})$, so $\mathrm{ulp}(w) = 2^{-6-7} = 2^{-13} \approx 1.22 \times 10^{-4}$, and $\mathrm{ulp}(w)/2 \approx 6.1 \times 10^{-5}$.

• An update of $3 \times 10^{-5}$ is below $6.1 \times 10^{-5}$: it is lost.
• An update of $10^{-4}$ is above it, so it rounds to one full gap, $1.22 \times 10^{-4}$: 22% too large.
• In fp32, $\mathrm{ulp}(0.02) = 2^{-29} \approx 1.9 \times 10^{-9}$: both updates are kept.

(c) Loss scaling and back-off in fp16. Take $S = 2^{16} = 65536$.

1. A gradient $4 \cdot 10^{-9}$ becomes $2.62 \times 10^{-4}$ after scaling: a normal fp16 number. Unscaled in fp32 it is $4.0018 \times 10^{-9}$, a relative error of $4.4 \times 10^{-4} \le u = 2^{-11}$.
2. Another gradient is 2.0. Scaled, it is $131072 > 65504$: inf. GradScaler skips the step and sets $S = 2^{15}$.
3. In the next step the same gradient gives $65536$, still inf (everything from 65520 up is inf). Skip again; $S = 2^{14}$.
4. Now it gives $32768$, which fits. Training goes on after two skipped steps.

5. In code #

NumPy has fp16 but no bf16. A bf16 value is the top 16 bits of an fp32 value, so we can round to bf16 with integer operations on the bits. The notebook asks you to write this function yourself.

import numpy as np

def bf16(x):
    """Round to bf16 (nearest, ties to even); the result is a float32 holding a bf16 value."""
    u = np.asarray(x, dtype=np.float32).view(np.uint32).astype(np.uint64)
    u = (u + 0x7FFF + ((u >> 16) & 1)) & 0xFFFF0000      # keep the top 16 bits, rounded
    return u.astype(np.uint32).view(np.float32)

# 1. Stored values: 0.1 and 0.02 in bf16; 4e-9 in bf16 and in fp16
print(bf16(0.1), bf16(0.02), bf16(4e-9), np.float16(4e-9))
# 0.100097656 0.020019531 3.9872248e-09 0.0

# 2. Stagnation: a bf16 accumulator, rounded after every addition
x, S = bf16(0.1), np.float32(0)
for _ in range(10_000):
    S = bf16(S + x)
print(S, S / 10_000)                                 # 32.0 0.0032
print(np.sum(np.full(10_000, x), dtype=np.float32))  # 1000.97656 (fp32 accumulator)

# 3. An update below ulp(w)/2 is lost in bf16
w = bf16(0.02)
print(bf16(w + 3e-5) == w, bf16(w + 1e-4) - w)       # True 0.00012207031

# 4. fp16: a sum of ones stops at 2^11 = 2048
print(float(np.cumsum(np.ones(5000, np.float16), dtype=np.float16)[-1]))   # 2048.0

In PyTorch, autocast chooses the format of each op and GradScaler runs the dynamic loss scale. This CPU example shows both:

import torch

# GradScaler: skip the step and halve S on inf; double S after growth_interval clean steps
p = torch.nn.Parameter(torch.ones(3))
opt = torch.optim.SGD([p], lr=0.1)
scaler = torch.amp.GradScaler("cpu", init_scale=2.0**16, growth_interval=2)
for c in [1.0, float("inf"), 1.0, 1.0]:
    opt.zero_grad()
    scaler.scale((c * p).sum()).backward()   # every gradient is multiplied by S
    scaler.step(opt)                         # unscales; skips the step if a gradient is inf/nan
    scaler.update()
    print(c, scaler.get_scale(), round(p[0].item(), 4))
# 1.0 65536.0 0.9
# inf 32768.0 0.9    <- step skipped, S halved
# 1.0 32768.0 0.8
# 1.0 65536.0 0.7    <- two clean steps: S doubled

# autocast picks the format per op: the matmul in bf16, the loss in fp32
lin = torch.nn.Linear(16, 4)
with torch.autocast("cpu", dtype=torch.bfloat16):
    y = lin(torch.randn(8, 16))
    loss = torch.nn.functional.cross_entropy(y, torch.zeros(8, dtype=torch.long))
print(lin.weight.dtype, y.dtype, loss.dtype)   # torch.float32 torch.bfloat16 torch.float32

The CPU autocast lists differ from the CUDA lists (on the CPU, softmax stays in bf16). Section 3 gives the CUDA list. In real training, growth_interval is 2000; we use 2 to see the doubling.

6. Where it shows up #

• LLM training. bf16 autocast with fp32 master weights is the default on A100 and H100 GPUs. FSDP exposes the recipe as param_dtype (the compute format) and reduce_dtype (the format of the gradient reduction). FP8 training adds the scale factors of section 3. Inference formats come back in lesson 11.
• Memory. Mixed-precision Adam keeps 16 bytes per parameter in model states (2 for the 16-bit weights, 2 for the gradients, 4 + 4 + 4 for the fp32 master weights, $m$ and $v$). Lesson 6 counts every byte.
• TF32. torch.set_float32_matmul_precision("high") lets "fp32" matmuls use tf32 (10 mantissa bits), and cuDNN convolutions use tf32 by default. Check this setting when scientific code gives slightly different results on an A100 or H100.
• ML for science. Long sums: global means and conservation checks over millions of grid points belong in fp32. Small updates: an Euler step $x + \Delta t \, v$ of a diffusion or flow sampler with $\Delta t = 10^{-3}$ and $|v| \approx |x|$ is below $\mathrm{ulp}(x)/2$ in bf16 and is lost, so keep the sampler state in fp32. Cancellation: PDE residuals and finite differences subtract nearly equal numbers and need fp32 or fp64.

7. Common mistakes #

• "bf16 is more precise than fp16." The opposite: bf16 has range, fp16 has precision (8 against 11 significant bits).
• Clipping before unscaling. The clipping threshold is then compared with $S g$, not $g$. Call scaler.unscale_(optimizer) before clip_grad_norm_.
• bf16 weights without an fp32 master copy. Small updates vanish (section 4b), and training quietly stalls when the learning rate decays.
• Reductions in bf16. A loss, a metric or a variance summed naively in bf16 stops growing after about 256 terms.
• Taking $\varepsilon$ as the smallest number. $\varepsilon$ is the gap after 1. The smallest positive fp16 number is $2^{-24}$.
• Expecting the loss scaler to fix forward overflow. It protects only the gradients. Activations above 65504 in the forward pass are inf in fp16 whatever $S$ is.
• Waiting for an inf from E4M3. It has no inf; an overflow saturates or becomes NaN.

8. Check yourself #

1. Give $p$, $\varepsilon$ and the largest value of bf16 and fp16. Which needs loss scaling, and why?
   Answer

   bf16: $p = 8$, $\varepsilon = 2^{-7}$, largest about $3.4 \times 10^{38}$. fp16: $p = 11$, $\varepsilon = 2^{-10}$, largest 65504. fp16 needs loss scaling: its range ends near $6 \times 10^{-8}$ and many gradients are smaller. bf16 has the range of fp32.
2. A bf16 accumulator starts at 0 and adds 0.5 again and again. Where does it stop?
   Answer

   At 128. In $[128, 256)$ the gap is $2^{7-7} = 1$, so $128 + 0.5$ is a tie, and round-to-even keeps $128 = 1.0000000_2 \times 2^7$. This is $2^p c = 256 \cdot 0.5$.
3. A weight $w = 0.5$ gets an update of $10^{-5}$. Is it kept in bf16? In fp32?
   Answer

   bf16: $\mathrm{ulp}(0.5) = 2^{-1-7} = 2^{-8} \approx 3.9 \times 10^{-3}$, and half of it, $1.95 \times 10^{-3}$, is larger than $10^{-5}$: the update is lost. fp32: $\mathrm{ulp}(0.5) = 2^{-24} \approx 6 \times 10^{-8}$: it is kept.
4. Why does a bf16 tensor-core matmul with inner dimension $K = 4096$ not stagnate?
   Answer

   The products are exact in fp32 and the additions round to fp32 ($u = 2^{-24}$). Stagnation needs about $2^{24}$ terms, and the error bound $K u \approx 2.4 \times 10^{-4}$ (relative to $\sum_i |a_i b_i|$) is small. The output is rounded to bf16 only once.
5. In fp16 with $S = 2^{10}$, a gradient of 100 appears. What does GradScaler do? What would happen with bf16?
   Answer

   $100 \cdot 1024 = 102400 > 65504$, so it is inf: the step is skipped and $S$ becomes $2^9$; then $51200$ fits. With bf16 there is no loss scaling and 100 is far inside the range.
6. Why does FP8 training need a scale factor per tensor (or per block), while bf16 training does not?
   Answer

   E4M3 covers only about $2^{-9}$ to 448. Without a scale, large values overflow and small ones lose their bits. Multiplying by $\sigma = 448/\mathrm{amax}$ moves the tensor into the range. bf16 has the range of fp32, so tensors fit without a scale.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. What is the machine epsilon of a format with $p$ significant bits, and its value for bf16?

   Show answer

2. Give the sign/exponent/mantissa bits of bf16 and fp16.

   Show answer

3. What is the largest finite fp16 number?

   Show answer

4. State the stagnation condition for adding $x \ge 0$ to a float $S > 0$.

   Show answer

5. Where does a bf16 accumulator that adds 1 again and again stop?

   Show answer

6. Why does mixed-precision training keep fp32 master weights?

   Show answer

7. What is the dynamic loss-scaling rule of GradScaler?

   Show answer

8. Why does multiplying the loss by $S$ scale every gradient by $S$?

   Show answer

9. In FP8 training, which format holds the gradients, and which the weights and activations?

   Show answer

10. What does tf32 mean on an A100 or H100?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "bf16 or fp16 for training?" bf16: it has the range of fp32 and needs no loss scaling. fp16 has 3 more mantissa bits but needs loss scaling and overflows above 65504.
• "Why fp32 master weights if the matmuls run in bf16?" An update below about $2^{-8}|w|$ is lost in bf16; Adam's updates are often smaller.
• "What does a tensor core compute?" $D = AB + C$ on small tiles, with 16- or 8-bit inputs and fp32 accumulation.
• "You get NaNs after switching from bf16 to fp16. Why?" Most likely an overflow above 65504 in the forward pass (activations, attention logits), or softmax or norms not in fp32. The loss scaler protects only the gradients; it cannot fix a forward overflow.
• "Why is a metric computed in bf16 wrong?" A naive bf16 sum stops growing after about 256 terms. Accumulate in fp32.

10. Practice #

The notebook for this lesson runs in Colab, on a CPU, with NumPy only. Each exercise has a check cell and a folded solution.

1. Round to bf16 with bit operations (round to nearest, ties to even), and check it against known values.
2. Simulate a low-precision accumulator, find where a bf16 and an fp16 sum stop, and compare with the theorem.
3. Implement pairwise and Kahan summation in bf16 and compare their errors as $n$ grows.
4. Write a dynamic loss scaler and run it on a stream of synthetic gradients with rare overflows.
5. Count how many of 1000 updates of size $10^{-4}$ change a bf16 weight of 1.0, and compare with an fp32 master weight.

11. Watch and read #

Let's reproduce GPT-2 (124M)Andrej Karpathy · 20 min (from 1:28:14)

Watch the part on tensor cores, TF32, fp16 with gradient scalers and bf16 autocast, with measured speed-ups.

Floating Point Numbers - ComputerphileComputerphile · 9 min

A gentle picture of why a float has a fixed number of significant digits and a moving exponent.

Unit 9.1 | Accelerated Model Training via Mixed-Precision Training | Part 1Lightning AI (Sebastian Raschka) · 6 min

A short overview of the mixed-precision recipe and the fp32 master copy.

Fp8 Training From Hopper To Blackwell - Luca Wehrstedt, MetaPyTorch Foundation · 25 min

Advanced: how FP8 training with scale factors works in practice on H100 and newer GPUs.

• Micikevicius et al., Mixed Precision Training (ICLR 2018): section 3.1 (fp32 master copy of the weights), 3.2 (loss scaling, figure 3), 3.3 (arithmetic precision).
• Micikevicius et al., FP8 Formats for Deep Learning (2022): section 3 and table 1 (E4M3, E5M2), and the scaling factors.
• PyTorch, Automatic Mixed Precision package - torch.amp: "Gradient Scaling" and the "Autocast Op Reference" (the CUDA ops that run in float32).
• DeepSeek-AI, DeepSeek-V3 Technical Report (2024), section 3.3: FP8 training with fine-grained scales and fp32 accumulation every 128 products.
• David Goldberg, What Every Computer Scientist Should Know About Floating-Point Arithmetic (1991): rounding, ulps and relative error.