Quantization for training and inference

Before this lesson: lesson 2 (the roofline and the matmul intensity) and lesson 5 (number formats, rounding, FP8 scale factors). For the $\Delta^2/12$ result you need the expectation of a continuous random variable (probability 2). The model view of quantization (GPTQ and AWQ in more detail, KV-cache quantization) will be in the LLMs track (in preparation). Time: about 70 minutes, plus the notebook.

---

Why this matters #

• Memory. A 70B-parameter model in bf16 has 140 GB of weights: two 80 GB GPUs just to hold them. In int4 with group scales (4.125 bits per weight, section 3) it needs about 36 GB.
• Bytes per step. LLM decoding reads every weight once per step (lesson 2). A 7B model at batch 1 needs at least $14\ \text{GB} / 3.35\ \text{TB/s} = 4.2$ ms per token in bf16 on an H100 SXM (2022), and about 1.1 ms in int4.
• Arithmetic. H100 tensor cores do 1,979 dense TFLOP/s in fp8 (and as many TOP/s in int8), twice the 989 of bf16.

So quantization buys bytes or FLOPs, and which one you need depends on the phase. Every large LLM deployment uses it: int4 weight-only models (GPTQ, AWQ) on single GPUs and laptops, int8 and fp8 in high-throughput serving, and fp8 matmuls in training (DeepSeek-V3).

1. Predict first #

2. The picture #

A grid on the number line. Put grid points at equal spacing $\Delta$ (the scale, or step size), with 0 exactly on a grid point. With $b$ bits there are at most $2^b$ grid points. To quantize a value, replace it by the nearest grid point. Values beyond the last grid point are replaced by the last grid point: this is clipping.

Two errors.

• Rounding error. Inside the grid, a value moves at most half a step: the error is at most $\Delta/2$. A larger $\Delta$ means a larger rounding error.
• Clipping error. Outside the grid, a value moves to the end of the grid. Its error is its distance to the end, with no bound.

With a fixed number of grid points, a wider range needs a larger $\Delta$. So the range is a trade-off: wide enough to clip little, narrow enough to keep $\Delta$ small. When many values share one scale and we choose the range to cover all of them, the largest value sets $\Delta$ for every other value.

Interactive figure: loading…

3. Definitions and main results #

Notation. In this lesson $\Delta$ is the quantization scale (the step size) and $z$ the zero-point. Papers also write $s$ for the scale; we keep $s_j$ for SmoothQuant's smoothing factors.

Uniform affine quantization #

Choose $b$ bits, an integer grid $[q_{\min}, q_{\max}]$, a scale $\Delta > 0$ and an integer zero-point $z$. Then

The first map is quantize (real to integer), the second dequantize (integer back to real). Here round is round to nearest, ties to even (as in NumPy and PyTorch), and $\mathrm{clamp}(v, a, b) = \min(\max(v, a), b)$.

• Symmetric: $z = 0$ and a signed grid. With the restricted range $[-(2^{b-1} - 1),\ 2^{b-1} - 1]$, that is $[-127, 127]$ for int8 and $[-7, 7]$ for int4, the grid is symmetric around 0, and $\Delta = \max|x| / q_{\max}$.
• Asymmetric: an unsigned grid $[0, 2^b - 1]$, a range $[\min(x, 0), \max(x, 0)]$, $\Delta = (\max - \min)/(2^b - 1)$ and $z = \mathrm{round}(-\min/\Delta)$. Zero is exactly representable ($q = z$), so zero padding and ReLU zeros cost no error. An asymmetric grid fits one-sided data (such as ReLU outputs) better.

Why weights are usually symmetric. Write one output of a quantized matmul as $\sum_l \Delta_x (q^x_l - z_x)\,\Delta_w (q^w_l - z_w)$. Expanding the product gives four sums. The term $z_w \sum_l q^x_l$ depends on the input, so it must be computed at run time for every token. The term $z_x \sum_l q^w_l$ depends only on the weights, so it is computed once, offline (Nagel et al., 2021, section 2.4). So the usual choice is symmetric weights ($z_w = 0$) and symmetric or asymmetric activations.

Theorem (rounding error bound) #

If $x$ lies in the clipping range $[\Delta(q_{\min} - z),\ \Delta(q_{\max} - z)]$, then $|x - \hat x| \le \Delta/2$.

Proof.

1. Let $v = x/\Delta$ and $q' = \mathrm{round}(v) + z$, the code before the clamp. Rounding moves a number by at most $1/2$: $|\mathrm{round}(v) - v| \le 1/2$. So the integer $q'$ is within $1/2$ of $v + z$.
2. $x$ in the range means $v + z \in [q_{\min}, q_{\max}]$. The ends are integers, so $q'$ cannot leave $[q_{\min}, q_{\max}]$. (If $q' \le q_{\min} - 1$, then $v + z \le q' + 1/2 \le q_{\min} - 1/2$, which is outside the range; the same argument works at the top.) So the clamp does nothing: $q = q'$.
3. $\hat x - x = \Delta(q' - z) - \Delta v = \Delta\,(\mathrm{round}(v) - v)$.
4. So $|\hat x - x| \le \Delta/2$. $\blacksquare$

Outside the range the error is the clipping error, which has no bound. With asymmetric min-max calibration, rounding $z$ shifts the grid by up to $\Delta/2$, so one end value may be clipped, but by at most $\Delta/2$.

Theorem (mean squared rounding error) #

The bound $\Delta/2$ is the worst case. The typical error is smaller.

Assumption. The rounding error $e = x - \hat x$ is uniformly distributed on $[-\Delta/2, \Delta/2]$. This is a good model when $x$ is not clipped and its density changes little over one step, that is, when the data spread over many steps.

Result. Under this assumption, $\mathbb{E}[e] = 0$ and

So the root-mean-square (RMS) error is $\Delta/\sqrt{12} \approx 0.29\,\Delta$.

Why this is true. Inside one step, $x$ is spread almost evenly, so its distance to the grid point is spread evenly too.

Proof.

1. The density of $e$ is $1/\Delta$ on $[-\Delta/2, \Delta/2]$ and 0 elsewhere.
2. The mean is zero, because the integrand is odd:

3. The second moment:

When the assumption fails. With very few levels (2–3 bits) or with many exact zeros, the error is not uniform. Values that sit on grid points (such as ReLU zeros with an asymmetric grid) have no error at all. Check it numerically: for $10^5$ Gaussian values and $\Delta = 0.1$, the measured mean squared error is $8.33 \times 10^{-4}$, and $\Delta^2/12 = 8.33 \times 10^{-4}$.

The clipping trade-off #

Take symmetric quantization with range $[-c, c]$ and $\Delta = c / q_{\max}$. The mean squared error has two parts:

Here $\mathbb{E}[\,\cdot\,;\ |x| > c]$ means the expectation over the clipped values only. The rounding part grows like $c^2$; the clipping part falls quickly as $c$ passes the bulk of the data. The sum has a minimum. Min-max calibration ($c = \max|x|$) never clips the calibration data, but its $c$ is set by the most extreme sample. Percentile or MSE-optimal ranges clip a few values to get a smaller $\Delta$ (Nagel et al., 2021, section 3.1).

Interactive figure: loading…

With $2 \times 10^5$ samples of unit variance, the MSE-optimal ranges are about $2.5\sigma$ (Gaussian) and $3.5\sigma$ (Laplace) at 4 bits, and about $4.0\sigma$ and $6.9\sigma$ at 8 bits. More bits make $\Delta$ smaller, so the rounding part matters less and the best range moves out. Heavy tails also push it out.

Granularity: which values share a scale #

Write a linear layer as $Y = XW$ with $X \in \mathbb{R}^{n \times k}$ ($n$ tokens, $k$ input channels) and $W \in \mathbb{R}^{k \times m}$ ($m$ output channels). The sum runs over the reduction dimension $l = 1, \dots, k$.

• Per-tensor: one scale for all of $X$, one for all of $W$.
• Per-channel: one scale $w_j$ per output column $j$ of $W$; for activations, one scale $a_i$ per token (row $i$ of $X$).
• Group-wise: one scale per group of $g$ consecutive input elements ($g$ = 32 to 128) in each column of $W$.

Which scales an integer matmul can use. With $X_{il} \approx a_i q_{il}$ and $W_{lj} \approx w_j p_{lj}$:

Scales that depend only on $i$ or $j$ (the outer dimensions) move out of the sum. The kernel runs an int8 × int8 matmul with int32 accumulation, then does one multiplication per output. A scale that depends on $l$ (the reduction dimension), such as a per-input-channel activation scale or a group-wise weight scale, changes every term of the sum. It must be applied inside the loop over $l$, once per group:

• weight-only kernels (W4A16) do this: they dequantize each group to bf16 in registers;
• DeepSeek-V3's FP8 training does it every 128 elements, on the CUDA cores;
• microscaling formats (blocks of 32 values with a shared power-of-two scale) do it in hardware on NVIDIA Blackwell GPUs (2024).

Memory cost of scales. int4 with one fp16 scale per group of 128 costs $4 + 16/128 = 4.125$ bits per weight.

Interactive figure: loading…

Outliers and SmoothQuant #

Activation outliers. Dettmers et al. (LLM.int8(), 2022) found that from about 6.7B parameters on, LLM activations contain systematic outlier features: a few hidden channels with very large values, in almost all layers and for most tokens. SmoothQuant (Xiao et al., 2022) reports outliers about 100× larger than most activation values. They sit in fixed channels. With one scale per tensor or per token, the outlier sets $\Delta$: a channel with maximum $m_l$ in a tensor with maximum $m$ gets only about $2^8 m_l / m$ of the levels. Weights are flat and well-behaved, so int8 and even int4 weights work well.

Per-input-channel activation scales would fix this, but they sit on the reduction dimension. LLM.int8() instead runs the outlier channels in fp16 and the rest in int8. SmoothQuant moves the problem into the weights. For any vector $s \in \mathbb{R}^k$ with positive entries,

exactly, because $\mathrm{diag}(s)^{-1}\mathrm{diag}(s) = I$. Column $j$ of $X$ is divided by $s_j$, and row $j$ of $W$ is multiplied by $s_j$. The paper chooses

so that

where $X_j$ is column $j$ of $X$, $W_j$ is row $j$ of $W$, and $\alpha \in [0, 1]$ is the migration strength.

Proof of the maxima. Dividing column $j$ of $X$ by $s_j$ divides its maximum by $s_j$, and multiplying row $j$ of $W$ by $s_j$ multiplies its maximum by $s_j$:

At $\alpha = 0.5$ both maxima equal $\sqrt{\max|X_j| \max|W_j|}$. A larger $\alpha$ moves more of the difficulty to the weights (the paper uses 0.5 for most models and 0.75 for GLM-130B). The maxima of $X$ come from calibration data. The cost at run time is zero: $\mathrm{diag}(s)^{-1}$ is folded into the scale and bias of the LayerNorm before the layer, and $\mathrm{diag}(s)$ into $W$, offline. The Q, K and V projections read the same LayerNorm output, so they share one $s$.

Interactive figure: loading…

PTQ and QAT #

• PTQ (post-training quantization) takes a trained model. Weight scales come from the weights; activation ranges come from a small calibration set (a few hundred samples, no labels). It takes minutes to hours. For LLM weights there are three common methods:
  • round to nearest (RTN), as in this lesson;
  • GPTQ (Frantar et al., 2022) quantizes the weights of a layer one input row at a time, and after each row updates the rows not yet quantized to compensate the error. It uses the inverse of the Hessian $2X^\top X$ of the layer's squared error $\|XW - X\hat W\|^2$;
  • AWQ (Lin et al., 2023) observes that about 1% of the input channels, found from the activation magnitudes, matter most. It scales these channels of $W$ up before rounding and the activations down by the same factor: the identity of SmoothQuant, used for weight-only quantization.
• QAT (quantization-aware training) inserts fake quantization (quantize, then dequantize) in the forward pass and trains. The model learns weights that work well after rounding. The difficulty: round has zero derivative almost everywhere, so no gradient flows. The straight-through estimator (STE) uses $\partial \hat x / \partial x = 1$ inside the clipping range and 0 outside (Nagel et al., 2021, section 4.1). Scales can be learned too (LSQ). QAT is worth its training cost at 4 bits or less, or when PTQ loses too much accuracy.
• FP8 training (lesson 5) is quantization with floating-point grids: E4M3 for weights and activations, E5M2 for gradients, a scale per tensor or block, fp32 accumulation and fp32 master weights.

Integer grid or float grid? An int8 grid has an absolute error bound $\Delta/2$ for every value. An fp8 grid has a relative bound for normal numbers ($2^{-4}$ for E4M3, $2^{-3}$ for E5M2: the unit roundoff of lesson 5). So with fp8 a small value keeps its relative precision next to a large one, and fp8 tolerates outliers better with simple per-tensor scales.

Why the STE gradient is 1 inside and 0 outside

Fake quantization is $\hat x = \Delta \cdot \mathrm{clamp}(\mathrm{round}(x/\Delta), q_{\min}, q_{\max})$. It is a staircase: piecewise constant, with jumps at the half-steps. Its true derivative is 0 everywhere except at the jumps, where it does not exist. The STE replaces the staircase by the straight line $\hat x = x$ for the backward pass only, as long as $x$ is inside the range, where the staircase follows the line within $\Delta/2$. Outside the range, the clamp is really flat: increasing $x$ does not change $\hat x$. So the STE keeps the gradient 0 there. A consequence: weights or activations that drift outside the range stop learning, so set good initial ranges before QAT. With very few levels, latent weights can also oscillate around a rounding boundary; the notebook shows a small example.

Which phase gets faster #

For a weight matmul with $B$ tokens and a $D \times F$ weight, the FLOPs are $2BDF$. In bf16 the weight bytes are $2DF$, so the intensity is about $B$ FLOP/byte when $B \ll D, F$ (lesson 2). The H100 ridge point is $R = 989/3.35 \approx 295$.

• W4A16 (int4 weights, bf16 activations and bf16 matmuls): 4× fewer weight bytes, the same bf16 FLOP/s. The intensity becomes about $4B$. Decode at small $B$ is memory-bound, so it gets up to 4× faster. The matmul becomes compute-bound at $4B = R$, that is $B \approx 74$. From there the gain is only $R/B$, and there is no gain at $B \ge 295$ (prefill, large batches).
• W8A8 (int8 or fp8 weights and activations): 2× fewer weight bytes and 2× the peak FLOP/s. Up to 2× faster in both regimes.
• With ideal kernels, W4A16 is faster below $B \approx 148$ (where its compute time $2B/P$ per weight equals the W8A8 memory time $1/\mathrm{BW}$ per weight) and W8A8 above. Real kernels lose some speed to dequantization and to reading the scales.

Interactive figure: loading…

Answers to section 1. (1) Prefill: no. It is compute-bound and still runs bf16 matmuls; W4A16 removes bytes, not FLOPs. Decode at batch 1: yes, up to about 4× (3.9× with group scales), because it is limited by the weight bytes. (2) $\Delta = 40/127 \approx 0.315$, so $[-0.8, 0.8]$ maps to $q \in \{-3, \dots, 3\}$: only 7 of the 255 levels.

4. Worked example #

(a) Three quantizers. $x = (-0.7, -0.15, 0, 0.4, 2.0)$.

• Symmetric int8. $\Delta = 2.0/127 = 0.015748$. Then $x/\Delta = (-44.45, -9.525, 0, 25.4, 127)$, so $q = (-44, -10, 0, 25, 127)$ and $\hat x = (-0.6929, -0.1575, 0, 0.3937, 2.0)$. Largest error $0.0075 \le \Delta/2 = 0.0079$.
• Asymmetric uint8. Range $[-0.7, 2.0]$, $\Delta = 2.7/255 = 0.010588$, $z = \mathrm{round}(0.7/0.010588) = \mathrm{round}(66.11) = 66$. Then $q = \mathrm{round}(x/\Delta) + 66 = (0, 52, 66, 104, 255)$ and $\hat x = (-0.6988, -0.1482, 0, 0.4024, 2.0012)$. Largest error $0.0024 \le \Delta/2 = 0.0053$. The grid ends at 2.0012, not 2.0: the rounded $z$ shifted it.
• Symmetric int4. $\Delta = 2/7 = 0.2857$, $x/\Delta = (-2.45, -0.525, 0, 1.4, 7)$, so $q = (-2, -1, 0, 1, 7)$ and $\hat x = (-0.571, -0.286, 0, 0.286, 2.0)$. Largest error $0.136 \le \Delta/2 = 0.143$.

(b) An outlier channel and SmoothQuant. Two tokens, two channels, $\alpha = 0.5$:

1. Per-tensor int8 of $X$. $\Delta = 40/127 = 0.315$. Channel 0 becomes $q = (3, -2)$ and $\hat x = (0.945, -0.630)$: errors of 18% and 26%.
2. Smoothing factors. $\max|X_0| = 0.8$, $\max|W_0| = 0.4$ (row 0 of $W$), so $s_0 = \sqrt{0.8/0.4} = 1.414$. $\max|X_1| = 40$, $\max|W_1| = 0.1$, so $s_1 = \sqrt{40/0.1} = 20$.
3. Smoothed matrices. Divide the columns of $X$ by $s$ and multiply the rows of $W$ by $s$:

The channel maxima are $\sqrt{0.32} = 0.566$ and $\sqrt{4} = 2$ in both matrices, and $\hat X \hat W = XW$ exactly. 4. W8A8 (per-tensor $X$, per-output-column $W$): the largest error of $Y$ falls from 0.042 to 0.0093, and the relative (Frobenius) error from 1.1% to 0.2%.

(c) The roofline for a 7B model on one H100.

	weights	decode, batch 1 (≥ bytes / 3.35 TB/s)	prefill, 4,096 tokens ($2 \cdot 7\text{B} \cdot 4096 = 5.7 \times 10^{13}$ FLOPs)
bf16	14 GB	4.2 ms	58 ms at 989 TFLOP/s
W4A16 (4.125 bits)	3.6 GB	1.1 ms	58 ms (bf16 matmuls)
W8A8 / FP8	7 GB	2.1 ms	29 ms at 1,979 TFLOP/s

5. In code #

NumPy checks the worked example, the $\Delta^2/12$ result and SmoothQuant. The notebook has these cells and the exercises.

import numpy as np

def quantize(x, D, z, qmin, qmax):
    return np.clip(np.round(x / D) + z, qmin, qmax)       # np.round: ties to even

def dequantize(q, D, z):
    return D * (q - z)

# 1. The three quantizers of the worked example
x = np.array([-0.7, -0.15, 0.0, 0.4, 2.0])
for name, D, z, qmin, qmax in [("int8 sym", 2.0 / 127, 0, -127, 127),
                               ("uint8 asym", 2.7 / 255, 66, 0, 255),
                               ("int4 sym", 2.0 / 7, 0, -7, 7)]:
    q = quantize(x, D, z, qmin, qmax)
    err = np.abs(x - dequantize(q, D, z)).max()
    print(f"{name:10s} q = {q.astype(int)}  max error {err:.4f} <= D/2 = {D / 2:.4f}")
# int8 sym   q = [-44 -10   0  25 127]  max error 0.0075 <= D/2 = 0.0079
# uint8 asym q = [  0  52  66 104 255]  max error 0.0024 <= D/2 = 0.0053
# int4 sym   q = [-2 -1  0  1  7]  max error 0.1357 <= D/2 = 0.1429

# 2. Mean squared rounding error is about D^2/12 when the values span many steps
v = np.random.default_rng(0).standard_normal(100_000)
D = 0.1
print(np.mean((v - D * np.round(v / D))**2), D**2 / 12)  # 0.000833 0.000833

# 3. SmoothQuant on the worked example (alpha = 0.5)
X = np.array([[0.8, -30.0], [-0.5, 40.0]])
W = np.array([[0.4, -0.3], [0.1, 0.05]])
s = np.sqrt(np.abs(X).max(axis=0) / np.abs(W).max(axis=1))
Xs, Ws = X / s, W * s[:, None]
print(s, np.allclose(Xs @ Ws, X @ W))                    # [ 1.41421356 20.        ] True

def w8a8(X, W):
    dx = np.abs(X).max() / 127                             # one scale for X
    dw = np.abs(W).max(axis=0) / 127                       # one scale per output column of W
    return (dx * np.round(X / dx)) @ (dw * np.round(W / dw))

for A, B in [(X, W), (Xs, Ws)]:
    E = w8a8(A, B) - X @ W
    print(f"max error {np.abs(E).max():.4f}, relative {np.linalg.norm(E) / np.linalg.norm(X @ W):.4f}")
# max error 0.0423, relative 0.0110
# max error 0.0093, relative 0.0020

In PyTorch, fake_quantize_per_tensor_affine quantizes and dequantizes in one op, and its gradient is the straight-through estimator:

import torch

x = torch.tensor([-3.0, -0.5, 0.2, 0.9, 2.5], requires_grad=True)
D = 2.0 / 127                                  # symmetric int8, clipping range [-2, 2]
y = torch.fake_quantize_per_tensor_affine(x, D, 0, -127, 127)
y.sum().backward()
print(y.detach())     # tensor([-2.0000, -0.5039,  0.2047,  0.8976,  2.0000]): -3 and 2.5 are clipped
print(x.grad)         # tensor([0., 1., 1., 1., 0.]): 1 inside the range, 0 outside

6. Where it shows up #

• LLM serving. W4A16 (GPTQ, AWQ) for small batches and single-GPU or laptop deployment; W8A8 or FP8 for high-throughput serving; an int8 or fp8 KV cache when the cache dominates memory (lesson 12).
• Training. FP8 matmuls on H100 and newer GPUs (DeepSeek-V3 was trained with FP8 matmuls); QAT for models that must run in int8 or int4 on phones and other devices.
• Fine-tuning. QLoRA stores the frozen base model in 4 bits and trains small bf16 adapters, so a large model fits on one GPU.
• ML for science. Weather, climate and seismic surrogate models run in large ensembles, so the bytes and FLOPs per sample matter. But in science the tails are often the events you care about (extreme rainfall, peak ground acceleration). A range set by the bulk clips them; a range set by them makes the step coarse for the bulk. Check the error on the extremes, not only the mean.

7. Common mistakes #

• Expecting weight-only int4 to speed up prefill or training. It removes bytes, not FLOPs. Compute-bound phases do not get faster.
• Per-input-channel activation scales in an int8 matmul. They sit on the reduction dimension and cannot be applied after the integer sum.
• Min-max calibration of activations with outliers. Use percentile or MSE-optimal ranges, or SmoothQuant.
• Forgetting the scale overhead. Group size 128 with fp16 scales is 4.125 bits per weight, not 4.
• Rounding mismatch. A simulation that rounds ties away from zero and a kernel that rounds ties to even give different codes on ties.
• Ignoring the STE's zero gradient outside the range. Clipped values stop learning; set good initial ranges before QAT.
• Taking $\Delta/2$ as the typical error. It is the worst case; the RMS error is about $\Delta/\sqrt{12} \approx 0.29\,\Delta$.

8. Check yourself #

1. Symmetric int4 $[-7, 7]$ of $w = (0.33, -1.4, 0.05)$: scale, codes and largest error?
   Answer

   $\Delta = 1.4/7 = 0.2$; $w/\Delta = (1.65, -7, 0.25)$, so $q = (2, -7, 0)$ and $\hat w = (0.4, -1.4, 0)$. The largest error is $0.07 \le \Delta/2 = 0.1$.
2. int4 with group size 64, with one fp16 scale and one fp16 zero-point per group: how many bits per weight, and how much memory for the weights of a 70B model?
   Answer

   $4 + 32/64 = 4.5$ bits per weight; $70 \times 10^9 \cdot 4.5 / 8 \approx 39.4$ GB.
3. Why does the bound $|x - \hat x| \le \Delta/2$ need "$x$ inside the range"?
   Answer

   Outside the range the clamp replaces the code by $q_{\min}$ or $q_{\max}$, so $\hat x$ is the end of the grid, and the error is the distance from $x$ to that end, which can be arbitrarily large.
4. 4-bit symmetric quantization of Gaussian data: why is the MSE-optimal range smaller than $\max|x|$, and what happens to it at 8 bits?
   Answer

   At 4 bits the step $c/7$ is large, so the rounding part $\Delta^2/12$ dominates; clipping a few tail values (about $2.5\sigma$) makes $\Delta$ much smaller and lowers the total. At 8 bits $\Delta = c/127$ is small, so rounding matters less, and the best range moves out to about $4\sigma$.
5. Why does SmoothQuant cost nothing at inference time?
   Answer

   $\mathrm{diag}(s)^{-1}$ is folded into the scale and bias of the LayerNorm before the layer, and $\mathrm{diag}(s)$ is multiplied into $W$ offline. The model runs the same ops as before, with different numbers.
6. What gradient does the STE give to a value outside the clipping range, and why?
   Answer

   Zero: the clamp is constant there, so $\hat x$ does not change when $x$ changes. Inside the range the rounding is treated as the identity, so the gradient is 1.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Write the uniform affine quantize and dequantize maps.

   Show answer

2. What is the largest rounding error inside the clipping range?

   Show answer

3. Under which assumption is the mean squared rounding error $\Delta^2/12$?

   Show answer

4. Compute the symmetric int8 scale for $\max|x| = 1.27$.

   Show answer

5. Why are weights usually quantized symmetrically?

   Show answer

6. Which scales can an int8 × int8 matmul apply after the integer sum?

   Show answer

7. How many bits per weight does int4 with one fp16 scale per group of 128 cost?

   Show answer

8. State the SmoothQuant identity and the choice of $s_j$.

   Show answer

9. What gradient does the straight-through estimator use for fake quantization?

   Show answer

10. Does W4A16 speed up the prefill of a long prompt? Why?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Decode is too slow at batch 1. What do you quantize?" The weights (int4 or int8 weight-only): decode is memory-bound on the weight bytes. Check the batch against the ridge point.
• "int8 or fp8 for an inference deployment?" Both give 8-bit tensor cores. fp8 has a relative error and tolerates outliers with simple per-tensor scales; int8 activations need SmoothQuant-like handling. fp8 tensor cores need an H100 (Hopper), an Ada GPU or newer; the A100 has int8 but no fp8.
• "Explain SmoothQuant in one sentence." Divide each activation channel by $s_j$ and multiply the matching weight row by $s_j$: the product does not change, the outliers move into the weights, and the factors are folded into the previous layer offline.
• "PTQ or QAT?" PTQ first (cheap, calibration data only); QAT when 4 bits or less lose too much accuracy.
• "Why is $\Delta^2/12$ the quantization noise?" If the error is uniform on one step, its variance is $\frac{1}{\Delta}\int_{-\Delta/2}^{\Delta/2} e^2\,de = \Delta^2/12$.

10. Practice #

The notebook for this lesson runs in Colab, on a CPU, with NumPy only. Each exercise has a check cell and a folded solution.

1. Write quantize and dequantize for symmetric and asymmetric int8 and int4, and check the error bound $\Delta/2$ and the $\Delta^2/12$ result.
2. Compute the MSE against the clipping range and find the best range for Gaussian and Laplace data.
3. Compare per-tensor, per-row, per-column and group-wise scales on a matrix with an outlier channel.
4. Implement SmoothQuant and plot the W8A8 error of $XW$ against $\alpha$.
5. Train a small linear model with fake-quantized weights and the straight-through estimator, and compare it with post-training quantization.

11. Watch and read #

EfficientML.ai Lecture 5 - Quantization (Part I) (MIT 6.5940, Fall 2023)MIT HAN Lab (Song Han) · 1 h 15 min

Number formats, linear (affine) quantization with scale and zero-point, and how an integer matmul is computed.

EfficientML.ai Lecture 6 - Quantization (Part II) (MIT 6.5940, Fall 2023)MIT HAN Lab (Song Han) · 1 h 15 min

Granularity, clipping, post-training quantization and quantization-aware training with the straight-through estimator.

Quantization explained with PyTorch - Post-Training Quantization, Quantization-Aware TrainingUmar Jamil · 51 min

Symmetric and asymmetric quantization worked out in code, calibration, and QAT in PyTorch.

• Nagel et al., A White Paper on Neural Network Quantization (2021): section 2.2 (uniform affine and symmetric quantization), 2.4 (granularity, symmetric vs asymmetric), 3.1 (range setting), 4.1 (the straight-through estimator).
• Xiao et al., SmoothQuant (2022): section 3 (why activations are hard), section 4 (the migration strength $\alpha$).
• Dettmers et al., LLM.int8() (2022): emergent outlier features and mixed-precision decomposition.
• Frantar et al., GPTQ (2022) and Lin et al., AWQ (2023): weight-only post-training quantization of LLMs.
• Micikevicius et al., FP8 Formats for Deep Learning (2022): E4M3, E5M2 and scaling.