Breadth: LLM pretraining, scaling laws, fine-tuning with LoRA, RLHF, DPO and evals

Before this lesson: lesson 07 (the training step, the 16 bytes per parameter of mixed-precision Adam, BPE) and lesson 08 (the decoder and its sampling). Background: KL divergence and cross-entropy in the Information Theory track, LoRA as low rank in the Linear Algebra track, and policy gradients and PPO in the Reinforcement Learning track. A full LLM course (see the resources) treats every topic here in full depth. Time: about 90 minutes, plus the notebook.

---

Why this matters #

LLM questions are now standard breadth in ML engineer, research engineer and research scientist interviews, not a specialty. A common opening is "Walk me through how a chat model is trained." A strong answer names the stages in order, gives one equation and one trade-off per stage, and uses numbers: tokens, FLOPs, GPU-hours. A weak answer lists method names. The same questions come back in daily work and in ML for science: how large should a foundation model be for my data and my compute? Should I fine-tune on labels or with a reward from a checker? Is the new model better, or did the number only move inside its error bar?

1. Predict first #

2. The picture #

Start with the simplest model. A language model gives a probability to each possible next token. A model that knows nothing spreads the probability evenly over the vocabulary of $V$ tokens. Its loss per token is then $-\ln(1/V) = \ln V$, which is $10.8$ nats for $V = 50{,}000$. The perplexity is $\exp(\text{loss})$, here exactly $V$: the model is as unsure as a choice among $V$ equal options. Every stage of training removes uncertainty in a different way.

Stage	Data	Training signal	What it adds
Pretraining	$10^{12}$–$10^{13}$ tokens of filtered, deduplicated web text, code, books	next-token cross-entropy	knowledge, language, in-context learning
Mid-training	a smaller high-quality mix (math, code, long documents) while the learning rate decays	the same loss	quality, skills, long context
SFT	$10^4$–$10^6$ (prompt, response) demonstrations	cross-entropy on response tokens only	the format: follow instructions, chat template, tool calls
Preference tuning	pairs (chosen, rejected)	reward model + PPO, or DPO	which of the possible answers people prefer
RL with verifiable rewards	prompts with a checker (a math answer, unit tests)	reward 1 if correct, else 0	reasoning: longer, self-checking solutions
Evaluation	held-out benchmarks, human or model judges	—	the decision whether a change helped

In one line: pretraining makes the model know; SFT makes it answer in the right form; preference tuning chooses among good answers; RL with a checker makes it search for correct answers. Evaluation decides at every step.

Scaling in one picture. For a fixed compute budget, model size and data trade against each other. You can train a big model on few tokens, or a small model on many tokens. Along this line of equal compute (an iso-FLOP curve) the final loss is U-shaped. The bottom of the U is the compute-optimal model. The bottom is flat: a model 2× smaller or larger loses very little.

Interactive figure: loading…

Post-training in one picture: the KL leash. Take one prompt with two possible answers. The SFT model $\pi_{\text{ref}}$ gives each answer probability $0.5$. A reward model scores them $1$ and $0$. We want a new policy $\pi$ with a high expected reward, but every nat of KL divergence from $\pi_{\text{ref}}$ costs $\beta$. For $\beta = 1$ the best policy is $(0.73, 0.27)$ (section 3.6 derives it). As $\beta \to 0$ it puts all mass on the higher reward; as $\beta \to \infty$ it stays at $(0.5, 0.5)$. So $\beta$ is the length of the leash that ties the policy to $\pi_{\text{ref}}$.

Interactive figure: loading…

3. Definitions and main results #

Notation. $N$ is the number of parameters, $D$ the number of training tokens, $C$ the training compute in FLOPs, $V$ the vocabulary size, $T$ the sequence length. $\pi_\theta(y \mid x)$ is the probability that the model with parameters $\theta$ gives the response $y$ to the prompt $x$.

3.1 Pretraining loss, perplexity, bits per byte #

The text is a sequence of tokens $x_1, \dots, x_T$. The pretraining loss is the average negative log-probability of each token given the tokens before it:

Minimizing it is maximum likelihood. It is the cross-entropy between the text and the model. In expectation it equals the entropy rate of the text plus a KL divergence, which is never negative. So the loss cannot go below the entropy of the text.

• Perplexity is $\mathrm{PPL} = e^{\mathcal{L}}$ (with $\mathcal{L}$ in nats). It depends on the tokenizer: a tokenizer with longer tokens has fewer, harder predictions. So perplexity compares only models with the same tokenizer.
• Bits per byte divides the total loss by the number of bytes of the text: $\text{bpb} = \dfrac{T\, \mathcal{L}}{n_{\text{bytes}} \ln 2}$. It does not depend on the tokenizer, so it compares models with different tokenizers.

The data pipeline matters as much as the model: extract text from HTML, keep the wanted languages, filter by quality (rules and a classifier), remove exact and near duplicates (MinHash), choose a mixture weight for each source, and remove benchmark test items (decontamination).

3.2 Compute: $C \approx 6ND$ #

Claim. Training a dense transformer with $N$ parameters on $D$ tokens costs about $C \approx 6ND$ floating-point operations (FLOPs).

1. Almost all FLOPs are products with weight matrices. A weight matrix of size $m \times n$ applied to one token's vector costs $mn$ multiplications and $mn$ additions: $2mn$ FLOPs. Summed over all matrices: $2N$ FLOPs per token in the forward pass.
2. For $y = Wx$ the backward pass computes two products of the same size: the gradient for the input, $W^\top \frac{\partial \mathcal{L}}{\partial y}$ ($2mn$ FLOPs), and the gradient for the weights, $\frac{\partial \mathcal{L}}{\partial y}\, x^\top$ ($2mn$ FLOPs). So $4N$ per token.
3. Total $6N$ per token, so $C \approx 6ND$. $\blacksquare$

What the count leaves out: the attention scores and their weighted sums (Kaplan et al. show that this extra cost is small while the context length is below about $12\, d_{\text{model}}$), activation recomputation (it repeats the forward pass: $8N$ instead of $6N$ per token), and the embedding lookup (no matrix product). Inference costs about $2N$ FLOPs per generated token.

From FLOPs to time. Time $= C / (n_{\text{GPU}} \cdot F \cdot \text{MFU})$, with $F$ the peak FLOP/s of one GPU. MFU (model FLOPs utilization) is the fraction of the peak that the model's FLOPs really use. Large tuned runs reach about 40%: the Llama 3 report gives 38–43% bf16 MFU.

Memory. Mixed-precision training with Adam stores about 16 bytes per parameter: bf16 weights (2) and gradients (2), plus fp32 master weights, first moment and second moment (12). This is the count of the ZeRO paper (lesson 07). Activations come on top. A 7 B model needs $16 \cdot 7 \cdot 10^9 = 112$ GB for these states alone, more than one 80 GB GPU, so the states must be sharded (lesson 12).

3.3 Scaling laws #

Kaplan et al. (2020) found smooth power laws of the loss in $N$, $D$ and $C$. Their compute-optimal model size grew as $N_{\text{opt}} \propto C^{0.73}$: "make models bigger, and train them on relatively few tokens".

Chinchilla (Hoffmann et al., 2022) fitted a different form:

$E$ is the irreducible loss (the entropy of text), $A/N^\alpha$ is the cost of a finite model, and $B/D^\beta$ is the cost of finite data. (Here $\beta$ is an exponent; it is not the $\beta$ of RLHF.)

Theorem (compute-optimal allocation). Minimize $L(N, D)$ subject to $6ND = C$. The minimum is at

Why this is true, in one sentence. Growing $N$ lowers the model term but, at fixed compute, shrinks $D$ and raises the data term; the best point balances the two rates of change.

1. Substitute $D = C/(6N)$: $\ L(N) = E + A N^{-\alpha} + B (C/6)^{-\beta} N^{\beta}$.
2. Set the derivative to zero: $-\alpha A N^{-\alpha-1} + \beta B (C/6)^{-\beta} N^{\beta-1} = 0$.
3. Multiply by $N$ and use $(C/6)^{-\beta} N^{\beta} = D^{-\beta}$: $\ \alpha A N^{-\alpha} = \beta B D^{-\beta}$. At the optimum the model term and the data term are in the fixed ratio $\beta : \alpha$.
4. From step 2, $N^{\alpha+\beta} = \frac{\alpha A}{\beta B} (C/6)^{\beta}$. Take the power $1/(\alpha+\beta)$ to get $N_{\text{opt}}$; then $D_{\text{opt}} = (C/6)/N_{\text{opt}}$ gives the second formula.
5. It is a minimum: with $u = \ln N$, the two terms $A e^{-\alpha u}$ and $B (C/6)^{-\beta} e^{\beta u}$ are convex in $u$, so their sum has exactly one stationary point, and it is the minimum. $\blacksquare$

Because $\alpha \approx \beta$, both exponents are near $0.5$: grow $N$ and $D$ by the same factor. The paper's three methods give $a = 0.46$–$0.50$; the rounded constants above give $a = 0.45$. The practical rule is about 20 tokens per parameter. Chinchilla itself (70 B parameters, 1.4 T tokens) used about the compute of Gopher (280 B parameters, 300 B tokens) and beat it. With $D = 20N$ we get $C = 6N \cdot 20N = 120 N^2$, so $N = \sqrt{C/120}$.

Why Kaplan differs (Chinchilla §2): Kaplan used one learning-rate schedule and one token count for all runs. The losses of shorter runs were read from checkpoints of a long cosine schedule, so they were too high. This undervalued data. Most of Kaplan's models were also small.

Why practice "over-trains". Chinchilla minimizes training compute only. Serving costs $2N$ FLOPs per generated token, for the whole life of the model. So a smaller model trained on many more tokens is cheaper overall. Llama 3 8B saw about 15 T tokens: about 1,900 tokens per parameter. The flat bottom of the iso-FLOP curve makes this cheap (section 4).

3.4 Supervised fine-tuning (SFT) #

Each example is a prompt $x$ and a response $y$, written into one token sequence $z$ by a chat template (special tokens mark the roles and the end of a turn). With a mask $m_t = 1$ on response tokens and $m_t = 0$ on prompt tokens:

Trade-offs. SFT is cheap, stable, and uses the same code as pretraining. But it only imitates. The model cannot beat its demonstrations, and it never sees what not to do. Demonstrations of facts the model does not know can teach it to make things up (Gekhman et al., 2024).

3.5 LoRA: fine-tuning a low-rank correction #

Full fine-tuning changes a weight matrix $W_0$ into $W_0 + \Delta W$, and it needs gradients and Adam states for every weight. LoRA (Hu et al., 2021) assumes that a useful $\Delta W$ has low rank.

Definition. A frozen pretrained weight $W_0 \in \mathbb{R}^{d \times k}$ maps an input $x \in \mathbb{R}^k$ to $W_0 x \in \mathbb{R}^d$. LoRA adds a trainable low-rank path:

The input is squeezed to $r$ numbers ($Ax$) and expanded back ($B(Ax)$). The update $\Delta W = sBA$ is a sum of $r$ rank-1 matrices, so its rank is at most $r$ (Linear Algebra lesson 15).

• Parameters: $r(d + k)$ instead of $dk$. For $d = k = 4096$ and $r = 8$: $65{,}536$ instead of $16{,}777{,}216$, which is $0.39\%$.
• Initialization: $A$ random (Gaussian), $B = 0$. So $\Delta W = 0$, and training starts exactly at the pretrained model.
• Scale: $\alpha/r$ lets you change $r$ without retuning the learning rate much. (rsLoRA uses $\alpha/\sqrt{r}$ for large $r$.)
• Merge for inference: $W = W_0 + sBA$ is one matrix again, so a merged LoRA adds no latency.
• In code, compute $(xA^\top)B^\top$ for a batch of row vectors. Never build the $d \times k$ matrix $BA$ in the training forward pass.

Interactive figure: loading…

Theorem (LoRA gradients). Let $u = Ax \in \mathbb{R}^r$ and let $g = \partial \mathcal{L} / \partial h \in \mathbb{R}^d$ be the gradient that arrives from the layers above. Then

1. $h = W_0 x + s B u$. $W_0$ is frozen, so only the term $sBu$ depends on $A$ and $B$.
2. For fixed $u$, $h$ is linear in $B$: $\partial h_i / \partial B_{ij} = s\, u_j$. So $\partial \mathcal{L} / \partial B_{ij} = s\, g_i u_j$, which is the matrix $s\, g u^\top$.
3. The gradient with respect to $u$ is $\partial \mathcal{L} / \partial u = s B^\top g$ (the transpose of the linear map $u \mapsto sBu$, applied to $g$).
4. $u = Ax$ is linear in $A$, so by the argument of step 2, $\partial \mathcal{L} / \partial A = (\partial \mathcal{L} / \partial u)\, x^\top = s B^\top g\, x^\top$. $\blacksquare$

Consequences for the initialization.

• With $B = 0$: $\partial \mathcal{L} / \partial A = 0$, but $\partial \mathcal{L} / \partial B = s\, g (Ax)^\top \ne 0$. So $B$ moves first. After one step $B \ne 0$, and then $A$ moves too.
• With $A = B = 0$: both gradients are zero at every step. Nothing trains.
• With both random: $\Delta W \ne 0$ at step 0. Training starts from a changed, worse model.
• $A = 0$ with $B$ random also works (the mirror case). What matters is $\Delta W = 0$ with not both factors zero.

What LoRA saves, and what it does not.

• Saves: the gradients and optimizer states of $W_0$. Full fine-tuning with Adam needs about 16 bytes per parameter. LoRA needs the frozen weights (2 bytes per parameter in bf16) plus about 16 bytes per adapter parameter. Hu et al. report 1.2 TB → 350 GB of GPU memory for GPT-3 175B, and a checkpoint of 35 MB instead of 350 GB ($r = 4$, query and value projections only).
• Does not save: activations. Backpropagation still needs the stored inputs of every adapted layer, and the gradient must still flow through every frozen $W_0$ to the layers below. The backward pass skips only the weight gradient of $W_0$ (the paper reports a 25% speed-up).
• Capacity: each adapted matrix can change only by a matrix of rank at most $r$. For a large domain shift (a new language, continued pretraining) LoRA can fall short of full fine-tuning; it also forgets less of what the base model knew (Biderman et al., 2024).
• QLoRA (Dettmers et al., 2023) stores the frozen base in 4-bit NF4 and puts adapters on all linear layers; this fine-tunes a 65 B model on one 48 GB GPU.

3.6 Reward model and RLHF with PPO #

Reward model. Labelers choose the better of two responses: $y_w$ (winner) and $y_l$ (loser). The Bradley–Terry model says

with $\sigma$ the logistic function and $r_\phi$ a scalar head on an LLM. The loss is $-\log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))$: logistic regression on the difference of two scores. Only differences matter, so $r_\phi$ is fixed only up to a constant for each prompt.

RL objective. With $\pi_{\text{ref}}$ the SFT model and $\beta > 0$:

Theorem (optimal policy). For each prompt $x$ the maximizer is

Why this is true, in one sentence. The objective is, up to a constant, minus $\beta$ times the KL divergence from $\pi$ to $\pi^*$, and a KL divergence is smallest (zero) when the two distributions are equal.

1. Fix $x$ and drop it from the notation. The objective is $J(\pi) = \sum_y \pi(y) r(y) - \beta \sum_y \pi(y) \log \frac{\pi(y)}{\pi_{\text{ref}}(y)}$.
2. Take the log of the definition of $\pi^*$ and solve for the reward: $r(y) = \beta \log \frac{\pi^*(y)}{\pi_{\text{ref}}(y)} + \beta \log Z$.
3. Substitute this into $J$. The two $\pi_{\text{ref}}$ terms cancel, and $\sum_y \pi(y) = 1$: $J(\pi) = \sum_y \pi(y)\, \beta \log \frac{\pi^*(y)}{\pi(y)} + \beta \log Z = -\beta\, \mathrm{KL}(\pi \,\|\, \pi^*) + \beta \log Z$.
4. $Z$ does not depend on $\pi$, and $\mathrm{KL}(\pi \,\|\, \pi^*) \ge 0$ with equality only at $\pi = \pi^*$. So $\pi^*$ is the unique maximizer, and the maximum value is $\beta \log Z$. $\blacksquare$

In the toy of section 2: $\pi^* \propto (0.5 e^{1}, 0.5 e^{0})$, so $\pi^* = (0.731, 0.269)$.

PPO in practice (InstructGPT): sample responses from the policy, score them with $r_\phi$, add a KL penalty per token, estimate advantages with a value model and GAE, and update with the clipped probability ratio. Four models are in memory: policy, reference, reward model, value model.

Trade-offs. PPO learns from its own samples, so it can improve beyond the data. But it is expensive (generation inside the training loop), has many hyperparameters, and can be unstable. And the reward model is only a proxy. Gao et al. (2022) showed that the true reward first rises and then falls as the policy moves further (in KL) from $\pi_{\text{ref}}$. This is reward over-optimization (also called reward hacking).

3.7 DPO: the reward model inside the policy #

Step 2 of the proof writes the reward through the optimal policy. DPO (Rafailov et al., 2023) puts this expression into the Bradley–Terry model:

1. $r(x, y_w) - r(x, y_l) = \beta \log \frac{\pi^*(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi^*(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}$. The term $\beta \log Z(x)$ cancels, because both responses answer the same prompt $x$.
2. Replace the unknown $\pi^*$ by the trained policy $\pi_\theta$, and minimize the Bradley–Terry loss on the preference pairs:

The implicit reward is $\hat r_\theta(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$. The gradient is

It raises the log-probability of the winner, lowers that of the loser, and the weight $\sigma(\hat r_l - \hat r_w)$ is large when the model still ranks the pair wrongly.

DPO needs two models (the policy and the frozen reference), no sampling, and the training loop of SFT. Trade-offs: it is offline. It learns only from the fixed pairs, often sampled from other models, so it cannot explore or get feedback on its own new answers. It can overfit the pairs, and it can lower the likelihood of both responses. Variants: IPO (a squared loss against overfitting), KTO (unpaired thumbs up or down; still uses $\pi_{\text{ref}}$), SimPO and ORPO (no reference model).

3.8 RL with verifiable rewards and GRPO #

For math and code, a program can check the answer. The reward is $1$ if the final answer matches or the unit tests pass, else $0$ (sometimes plus a small format reward). GRPO (DeepSeekMath, 2024) samples a group of $G$ responses for each prompt and gives every token of response $i$ the advantage

The group mean replaces the value model as the baseline: an answer is good if it beats the other answers to the same prompt. The update is a PPO-style clipped ratio per token, plus a KL term to $\pi_{\text{ref}}$. DeepSeek-R1-Zero ran this on a base model with only accuracy and format rewards; its responses grew longer, and it learned to re-check its own steps.

Interactive figure: loading…

Trade-offs. It works only where a checker exists. Generation dominates the cost. A group where all answers get the same reward has zero advantage and gives no signal (DAPO filters such groups and samples new prompts). The checker can be gamed. Variants: Dr. GRPO removes the length and std normalizations, because they bias the update; DAPO drops the KL term. It is still debated whether this RL teaches new abilities or mainly makes the model sample its best answers more often (higher pass@1, similar pass@$k$ for large $k$).

Other tools of the same stack: rejection-sampling fine-tuning (sample many answers, keep the best, run SFT on them), distillation (SFT on a stronger model's outputs), and RLAIF (AI feedback instead of human labels).

Method	Data	Models in memory	Online	Strength	Main failure mode
SFT	demonstrations	1	no	simple, stable; sets the format	imitation only; no negative signal
RLHF (PPO)	pairs → reward model; prompts	4: policy, reference, reward, value	yes	can go beyond the data; any learned reward	cost, instability, reward hacking
DPO	pairs	2: policy, reference	no	as simple as SFT; no reward model	off-policy; overfits; no exploration
GRPO + checker	prompts + a checker	policy + reference (1 without the KL term)	yes	strong for math and code reasoning	only checkable tasks; zero-signal groups

3.9 Evaluation #

• Perplexity and bits per byte measure language-modeling quality, not usefulness. Compare perplexity only between models with the same tokenizer.
• Benchmarks: multiple choice (MMLU), scored by the log-likelihood of each option or by the generated letter; generative tasks (GSM8K, HumanEval), scored by a checker. An accuracy $\hat p$ on $n$ independent questions has standard error $\sqrt{\hat p(1-\hat p)/n}$.
• pass@$k$ (Chen et al., 2021) is the probability that at least one of $k$ samples is correct.
• LLM-as-judge is cheap and scales, but it prefers the first position, longer answers, and its own outputs (Zheng et al., 2023). Mitigate: swap the order and average, control for length, use a judge from a different model family, and check agreement with human labels.
• Contamination: test items inside the training data inflate scores. Detect it with $n$-gram overlap (GPT-3 used 13-grams), report results on a clean subset, and prefer new or private test sets.

Theorem (unbiased pass@$k$). Draw $n \ge k$ independent samples for one problem, and let $c$ of them be correct. Then

is unbiased for $\text{pass@}k = 1 - (1-p)^k$, where $p$ is the probability that one sample is correct. The plug-in estimate $1 - (1 - c/n)^k$ is biased low.

1. $\binom{n-c}{k}/\binom{n}{k}$ is the probability that a random subset of $k$ of the $n$ samples contains no correct sample.
2. Average this over the random samples. A random $k$-subset of $n$ independent samples is itself $k$ independent samples, so its chance to be all wrong is $(1-p)^k$. Hence $\mathbb{E}\big[\binom{n-c}{k}/\binom{n}{k}\big] = (1-p)^k$, and the estimator is unbiased.
3. The plug-in uses $f(\hat p) = 1 - (1 - \hat p)^k$ with $\hat p = c/n$. The function $f$ is concave, so by Jensen's inequality $\mathbb{E}[f(\hat p)] \le f(\mathbb{E}\hat p) = f(p)$: it is too low on average. $\blacksquare$

Interactive figure: loading…

Answers to section 1. (1) It is 2 tokens per parameter, far from about 20. The budget is $C = 6 \cdot 10^{10} \cdot 2 \cdot 10^{10} = 1.2 \cdot 10^{21}$ FLOPs. With $D = 20N$: $N = \sqrt{C/120} = \sqrt{10^{19}} \approx 3.2$ B and $D \approx 63$ B tokens. The Chinchilla fit gives a loss of $2.388$ for the plan and $2.318$ for the 20-tokens choice (its own minimum, $2.311$, is at 2.0 B parameters on 101 B tokens). If the model will serve many tokens, go even smaller and train longer. (2) DPO learns only from a fixed set of pairs. Online RL samples from the current policy and gets a reward for exactly those samples. So it can explore, it can improve beyond the data, and it can use a checker. The price is cost and instability.

4. Worked example #

A. Plan a small pretraining run. The budget is 8 H100 GPUs for 24 hours. Peak dense bf16 throughput is $F = 989$ TFLOP/s per GPU. Assume 40% MFU, so each GPU gives $0.4 \cdot 989 \cdot 10^{12} = 3.96 \cdot 10^{14}$ useful FLOP/s.

1. Compute: $C = 8 \cdot 86{,}400\ \text{s} \cdot 3.96 \cdot 10^{14}\ \text{FLOP/s} = 2.73 \cdot 10^{20}$ FLOPs.
2. Chinchilla rule: $N = \sqrt{C/120} = \sqrt{2.28 \cdot 10^{18}} = 1.5 \cdot 10^9$ parameters, and $D = 20N = 30$ B tokens.
3. Memory: $16 \cdot 1.5 \cdot 10^9 = 24$ GB for weights, gradients and Adam states. It fits on one 80 GB GPU with room for activations, so plain data parallelism works.
4. Serving cost: $2N = 3.0$ GFLOP per generated token.
5. Over-trained option: $N = 0.5$ B on $D = C/(6N) = 91$ B tokens (182 tokens per parameter). With the Chinchilla fit, the loss rises only from $2.475$ to $2.489$ nats ($+0.6\%$), and every generated token costs 3× less.

B. One DPO pair. $\beta = 0.1$. The sequence log-probabilities are $\log \pi_\theta(y_w \mid x) = -10$, $\log \pi_{\text{ref}}(y_w \mid x) = -12$, $\log \pi_\theta(y_l \mid x) = -15$, $\log \pi_{\text{ref}}(y_l \mid x) = -14$.

1. Implicit rewards: $\hat r_w = 0.1 \cdot (-10 + 12) = 0.2$ and $\hat r_l = 0.1 \cdot (-15 + 14) = -0.1$.
2. Margin: $\hat r_w - \hat r_l = 0.3$.
3. Loss: $-\log \sigma(0.3) = -\log 0.574 = 0.554$.
4. Gradient weight: $\sigma(\hat r_l - \hat r_w) = \sigma(-0.3) = 0.426$. The pair still gets a large update, because the policy separates the two responses only a little.

C. One GRPO group. $G = 8$ answers to one math prompt, with rewards $(1, 0, 0, 1, 1, 0, 0, 0)$.

1. Mean: $3/8 = 0.375$.
2. Standard deviation (population, divide by $G$): $\sqrt{0.375 \cdot 0.625} = 0.484$. (For 0/1 rewards with mean $\bar r$, the variance is $\bar r(1 - \bar r)$.)
3. Advantages: $(1 - 0.375)/0.484 = +1.29$ for each correct answer and $(0 - 0.375)/0.484 = -0.77$ for each wrong one.
4. If all 8 were correct, every $r_i - \text{mean}$ would be $0$: no signal. Implementations add a small $\varepsilon$ to the std to avoid $0/0$. Some libraries divide by $G - 1$ in the std; this multiplies all advantages by the same factor $\sqrt{(G-1)/G}$.

D. Is model B better? Model A scores 71.0% and model B 72.5% on the same 1,000 questions.

1. Each standard error is about $\sqrt{0.72 \cdot 0.28 / 1000} = 1.4$ points.
2. If we treat the two scores as independent, the difference has standard error $\sqrt{1.43^2 + 1.41^2} = 2.0$ points, so the 95% interval is $1.5 \pm 3.9$ points. Not significant.
3. A paired test on the same questions is tighter, because it removes the variation in question difficulty. Report the interval either way.

E. pass@$k$. $n = 10$ samples, $c = 3$ correct.

1. pass@1 $= 1 - \binom{7}{1}/\binom{10}{1} = 1 - 7/10 = 0.30$.
2. pass@5 $= 1 - \binom{7}{5}/\binom{10}{5} = 1 - 21/252 = 0.917$.
3. The plug-in gives $1 - 0.7^5 = 0.832$: too low.

F. LoRA on a 7 B model. A Llama-2-7B-shaped model has 32 layers, width $d = 4096$ and full multi-head attention, so $W_q, W_k, W_v, W_o$ are all $4096 \times 4096$. LoRA with $r = 8$ on these four:

1. Per matrix: $8 \cdot (4096 + 4096) = 65{,}536$. Per layer: $4 \cdot 65{,}536 = 262{,}144$.
2. All layers: $32 \cdot 262{,}144 = 8{,}388{,}608$ trainable parameters, about $0.12\%$ of 7 B.
3. Their training states: $16 \cdot 8.39\text{M} \approx 134$ MB. The frozen base: $2 \cdot 7\text{B} = 14$ GB in bf16.
4. Full fine-tuning would need $16 \cdot 7\text{B} = 112$ GB for the same states.

5. In code #

First, a check of $2N$ forward and $4N$ backward FLOPs per token, with PyTorch's FLOP counter on a stack of bias-free linear layers (as in modern LLMs):

import torch
from torch.utils.flop_counter import FlopCounterMode

torch.manual_seed(0)
d, T = 64, 128                       # width, tokens in the batch
model = torch.nn.Sequential(         # 4 bias-free linear layers
    torch.nn.Linear(d, 4 * d, bias=False), torch.nn.GELU(),
    torch.nn.Linear(4 * d, d, bias=False),
    torch.nn.Linear(d, 4 * d, bias=False), torch.nn.GELU(),
    torch.nn.Linear(4 * d, d, bias=False),
)
N = sum(p.numel() for p in model.parameters())
x = torch.randn(T, d, requires_grad=True)   # as for a layer deep inside a network

with FlopCounterMode(display=False) as fwd:
    y = model(x)
with FlopCounterMode(display=False) as bwd:
    y.square().mean().backward()

print(N, fwd.get_total_flops() / (N * T), bwd.get_total_flops() / (N * T))   # 65536 2.0 4.0

With requires_grad=False on x, the backward ratio drops to 3.5: the first layer needs no gradient for its input.

Second, the closed form of section 3.3 against a numerical minimizer, and the other numbers of section 4, in NumPy:

import numpy as np
from math import comb
from scipy.optimize import minimize_scalar

E, A, B, alpha, beta = 1.69, 406.4, 410.7, 0.34, 0.28   # Hoffmann et al. (2022), approach 3
C = 2.73e20                                              # the budget of section 4A

def loss_on_isoflop(logN):                               # D is fixed by C = 6ND
    N = np.exp(logN)
    return E + A / N**alpha + B / (C / (6 * N))**beta

res = minimize_scalar(loss_on_isoflop, bounds=(np.log(1e7), np.log(1e12)), method="bounded")
G = (alpha * A / (beta * B)) ** (1 / (alpha + beta))
print(f"{np.exp(res.x):.3e} {G * (C / 6) ** (beta / (alpha + beta)):.3e}")
# 1.016e+09 1.016e+09
for N in [0.5e9, 1e9, 1.5e9]:
    print(f"N = {N:.1e}  D/N = {C / (6 * N) / N:5.1f}  L = {loss_on_isoflop(np.log(N)):.3f}")
# N = 5.0e+08  D/N = 182.3  L = 2.489
# N = 1.0e+09  D/N =  45.6  L = 2.470
# N = 1.5e+09  D/N =  20.3  L = 2.475

p = np.array([0.5, 0.5]) * np.exp(np.array([1.0, 0.0]) / 1.0)   # KL leash, beta = 1
print((p / p.sum()).round(3))                                      # [0.731 0.269]

sig = lambda z: 1 / (1 + np.exp(-z))                               # DPO pair of section 4B
rw, rl = 0.1 * (-10 + 12), 0.1 * (-15 + 14)
print(round(-np.log(sig(rw - rl)), 3), round(sig(rl - rw), 3))     # 0.554 0.426

r = np.array([1, 0, 0, 1, 1, 0, 0, 0], float)                      # GRPO group of section 4C
print(((r - r.mean()) / r.std()).round(2))
# [ 1.29 -0.77 -0.77  1.29  1.29 -0.77 -0.77 -0.77]

n, c, k = 10, 3, 5                                                 # pass@k of section 4E
print(round(1 - comb(n - c, k) / comb(n, k), 3), round(1 - (1 - c / n) ** k, 3))
# 0.917 0.832

The fit puts the optimum at $1.0$ B parameters (45 tokens per parameter); the 20-tokens rule gives $1.5$ B. Inside a factor of 3 in $N$, the loss changes by less than 1%. This flatness is why the exact ratio matters less than the inference cost.

6. Where it shows up #

• LLM work in industry. Every team that adapts an open model chooses SFT, DPO or RL by the data it has: demonstrations, preference pairs, or a checker. LoRA adapters let one server keep one base model and many small task adapters (lesson 13).
• Scientific foundation models. The same accounting sizes a weather or Earth-system model: a "token" is a patch of a field at one time step, and $6ND$ still counts the matrix FLOPs. The rule of 20 tokens per parameter was measured on web text; measure your own exponents with small runs before you spend the big budget. LoRA adapts a pretrained weather or protein model to a new region, variable or small assay data set, with a few MB per task.
• Rewards from physics. A simulator or a conservation check is a verifiable reward, so the ideas of section 3.8 carry over to generative models of physical fields. So does the risk: a field can conserve mass and still be wrong.
• Reading papers. Ask: is the gain larger than about 2 standard errors? Is the benchmark contaminated? Is perplexity compared across tokenizers?

7. Common mistakes #

• Treating 20 tokens per parameter as a law. It is the training-compute optimum of one fit on one data mix. Inference cost and data quality move the best choice.
• Forgetting what $6ND$ leaves out. Attention FLOPs at long context, recomputation, and the gap between peak and achieved FLOP/s (MFU).
• Comparing perplexities of models with different tokenizers. Use bits per byte.
• Computing the SFT loss on prompt tokens. The model then spends capacity on predicting user prompts. Mask them.
• Initializing both LoRA factors to zero (nothing trains) or both at random (the wrong starting model). Forgetting to freeze $W_0$ also removes all the memory savings.
• Saying "DPO has no reference model". It has one (frozen). SimPO and ORPO remove it.
• Saying "the reward went up, so the model is better". The reward model is a proxy. Watch the KL and a held-out gold evaluation.
• Using $1 - (1 - c/n)^k$ for pass@$k$, and reporting differences smaller than their standard error.

8. Check yourself #

1. How many H100 GPU-days does it take to train a 7 B model on 2 T tokens at 40% MFU?
   Answer

   $C = 6 \cdot 7 \cdot 10^9 \cdot 2 \cdot 10^{12} = 8.4 \cdot 10^{22}$ FLOPs. One GPU gives $0.4 \cdot 989 \cdot 10^{12} = 3.96 \cdot 10^{14}$ FLOP/s, so $2.12 \cdot 10^8$ GPU-seconds, about $2{,}460$ GPU-days (about 10 days on 256 GPUs).
2. Why is the KL penalty in RLHF needed, and what happens as $\beta \to 0$?
   Answer

   It keeps the policy near $\pi_{\text{ref}}$, where the reward model was trained and is reliable, and it keeps the text fluent. As $\beta \to 0$ the optimum $\pi^* \propto \pi_{\text{ref}}\, e^{r/\beta}$ puts all mass on the highest proxy reward: reward hacking.
3. In DPO, why does the partition function $Z(x)$ disappear?
   Answer

   The reward is $\beta \log(\pi^*/\pi_{\text{ref}}) + \beta \log Z(x)$. Bradley–Terry uses only the difference of the rewards of two responses to the same prompt, so $\beta \log Z(x)$ cancels.
4. A GRPO group of 4 answers is all correct. What is the policy-gradient signal from it, and what can you do?
   Answer

   Zero: every $r_i - \text{mean} = 0$, so all advantages are 0 (only the KL term acts). Filter such prompts and sample new ones (DAPO's dynamic sampling), or choose prompts of medium difficulty for the current model.
5. A $4096 \times 4096$ projection gets LoRA with rank 16. How many trainable parameters, and what fraction of the matrix?
   Answer

   $16 \cdot (4096 + 4096) = 131{,}072$, which is $131{,}072 / 16{,}777{,}216 = 0.78\%$.
6. When would you choose DPO over PPO, and when the other way round?
   Answer

   DPO: you have good preference pairs, limited compute, and want stability and simple code. PPO or GRPO: you need on-policy improvement beyond the data, you have a reliable reward model or a checker (math, code), and you can pay for generation in the loop.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. About how many FLOPs does it take to train a dense transformer with $N$ parameters on $D$ tokens, and why that factor?

   Show answer

2. What is the loss and the perplexity of a model that predicts uniformly over $V$ tokens?

   Show answer

3. What is the Chinchilla rule for a compute-optimal model?

   Show answer

4. Why do labs train far past 20 tokens per parameter?

   Show answer

5. What is the optimal policy of $\max_\pi \mathbb{E}_\pi[r] - \beta\, \mathrm{KL}(\pi \| \pi_{\text{ref}})$?

   Show answer

6. Write the DPO loss for one pair $(y_w, y_l)$.

   Show answer

7. What is the GRPO advantage of answer $i$ in a group of $G$ answers?

   Show answer

8. How many trainable parameters does LoRA of rank $r$ add to a $d \times k$ matrix?

   Show answer

9. Why does LoRA start with $B = 0$ and $A$ random, and not both zero?

   Show answer

10. What does pass@$k$ measure, and what is its unbiased estimator from $n$ samples with $c$ correct?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

Recognize the question from the wording.

The question says	Start with
"how is a chat model trained", "walk me through the pipeline"	the stage table of section 2; one equation and one number per stage
"how big a model for this budget", "how long will it train"	$C \approx 6ND$, $N \approx \sqrt{C/120}$, time $= C/(n_{\text{GPU}} F\, \text{MFU})$
"Kaplan vs Chinchilla", "why does Llama train so long"	the schedule flaw; inference cost; the flat iso-FLOP curve
"derive DPO", "why no reward model"	the optimal-policy proof, then $\log Z$ cancels
"we only have a unit-test checker", "reasoning"	RL with verifiable rewards, GRPO advantage, zero-signal groups
"fine-tune on one GPU", "many customers, one base model"	LoRA (and QLoRA): $r(d + k)$ parameters, memory, merge
"is model B better?"	standard error, paired test, contamination, judge biases

The routine to say aloud for any breadth question (60–90 seconds): give the one-sentence answer first; then one equation or one number; then one trade-off ("it costs …, it fails when …"); then stop and let the interviewer choose the follow-up.

Typical questions and short model answers.

1. "Walk me through how a chat model is trained." — "Pretraining on about $10^{13}$ tokens with the next-token loss; that gives knowledge. Then mid-training on a cleaner mix while the learning rate decays. Then SFT on prompt–response pairs, with the loss only on response tokens; that gives the format. Then preference tuning, with a reward model and PPO or with DPO; that picks the answers people prefer. For math and code, RL with a checker, for example GRPO, which improves reasoning. Compute is about $6ND$, so a 7 B model on 2 T tokens is about $8 \cdot 10^{22}$ FLOPs."
2. "Chinchilla vs Kaplan. Why does Llama train far past 20 tokens per parameter?" — "Chinchilla fitted $L = E + A/N^\alpha + B/D^\beta$ and found that at fixed compute $N$ and $D$ should grow together, about 20 tokens per parameter. Kaplan used one schedule for all runs, so short runs looked worse than they were, and data was undervalued. Llama trains longer because Chinchilla counts only training compute. Inference costs $2N$ per token forever, and the loss is flat near the optimum, so a smaller model trained longer is cheaper overall."
3. "Derive DPO." — "Start from $\max_\pi \mathbb{E}[r] - \beta\,\mathrm{KL}(\pi\|\pi_{\text{ref}})$. Its solution is $\pi^* \propto \pi_{\text{ref}} e^{r/\beta}$, because the objective equals $-\beta\,\mathrm{KL}(\pi\|\pi^*)$ plus a constant. Invert it: $r = \beta \log(\pi^*/\pi_{\text{ref}}) + \beta\log Z$. Put this into Bradley–Terry; $\log Z$ cancels because both answers share the prompt. Replace $\pi^*$ by $\pi_\theta$ and minimize the logistic loss on the pairs."
4. "Why does GRPO need no value model? What can go wrong?" — "It samples a group of answers per prompt and uses the group mean as the baseline, normalized by the group std. That removes the value model and its memory. Problems: a group where all answers get the same reward gives zero signal; the std and length normalizations bias the update (Dr. GRPO removes them); and the checker can be gamed."
5. "How do you know model B is better than model A?" — "Report the standard error: on 1,000 questions at 70% it is 1.4 points, so a 1.5-point gain is not significant. Use a paired test on the same questions, several seeds, and a check for contamination. If a model judges, swap the order and control for length."
6. "LoRA or full fine-tuning?" — "LoRA trains $r(d+k)$ parameters per matrix, so it saves the gradients and Adam states, gives small checkpoints, and lets one base serve many adapters; it also forgets less. It does not save activations. For a large domain shift, full fine-tuning has more capacity."

Typical follow-ups. "What does $6ND$ leave out?" (attention at long context, recomputation, MFU). "What happens if $\beta$ is too small?" (reward hacking; the KL grows). "Why does DPO lower the likelihood of both answers sometimes?" (it only needs the margin to grow). "What is pass@$k$ and why not $1 - (1-\hat p)^k$?" (biased low, by Jensen). "Where do you put LoRA adapters?" (all linear layers when quality matters, as in QLoRA).

10. Practice #

The notebook for this lesson runs in Colab, in NumPy. Each exercise has a check cell and a folded solution:

1. chinchilla_optimum(C): the closed form of section 3.3, checked against a grid search on the iso-FLOP curve.
2. kl_regularized_policy(pi_ref, r, beta): the optimal policy, checked to beat many random policies on the objective.
3. dpo_loss and its gradient with respect to the two policy log-probabilities, checked by finite differences.
4. grpo_advantages(rewards): the group-normalized advantages, with the zero-signal case.
5. pass_at_k(n, c, k): the unbiased estimator, checked against a simulation.

The notebook also has a LoRA layer in NumPy: the output equals the base at the start, the gradients follow the theorem, and the merged weight gives the same output.

Timed practice (25 minutes each, talking aloud): derive $C \approx 6ND$ on a blank page; derive the compute-optimal $N_{\text{opt}}$ and $D_{\text{opt}}$; derive $\pi^* \propto \pi_{\text{ref}}\, e^{r/\beta}$ and the DPO loss; implement a LoRA linear layer with merge and unmerge; implement the DPO loss for a batch of sequence log-probabilities.

Say it aloud drill. Start a 2-minute timer and answer "walk me through how a chat model is trained" aloud. Record yourself and compare with the stage table: did you give one equation and one number per stage? Then do the same for "derive DPO" in 3 minutes.

11. Watch and read #

[1hr Talk] Intro to Large Language ModelsAndrej Karpathy · 60 min

The pipeline from pretraining to the assistant, and scaling, at the level of a clear interview answer.

Deep Dive into LLMs like ChatGPTAndrej Karpathy · 3 h 31 min

Pretraining data and tokenization, SFT, and RL including RLHF and reasoning models, step by step.

Direct Preference Optimization (DPO) explained: Bradley-Terry model, log probabilities, mathUmar Jamil · 49 min

The Bradley–Terry model, the optimal KL-regularized policy, and the DPO derivation of section 3.7.

What is Low-Rank Adaptation (LoRA) | explained by the inventorEdward Hu · 7 min

Why a low-rank update is enough, and what LoRA saves, from the first author.

[GRPO Explained] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language ModelsYannic Kilcher · 1 h 9 min

The DeepSeekMath paper, with GRPO and its group baseline explained.

• Stanford CS336 (Spring 2025), Lecture 9: Scaling laws 1 (video on YouTube; embedding is disabled): iso-FLOP curves and the Chinchilla analysis.
• Hoffmann et al. (2022), Training Compute-Optimal Large Language Models: §3 (the three methods) and the fit of approach 3.
• Besiroglu et al. (2024), Chinchilla Scaling: A replication attempt: the re-fit of the parametric loss.
• Kaplan et al. (2020), Scaling Laws for Neural Language Models: §2.1 (the FLOP count).
• Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models: §4 (the method and the initialization).
• Ouyang et al. (2022), Training language models to follow instructions with human feedback (InstructGPT): SFT, the reward model and PPO.
• Rafailov et al. (2023), Direct Preference Optimization: §4 and Appendix A.1 (the derivation of sections 3.6–3.7).
• Shao et al. (2024), DeepSeekMath: §4.1 (GRPO).
• Chen et al. (2021), Evaluating Large Language Models Trained on Code: §2.1 (pass@$k$ and its unbiased estimator).
• Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: the biases of model judges.