Interactive figures
Every figure in the track. Each one is a small experiment: drag, slide, and watch the numbers. The link under each figure takes you to the lesson that explains it.
Lesson 1 · GPU architecture
Click a level (grid, block, warp, thread, SM) and read what it is. Change and the block size and watch the counts. Blue marks work, violet marks idle hardware: idle threads in the last block, and idle SMs in the last wave. Try with blocks of 1,024: most SMs stay idle in the last wave.
Increase the number of warps from 1 and watch the bottom row, the scheduler. Each warp issues instructions (blue) for cycles, then waits for memory (orange) for cycles. The scheduler is idle (violet) when no warp is ready. With one warp the scheduler is idle almost all the time. Find the number of warps where the idle gaps disappear, and compare it with the formula in the panel.
Move the three sliders (threads per block , registers per thread , shared memory per block in KB) and watch which bar is shortest: that resource sets the number of resident blocks (it turns magenta). Start from the example above, then raise the shared memory to 100 KB, or the registers to 128. With 1,024 threads and 65 registers, not even one block fits: the launch fails.
Switch between size, bandwidth and latency, and pick a level to read its numbers. The horizontal axis is logarithmic: each grid line is a factor of 10. Yellow levels are on the chip, orange is HBM, and the aqua network links (lessons 7–10) appear when you turn them on. Notice that HBM is about 1,600 times larger than L2 but has less bandwidth.
Lesson 2 · Roofline and arithmetic intensity
The roofline of one GPU. The horizontal axis is the intensity (FLOP per byte), the vertical axis the FLOP/s a kernel can reach; both are logarithmic. The sloped orange line is the memory roof , the flat blue line is the compute roof . Click an op to read its numbers, and drag it left or right to change its intensity. Then switch the GPU: the memory-bound points on the left move up only by the bandwidth ratio.
Predict first. Set your guess for the batch size at which the matmul becomes compute-bound, then press Reveal. The curve is the intensity , the blue line the ridge point and the dashed line the ceiling . Move and read the table: from to the work grows 64×, but the time hardly changes. Then make : the ceiling drops below the ridge, and no batch is large enough.
One transformer layer, op by op. Each bar is the roofline time of one kernel: blue if compute-bound, orange if memory-bound. Start in decode: almost every bar is orange, and the weight matmuls dominate. Raise the batch: the matmul bars stay almost the same until about 300 sequences, but attention grows with every sequence. Switch to GQA, then to prefill, where the matmuls turn blue.
A calculator for the four steps. Pick an op (or type your own FLOPs and bytes) and a GPU. The two bars are compute time and memory time; the longer one is the bound. The panel shows what three changes would give: a faster GPU, half the bytes, or twice the FLOP/s. For a memory-bound op only fewer bytes or more bandwidth help.
Lesson 3 · Kernels, tiling and fusion
Move the slider for from down to and watch the two lines. The orange line is the unfused chain ( kernels), the yellow line the fused kernel. In the violet region each unfused kernel is shorter than its launch cost. For unary ops both regimes give a speed-up of , for two different reasons. Then switch to ops with a second input: the speed-up is now different in the two regimes.
Press Next step and follow one output tile. At each step the kernel loads one tile of and one tile of into shared memory and adds their product to the C tile, which stays on the chip. The numbers in and count how often each element was read from HBM. Run to the end with : every element is read times. The bar at the bottom is the shared memory that a full-size kernel needs for two bf16 tiles; move the full-size slider until it turns magenta.
Change the stride and watch which sectors the warp fetches. Each small box is one element; the solid ones are the elements the 32 threads read (the number is the thread). With stride 1 every fetched byte is used. With stride 2 half of each sector is wasted. Try column of a matrix: 32 sectors for 32 elements. Then try misaligned row: a contiguous read that does not start at a sector boundary needs one extra sector.
Each column of the grid is one bank; each row is 32 consecutive words. The
magenta cells are the words one warp reads; the bars count how many
different words each bank must serve. Start with a column read of tile[32][32]: all 32 words sit
in bank 0. Switch to tile[32][33]: the padding words shift each row, and
the same column read spreads over all 32 banks. Also try the diagonal and one word.
Lesson 4 · FlashAttention and online softmax
Press Next block and watch the yellow state. The top row shows the scores; the current block is magenta. The bars below are the weights of the scores seen so far, relative to the current max. When a block raises the max, all old bars shrink by the same rescale factor. At the end, equals the exact softmax average (difference about ). Try one huge score: the naive overflows, the online version does not.
Press Play (or Step) and compare the two modes. The grid is the score matrix in tiles. In FlashAttention mode, one thread block keeps its yellow tile on chip while the magenta tile moves along the row; the score tiles it computes (blue) never go to HBM. In standard mode, the whole matrix is written to and read from HBM (orange). Compare the HBM counters, then turn on causal to see the skipped tiles.
Drag the split point to cut eight keys into a set A and a set B. Each set computes its own softmax average and its ; the merge weights them by . The merged output equals full attention for every split. The magenta line is the common mistake, the plain average of and . Try mask B to see a set with .
Move the sequence length and watch the two lines. The chart shows the minimum time of one head's forward pass on the roofline: standard and FlashAttention; the blue line is the compute time . Standard attention stays far above the compute line. Change to see the reuse factor at work, and turn on L2 serves re-reads to see the ideal case. The readout also shows the memory each version keeps for the backward pass.
Raise the number of splits and watch the SMs fill. Each square is one SM: orange SMs stream the KV cache from HBM, violet SMs are idle. The time model assumes that the attention time scales with the number of busy SMs. Notice that a large batch fills the GPU without splitting, and that a short KV cache gains little because the extra merge kernel costs a few microseconds.
Lesson 5 · Mixed-precision training
Pick a format and type a value. The top row shows the bits of the stored value: sign (grey), exponent and mantissa. The middle line shows the representable numbers next to ; the number under a gap is its size, the ulp. The bottom strip shows the range of the format on a log scale, with the underflow and overflow zones. Try in bf16 (a tie: it rounds to 1), then 70000 in fp16 (overflow) and in fp16 and in bf16. Notice that bf16 has the range of fp32 but a much coarser grid.
Add the same term again and again. The magenta curve is the computed sum, the dashed line the exact sum. With a bf16 accumulator and the naive method, the sum stops at 32. Switch the accumulator to fp32: the curve follows the exact line. Then go back to bf16 and try pairwise and Kahan: they stay close to the exact line even with a bf16 accumulator. Try the term 1: in bf16 the sum stops at exactly , like the toy counter at .
One training step, coloured by format: fp32, 16-bit, fp8 and loss-scaling steps. Click a box to read why it has its format. Then switch to fp16 + loss scaling: two magenta steps appear, "× S" after the loss and "÷ S, inf check" before the optimizer. In FP8 matmuls, only the matmul inputs change; the master weights, the optimizer and the reductions stay in fp32.
Move the loss scale. The bars are the magnitudes of the gradients of a synthetic network, on a log₂ scale, after multiplication by . The magenta zones are lost: values at or below become 0, values from 65520 up become inf. At about 42% of the gradients are flushed to 0. Raise : from on, less than 0.1% are flushed; from on, the largest gradients overflow and GradScaler would skip the step. Here only loses nothing at all. Then switch to bf16: its range is so wide that no scaling is needed.
Lesson 6 · Training memory and checkpointing
Step through one training step. The grey band is the model states, the orange area is the stored activations of a 24-layer model (the 1.3B model of section 4). Press Play or drag the time slider. Notice that the peak is at the turn from forward to backward, and that the model states do not move. Switch on checkpointing to see the sawtooth that we explain in section 3.
Count the parameters of one layer. Each bar is one weight matrix (or all biases and norms together), in units of . Switch between the GPT-style and the Llama-style layer, change the MLP width and the number of key/value heads, and compare the model total with . Notice that the embedding matters for small models: for the 1.3B preset, is only 92% of the total.
Walk through one layer. Press Next (or click an operation) to add the tensor it keeps. The running total in units of is orange; the magenta items are the attention core, counted in units of . The orange total reaches 34. Switch on FlashAttention: the magenta items disappear.
Checkpoint every layers. The yellow band is the stored segment inputs (the checkpoints; the band is thin because one checkpoint is only ); the magenta marks on the time axis are the recomputed forward passes. Move from 1 to 24 and watch the peak and the step time. Notice the sawtooth: each segment is rebuilt, then consumed by its backward pass.
Does it fit? The stacked bar is the memory of one GPU: weights (blue), gradients (yellow), optimizer states (violet), activations (orange) and logits (magenta). The solid line is the GPU's memory; the dashed line leaves 15% free. Start with the 1.3B preset without FlashAttention, then try the fixes in the list below. Notice which fixes change the orange part and which change nothing.
Lesson 7 · Collectives and the α-β model
Pick a collective and press Next step (or Play). Each column is a GPU and each row a chunk. Grey cells are a GPU's own data. Aqua cells arrived over a link or are sums: "ABD₂" means , and "Σ₂" is the full sum of chunk 2. Compare the start and end states with the table in section 3. Notice that all-to-all ends as a transpose: GPU holds chunk of every GPU.
Step through the ring. Each row is a GPU and each column a chunk; a cell shows how many of the inputs it already contains. In each step the aqua arrows carry the chunk written on them, the dashed cells are the chunks being sent, and the magenta frames mark the chunks that just arrived. Watch the full chunks (4/4) appear on the diagonal after step 3. Then change : the number of steps grows as , but the bytes each GPU sends stay close to . With the cells show the numbers of the worked example in section 4.
Move the message size (the magenta line) across the dashed grey crossover and watch the winner change. The solid aqua line is the ring, the dashed aqua line is recursive doubling and the dark grey line is Rabenseifner. Then raise the latency α: moves right. Raise : the ring's flat left part (its latency term ) rises. The α values of the presets are assumptions for illustration; measure your own cluster.
Switch between flat ring and hierarchical, then change the number of nodes. The flat ring's magenta hops cross the network, and every step waits for them. In the hierarchical scheme the light aqua parts use NVLink and the solid aqua part is the network; the dashed grey line is the network lower bound. Notice the bus bandwidth: it is far above the 50 GB/s of one network card. Latencies (3 µs NVLink, 10 µs network) are assumed.
Lesson 8 · Data parallelism, ZeRO and FSDP
Compare the stages. The left bars are the model states of one GPU: weights (blue), gradients (yellow) and optimizer states (violet); the line is 80 GB. The right bars are the bytes each GPU sends per step (aqua). Increase : the optimizer part shrinks first (ZeRO-1), then the gradients (ZeRO-2), then everything (ZeRO-3). Notice that the aqua bars of DDP, ZeRO-1 and ZeRO-2 are equal.
Change the bucket size. The top lane is the backward pass, layer by layer from the last layer (blue). The bottom lane is the all-reduce of each bucket (aqua). The violet part is exposed: the GPU waits for communication. At the start (25 MB buckets, 16,384 tokens per GPU, 50 GB/s) only the last bucket is exposed. Then try very small buckets (the latency of each all-reduce adds up), one huge bucket (nothing overlaps), fewer tokens and a slower network. The model: 24 layers of 50M parameters, 8 GPUs, TFLOP/s.
Turn prefetch and resharding on and off. Each FSDP unit needs an all-gather (aqua) before its compute (blue); in the backward pass each unit ends with a reduce-scatter. The curve below counts the units whose full parameters are in memory. Without prefetch, compute waits for every all-gather (violet). Without resharding, the backward all-gathers disappear, but all units stay in memory at the end of the forward pass.
Find the threshold. The curve is (communication time) / (backward time) against the tokens per GPU; below 1 (the line) the all-reduce hides. The bars show the two times for your settings: compute (blue) and communication (aqua). Move the model-size slider: both bars change, but the curve and the threshold do not move.
Lesson 9 · Tensor and pipeline parallelism
Press Next step to see which block each GPU computes. All blue blocks are local work; the only communication is the aqua all-reduce at the end. Then switch to backward: the all-reduce moves to . Then try wrong split: splitting by rows puts a sum before the GeLU, which costs an extra magenta all-reduce on a 4× larger tensor. The side panel checks each claim on random matrices: the split result equals the unsplit one to about .
The central figure of this lesson. Each row is a GPU (a stage); time runs to the right. Light blue boxes are forward passes, dark blue boxes backward passes (the number is the micro-batch; with interleaving, the subscript is the chunk), and the violet background is idle time. The orange bars on the right show the most micro-batches each stage keeps in memory at once. Switch GPipe → 1F1B: the step time and the bubble do not change, but the memory bars shrink. Then raise , and then try interleaved with . Press Play to watch the line fill and drain.
Change , and the link. The solid aqua line is the ratio over NVLink, the dashed line over InfiniBand; above the magenta line (ratio 1) the all-reduces take longer than the matmuls. Notice that the InfiniBand line is 9× higher for every , and that forces the network on 8-GPU nodes. The bars in the panel are for tokens, but the ratio does not depend on .
Plan a layout. Each box is a node of 8 GPUs; colour the GPUs by TP group, PP stage or DP replica (here the colours only tell groups apart). The bar shows the memory of one GPU of the first stage: weights, gradients, optimizer states and activations, against the 80 GB of an H100. Start from the worked example below (70B, 8 nodes, TP 8, PP 8), then try PP 4 with ZeRO-1, then TP 16, then the 7B model. Warnings appear in magenta. A back-of-the-envelope model: it ignores embeddings, logits and buffers, so keep 10–20% free.
Lesson 10 · Sequence, context, expert parallel
Press Next round to rotate the K/V blocks. The grid is the score matrix: rows are queries, columns are keys, and each rank owns the rows of its tokens. Blue cells are computed in this round, faint blue cells were computed before, and grey cells are masked. The bars show the work of each rank in this round; violet is idle time, because all ranks wait for the slowest one. Switch to zig-zag and watch the idle time disappear.
Switch between TP and TP + SP. The strip is one transformer layer; each box shows which part of the activation GPU 0 stores. Blue: split by heads or columns (TP). Violet: a full copy on every GPU (wasted memory). Magenta: split by tokens (SP). The aqua markers are the collectives. Notice that SP turns every violet box magenta and every all-reduce into a reduce-scatter (RS) and an all-gather (AG), and the bytes readout does not change.
Move the block size and switch the link. The blue line is the compute time per round, ; the aqua line is the K/V transfer time, . Left of the crossing the transfer is exposed; right of it, it is hidden. Change : both times change, but does not move. GQA and NVLink move to the left; the causal mask moves it to the right.
Press All-to-all to switch from sequence sharding to head sharding and back. The grid is one tensor ( tokens × 8 heads); each cell shows the rank that holds it, and the cells of the selected rank are magenta. Cells with an aqua border just arrived over the network. Then reduce the number of K/V heads below the number of ranks and read the warning.
Move the router skew to make some experts popular. Each dot is one (token, expert) copy, stacked on the expert that receives it; the dashed line is the capacity. Blue copies are computed, magenta copies are dropped. The lower bars are the load of each GPU, and violet is the time the other GPUs wait for the busiest one. Set the capacity factor to "none": nothing is dropped, but the slowest GPU sets the step time.
Lesson 11 · Quantization
Move the sliders. Each grey dot on the top line is a value , and its line goes to its quantized value on the grid below (the grey ticks are the grid points). Raise the bits: the grid gets finer and the errors shrink. Now add one outlier: the step grows, and the 16 small values crowd onto 3 grid points. Lower the range: the outlier is clipped, but the small values get a finer grid. The readout compares the largest error with the bound .
Find the minimum. The curves show the mean squared error of 4-bit (or 8-bit) symmetric quantization of 8192 samples with unit variance, against the range . The rounding part grows with and follows the dashed line ; the clipping part falls. Drag to the bottom of the black curve. At 4 bits, the best range for this Gaussian sample is about , and min-max gives about twice the error. Switch to Laplace (heavier tails) and to 8 bits: the best range moves outwards.
Change the granularity. Rows are the outer dimension , columns the reduction dimension . The left map shows ; the right map shows the error on one fixed colour scale. With one scale per tensor, the outlier column sets the step and every other value has a large error. One scale per row helps little: the outlier channel is large in every row. One scale per column removes the problem, but the verdict says why an int8 matmul cannot use it for activations. Then try the weights: their rows have different sizes, and one scale per row (per output channel) is enough.
Move . The bars are the channel maxima of the activations and the weights , faint before smoothing and solid after (log scale). Channels 2 and 6 hold outliers. At every activation channel has maximum 1 and the weights take all the range; at it is the other way round. The lower chart is the relative error of in W8A8 (one scale for , one per output column of ). The last readout shows that equals up to float rounding: the identity is exact.
W4A16 on the roofline of an H100 in bf16, for one weight matmul with tokens. "W16" is the bf16 weight, "W4" the int4 weight with group scales (4.125 bits). Click a point and read its minimum time. At and the int4 weight moves the point to the right on the memory roof: the time drops by up to 3.9×. At (prefill) both points are on the compute roof, and the time is the same.
Lesson 12 · LLM inference serving
Move the batch from 1 to 64, then to 186. The top bars are the roofline times of one step: orange is the time to read the weights (dark) and the KV cache (light), blue is the compute time. The step takes the longer of the two. From to the work grows 64×, but the step time only doubles. Then the light part (the caches) takes over. The chart below shows the trade-off over all batch sizes: more tokens per second, but a longer time per output token. Try a long context, fp8 for the cache, or the 7B model without GQA, and watch and the memory bar.
Compare the two schedules of the same requests. Each row is a batch slot; each request starts with a prefill step and continues with decode steps. Violet cells are slots that do no work. In static batching, a short request holds its slot until the longest request of its batch ends. Press New random requests a few times, and make the output lengths more different (longest output): static batching gets worse, continuous batching stays near full use. One step costs about the same for any batch that fits, so fewer steps means more tokens per second.
Press Step a few times in the contiguous mode, then switch to paged and do the same. Each square is one token slot; a group of 4 is one block. Orange slots hold tokens; violet slots are reserved or allocated but empty (waste); a short vertical line marks the start of each reserved range. Notice the used share: low with reservation, close to 100% with paging. In paged mode, press Fork the oldest request: the new sequence shares all blocks (magenta frame, ×2). At the next step, watch copy-on-write in the message. Add requests until the pool is full to see preemption.
One position, two views of the same tokens. The outlined bar is the draft ; the stacked bar is what the rule emits: the kept part and the rejected-then-resampled part. The dashed line is the target . The stacked bar always ends at the dashed line. Move the draft quality : the acceptance rate grows and the magenta part shrinks, but the total stays . Press Sample to check with 2,000 random draws (dots). Switch to 4 tokens for a less simple case.
Press Simulate 12 rounds. Each row is one round: accepted draft tokens, then either the replacement token at the first rejection (later draft tokens are discarded) or, if all are accepted, a bonus token. The number at the right is the tokens of that round. The bars below are ; their sum is the expected number of tokens. Raise : the new bars get small fast, while the draft cost grows linearly, so the speed-up has a best .
A 32,768-token prompt arrives (magenta line) while 64 sequences decode. Each bar is one step; its height is the TPOT that all 64 sequences see (log scale). With one piece, one blue step lasts more than a second. Switch to chunked prefill: the prompt is spread over many short steps, and TPOT stays near the 50 ms target; the new prompt's TTFT grows a little. Switch to disaggregated: the decode GPU is not disturbed at all; the cost moves to a second GPU and a KV transfer.
Lesson 13 · Profiling, MFU and compilation
Pick a timeline (A to E) and name its problem with the buttons. Each timeline has one problem. Blue boxes are compute kernels, aqua boxes are NCCL kernels, the violet background is GPU idle time, and grey and pink boxes on the CPU row are launches and waits. Then press Apply the fix and compare the step time with the dashed line of the old step.
Move the size from small to large. The workload is one bf16 matmul followed by elementwise ops, on an H100, with 3.6 µs of CPU time per launch. The lower chart shows which part of the step time is overhead, memory or compute. Now switch on fuse, CUDA graph and fp8 matmul one at a time at , and . Each toggle helps only in its own regime. (The model assumes a GPU-side minimum of 1 µs per kernel inside a CUDA graph.)
Start from the log line of section 1 (7B, 32 H100s, 256 sequences of 4096 tokens per 3.60 s): read the MFU and the HFU with full recomputation. Switch to plan a run and set 256 GPUs, 40% MFU and tokens to get the example above. Then try PaLM 540B: its published MFU of 46.2% comes out only with its attention width , not its hidden size of 18,432. PaLM recomputed only part of the activations, so its published HFU (57.8%) lies between the values for none and full.
Switch between eager and torch.compile and count the kernels. Then add the print(...item())
line and the if line: each one is a graph break, and the ops after it land
in a new graph and a new fused kernel. On the right, call the compiled
function with different shapes and watch when it compiles again. These groupings were checked with
PyTorch 2.15 on a CPU.