ML Systems
Every training step and every generated token is a sequence of GPU kernels, and each kernel is limited by arithmetic, by memory traffic or by communication. Learn to count FLOPs and bytes, to predict which limit you hit, and to choose the fix: fused kernels, FlashAttention, lower precision, sharding, parallelism across thousands of GPUs, quantization and efficient serving.
For students, engineers and researchers who train or serve neural networks and want to know where the time and the memory go. You need matrix multiplication (linear algebra lesson 1), floating-point numbers (linear algebra lesson 13), backpropagation and the transformer block (deep learning), and Adam (optimization). No GPU is needed: every exercise runs on a laptop CPU with NumPy.
Quick recap
The key takeaways of all lessons on one page
All interactive figures
Every figure of the track on one page
Videos and books
The best free courses
Glossary
Every technical term of the track with a short definition
Concept map
How the lessons of this track connect to each other and to other tracks.
Keeps every lesson page and figure of this track in this browser, so you can read it without internet.
One GPU: hardware, roofline and kernels
How a GPU runs a kernel, the roofline model that says whether arithmetic or memory is the limit, and how tiling, fusion and FlashAttention move fewer bytes.
-
01
GPU architecture: SMs, warps, the memory hierarchy and tensor coresHow a GPU runs a kernel (grid, blocks, warps, SMs), why it needs many warps to hide memory latency, what limits occupancy, where data lives, and what tensor cores compute.
50 min
4 figures
Beginner -
02
The roofline model and arithmetic intensityCount FLOPs and bytes, divide, and you know whether a kernel is limited by arithmetic or by memory. The matmul intensity theorem, the batch size where a matmul becomes compute-bound, and the regime of every op in a transformer.
60 min
4 figures
Intermediate -
03
Kernels, tiling, coalescing and fusionHow a GPU kernel is written (CUDA and Triton), why the access pattern decides how many bytes really move, and how fusion and tiling cut the HBM traffic of elementwise chains and matrix multiplies.
65 min
4 figures
Intermediate -
04
FlashAttention and online softmaxExact attention without the N × N matrix. The online softmax keeps a running max, sum and output and rescales them; tiles of keys stream through on-chip memory; partial results merge with their logsumexp. HBM traffic, recomputation in the backward pass, FlashAttention-2 and split-KV decoding.
80 min
5 figures
Advanced
Numbers and memory
Which floating-point format to use where, why low-precision sums fail and how mixed precision fixes them, and where every byte of training memory goes.
-
05
Mixed precision: formats, loss scaling and accumulationFewer bits per number give faster tensor cores and half the bytes, but cost precision and range. The formats fp32, tf32, bf16, fp16 and fp8; why a low-precision sum stops growing; and the mixed-precision recipe with fp32 master weights, fp32 accumulation and loss scaling.
60 min
4 figures
Intermediate -
06
Training memory: model states, activations and checkpointingWhere every byte of training memory goes. Model states take 16 bytes per parameter with mixed-precision Adam; activations take bytes per transformer layer. FlashAttention, recomputation and checkpointing shrink the activations, and a short sum tells you whether a run fits.
70 min
5 figures
Intermediate
Training on many GPUs
Collective communication and its cost model, data parallelism with ZeRO and FSDP, tensor and pipeline parallelism, and how long sequences and mixture-of-experts layers are split.
-
07
Collective communication and the alpha-beta cost modelBroadcast, reduce, all-reduce, reduce-scatter, all-gather and all-to-all, and one cost model, latency α plus time per byte β, that predicts how long each takes. Ring all-reduce and why it is bandwidth-optimal, recursive doubling and the crossover size, bus bandwidth, and hierarchical all-reduce on real clusters.
65 min
4 figures
Intermediate -
08
Data parallelism, ZeRO and FSDPData parallelism gives each GPU part of the batch and averages the gradients. ZeRO removes the copies of the model states for the same communication (stages 1 and 2) or 1.5 times more (stage 3), FSDP implements stage 3 layer by layer, and the bytes hide behind compute when each GPU has more than about tokens per step.
70 min
4 figures
Intermediate -
09
Tensor and pipeline parallelism, and choosing a layoutSplit the model itself. Megatron tensor parallelism (column-then-row splits, one all-reduce per block and pass) and its communication-to-compute ratio; pipeline schedules GPipe, 1F1B and interleaved with their bubble and memory; and how to place TP, PP and DP on a cluster, with a 70B model on 64 GPUs as the worked example.
85 min
4 figures
Advanced -
10
Sequence, context and expert parallelismHow to split a long sequence and a mixture-of-experts layer across GPUs. Megatron sequence parallelism, ring attention with zig-zag load balancing, DeepSpeed-Ulysses, and expert parallelism with its all-to-all, capacity factor and load balancing.
80 min
5 figures
Advanced
Inference and performance in practice
Quantization, LLM serving with KV caches, paging, batching and speculative decoding, and how to measure, profile and compile a workload.
-
11
Quantization for training and inferenceStore weights and activations as small integers with a scale. The rounding error bound Δ/2 and the mean squared error Δ²/12, the clipping trade-off, per-tensor, per-channel and group-wise scales, outliers and SmoothQuant, PTQ and QAT, and which phase each scheme makes faster.
70 min
5 figures
Intermediate -
12
LLM inference serving: KV cache, batching, paging and speculative decodingWhy decode is limited by memory traffic, how many sequences fit next to the weights, and how a server shares one read of the weights. KV-cache size, the decode roofline, TTFT and TPOT, continuous batching, PagedAttention, speculative sampling (with proof), chunked prefill and disaggregation.
85 min
6 figures
Advanced -
13
Profiling, MFU and compilationCount the FLOPs a model needs, compare them with what the hardware delivers (MFU), estimate the training time, read a profiler timeline to find where the rest goes, and choose the tool that fixes it (fusion, CUDA graphs, torch.compile or JAX jit).
75 min
4 figures
Intermediate