Videos and books
The free courses, lecture series, books and short reads that pair best with this track, with a note on what each one is good for.
Every resource on this page is free to watch or read, from the author's, the course's or the publisher's own page. Where a standard book is not free, we say so and point to its official page. Each lesson of the track also ends with a Watch and read section that names the exact lecture, paper section or blog post.
How to use it. This field changes fast, and most of its knowledge is in papers, documentation and talks, not in textbooks. A good default: one talk before a lesson for the picture, then the lesson and its notebook, then the paper sections from the table at the end of this page. Check every number you read against your own count of FLOPs and bytes: that is the main skill of the track.
Video courses #
GPUs and kernels #
Warps, scheduling, occupancy and the memory types of an SM, with CUDA examples. Pairs with lessons 01 and 02.
- GPU MODE lectures (YouTube channel, notes and code). A community lecture series by people who write the kernels: profiling (lecture 1), compute and memory (4), the CUDA performance checklist (8), FlashAttention (12), ring attention (13), Triton (14), NCCL (17) and many more. The best single source for lessons 01–04, 07 and 13.
- Stanford CS336, Language Modeling from Scratch (Percy Liang and Tatsunori Hashimoto, 2025; course page, YouTube playlist). Lecture 5 (GPUs), 6 (kernels, Triton), 7 and 8 (parallelism) and 10 (inference) cover most of this track for language models. The videos cannot be embedded here; watch them on YouTube.
Why ML workloads are limited by compute, memory bandwidth or overhead, from a developer of PyTorch. Pairs with lessons 02, 03 and 13.
Training at scale #
Data parallelism and the communication primitives (all-reduce, ring, tree), then ZeRO, pipeline and tensor parallelism. Pairs with lessons 07–09.
- MIT 6.5940, TinyML and Efficient Deep Learning Computing (Song Han; course page, Fall 2023 lectures on YouTube). Quantization (lectures 5 and 6), distributed training (17 and 18 in 2023), LLM serving and on-device training. Pairs with lessons 06–09 and 11.
- CMU 10-414/714, Deep Learning Systems (Tianqi Chen and Zico Kolter; course page with lecture videos). Build an automatic differentiation library and a small deep learning framework, then learn about hardware acceleration, memory saving and parallel training. Good background for lessons 03, 06 and 13.
Starts at tensor parallelism (1:32:11); pipeline parallelism starts at 2:11:20. Pairs with lesson 09.
Clear drawings of data parallelism, the three ZeRO stages and the FSDP schedule. Pairs with lesson 08.
Attention, inference and serving #
The author explains IO-awareness, tiling and recomputation, and the measured speed-ups. Pairs with lesson 04.
Number formats, linear (affine) quantization with scale and zero-point, and how an integer matmul is computed. Pairs with lessons 05 and 11.
The authors of vLLM explain the memory waste of earlier servers, paging, sharing and the measured gains. Pairs with lesson 12.
How to find graph breaks and recompilations, and what to do about them. Pairs with lesson 13.
Free books #
- Austin et al., How to Scale Your Model (Google DeepMind, 2025; free online). Rooflines, the transformer FLOP and memory count, all parallelism strategies and inference, with many worked exercises. Written for TPUs, with a chapter on GPUs. The closest book to this whole track.
- Hugging Face, The Ultra-Scale Playbook: Training LLMs on GPU Clusters (2025; free online). Data, tensor, pipeline, context and expert parallelism, ZeRO, FlashAttention and mixed precision, with measured numbers. The code of picotron implements each method in a few hundred lines. Lessons 05–10.
- Modal, GPU Glossary (free online). Short entries on every part of a GPU and of the CUDA software stack, linked to each other. Lessons 01–03.
- Stas Bekman, Machine Learning Engineering Open Book (free on GitHub). The practical side of large training runs: hardware, networks, debugging, fault tolerance and performance tuning. Lessons 07, 08 and 13.
- NVIDIA, CUDA C++ Programming Guide (free online) and NVIDIA, Deep Learning Performance Guide (free online). The programming model (grids, blocks, warps, shared memory) and how GEMM shapes, tensor cores and waves set the speed of a layer. Lessons 01–03.
- Hwu, Kirk and El Hajj, Programming Massively Parallel Processors (4th edition, 2022; not free, publisher page). The standard textbook on CUDA: memory coalescing, shared-memory tiling, reductions and scans. Lessons 01 and 03.
Short reads #
- Horace He, Making Deep Learning Go Brrrr From First Principles. Compute, memory bandwidth and overhead: the three regimes of lessons 02, 03 and 13.
- Simon Boehm, How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance. From a naive kernel to block and register tiling, with the speed of each step. Lesson 03.
- Sasha Rush, GPU Puzzles. Short exercises that teach threads, blocks and shared memory in Python. Lessons 01 and 03.
- Dao, Haziza, Massa and Sizov, Flash-Decoding for long-context inference (2023). Split-KV for decoding, with pictures. Lesson 04.
- David Goldberg, What Every Computer Scientist Should Know About Floating-Point Arithmetic (1991). Rounding, ulps and relative error. Lesson 05.
- EleutherAI, Transformer Math 101. Rules of thumb for the compute and memory of training. Lessons 06 and 13.
- Andrew Gibiansky, Bringing HPC Techniques to Deep Learning (2017). The ring all-reduce explained step by step. Lesson 07.
- NVIDIA nccl-tests, Performance reported by NCCL tests. Algorithm bandwidth, bus bandwidth and the factor of each collective. Lesson 07.
- DeepSpeed, Zero Redundancy Optimizer tutorial and PyTorch, Getting Started with Fully Sharded Data Parallel. ZeRO and FSDP in code. Lesson 08.
- Lilian Weng, Large Transformer Model Inference Optimization (2023). A survey of quantization, pruning, distillation and attention changes for inference. Lessons 11 and 12.
- vLLM team, vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (2023), and nano-vllm, a serving engine with paging, prefix caching and CUDA graphs in about 1,200 lines. Lesson 12.
- PyTorch, torch.compiler documentation.
How
torch.compileworks, and how to find graph breaks and recompilations. Lesson 13.
Which lesson pairs with what #
| Lesson | Watch | Read |
|---|---|---|
| 01 GPU architecture | GPU MODE lecture 4; NVIDIA GTC 2025, Getting Started with CUDA; CS336 lecture 5 | How to Scale Your Model, GPUs chapter; CUDA programming guide; Modal GPU Glossary |
| 02 Roofline and arithmetic intensity | GPU MODE lecture 8; Horace He, Jane Street talk; NHR@FAU roofline intro | Horace He, Go Brrrr; How to Scale Your Model, rooflines; Williams et al. (2009) |
| 03 Kernels, tiling and fusion | GPU MODE lectures 5 and 14 | Triton tutorials; Boehm, CUDA matmul; CUDA best practices guide; Hwu et al. |
| 04 FlashAttention and online softmax | Tri Dao, Stanford MLSys; GPU MODE lecture 12; Jia-Bin Huang | Dao et al. (2022) §3; Dao (2023); Flash-Decoding blog; Milakov–Gimelshein (2018) |
| 05 Mixed-precision training | Computerphile, floating point; Karpathy, GPT-2 (from 1:28:14); PyTorch FP8 talk | Micikevicius et al. (2018) §3; FP8 formats (2022); torch.amp docs; Goldberg (1991) |
| 06 Training memory and checkpointing | PyTorch activation checkpointing talk; Aleksa Gordić, Scaling ML Models | Korthikanti et al. (2022) §4–5; Chen et al. (2016); ZeRO §3; Transformer Math 101 |
| 07 Collectives and the α-β model | MIT 6.5940 lecture 19 (2024); GPU MODE lecture 17 (NCCL) | Thakur et al. (2005); Patarasuk–Yuan (2009); nccl-tests performance notes; Gibiansky (2017) |
| 08 Data parallelism, ZeRO and FSDP | Jia-Bin Huang, ZeRO & FSDP; Yannic Kilcher, ZeRO; PyTorch DDP talk | ZeRO §5 and §7; PyTorch FSDP paper (2023); PyTorch DDP paper (2020); Ultra-Scale Playbook |
| 09 Tensor and pipeline parallelism | Jared Casper, Megatron-LM; GPU MODE lecture 48 | Shoeybi et al. (2019) §3; Narayanan et al. (2021) §2–3; Llama 3 §3.3.2 |
| 10 Sequence, context and expert parallelism | GPU MODE lecture 13 (ring attention); Stanford CS25, MoE; Yannic Kilcher, Switch Transformers | Korthikanti et al. §4.2; Liu et al. (2023); Jacobs et al. (2023); Fedus et al. (2021) §2 |
| 11 Quantization | MIT 6.5940 lectures 5 and 6; Umar Jamil, quantization | Nagel et al. (2021) §2–4; SmoothQuant; LLM.int8(); GPTQ; AWQ |
| 12 LLM inference serving | vLLM talks (Ray Summit 2023, PyTorch Conference); Jia-Bin Huang, speculative decoding; GPU MODE lecture 58 | Kwon et al. (2023) §3–4; Leviathan et al. (2023); Orca (2022); Sarathi-Serve (2024) |
| 13 Profiling, MFU and compilation | GPU MODE lecture 1; torch.compile: The Missing Manual; NVIDIA Nsight Systems timeline | PaLM §4.1 and App. B; PyTorch 2 paper (2024); PyTorch profiler recipe; JAX jit docs |