Videos and books

Every resource on this page is free to watch or read, from the author's or the publisher's own page. Each lesson of the track also ends with a Watch and read section that names the exact lecture, chapter or paper section.

How to use it. Watch for the picture, read for the derivations, and write code to check both. A good default: one short video before a lesson, then the lesson and its notebook, then one book section from the table at the end of this page. For the large-model lessons (11, 12, 15, 16), read the original paper section that the lesson names.

Video courses #

For the picture #

• 3Blue1Brown, Neural networks (course page). Short animated chapters: what a network is, gradient descent, backpropagation (intuition and calculus), transformers, attention, and how language models store facts. No code. Watch chapters 1–4 with lessons 01–03 and chapters 5–7 with lessons 09 and 11.
• StatQuest with Josh Starmer (neural networks and AI). Slow, step-by-step videos with small numbers: the main ideas of neural networks, RNNs, LSTMs and the matrix math of transformers. Good when a lesson feels too fast.

Build it in code #

• Andrej Karpathy, Neural Networks: Zero to Hero (course page). You write everything yourself: micrograd (backpropagation), makemore (language models, activations and BatchNorm, manual backprop), GPT from scratch, and a GPT-2 reproduction with mixed precision and learning-rate schedules. The best companion to lessons 02–05, 09, 11 and 12.
• fast.ai, Practical Deep Learning for Coders (course). A top-down, code-first course: train real models first, then open them up. Useful if you want results before derivations.
• Phillip Lippe, UvA Deep Learning Tutorials (notebooks). Clear PyTorch notebooks on activations, initialization and optimization, CNNs, transformers, GNNs and self-supervised contrastive learning. Pairs with lessons 04, 07, 11, 13 and 14.

University courses #

• Stanford CS231n, Deep Learning for Computer Vision (course, notes). The 2017 lectures on YouTube cover loss functions, backpropagation, training (initialization, BatchNorm, dropout, optimizers), CNN architectures and RNNs. The notes are among the best short texts on these topics. Lessons 02–08.
• MIT 6.S191, Introduction to Deep Learning (course). A fast yearly overview with labs: MLPs, sequence models, CNNs, generative models. A good first pass before the track.
• Stanford CS224N, Natural Language Processing with Deep Learning (course). The lectures on RNNs, LSTMs, self-attention and transformers. Lessons 08–11.
• Stanford CS25, Transformers United (course). Guest talks by the authors of important models, including mixture of experts and state-space models. Lessons 11, 15 and 16.
• NYU Deep Learning (LeCun and Canziani) (course). Lectures and notes with an energy-based view, including a long lecture on self-supervised learning in vision. Lesson 14.
• Geometric Deep Learning (Bronstein, Bruna, Cohen, Veličković) (lectures). Symmetry, invariance and equivariance as the common idea of CNNs, GNNs and transformers. Lessons 07 and 13.

Free books #

Main textbooks #

• Simon J. D. Prince, Understanding Deep Learning (MIT Press 2023; free PDF and notebooks). The main textbook of this track. Clear figures, consistent notation and Python notebooks for every chapter: shallow and deep networks, loss functions, fitting, backpropagation and initialization, generalization and double descent, regularization, CNNs, residual networks, transformers, GNNs, and why deep learning works.
• Zhang, Lipton, Li and Smola, Dive into Deep Learning (free online). Every idea with runnable PyTorch code: MLPs, numerical stability, dropout, BatchNorm, ResNets, RNNs and LSTMs, attention and transformers. Use it when you want to see the code next to the math.
• Christopher M. Bishop and Hugh Bishop, Deep Learning: Foundations and Concepts (Springer 2024; free to read online). Careful and probabilistic: loss functions from likelihoods, normalization, CNNs, transformers, GNNs.
• Goodfellow, Bengio and Courville, Deep Learning (free online). The classic reference (2016). Strong on optimization (chapter 8), regularization (chapter 7) and recurrent networks (chapter 10). It predates transformers.
• Michael Nielsen, Neural Networks and Deep Learning (free online). A gentle first book with code. Chapter 2 explains backpropagation and chapter 4 gives a visual proof of universal approximation. Lessons 01–03.
• François Fleuret, The Little Book of Deep Learning (free PDF). About 170 small pages that cover the whole field, from training to transformers. A good review after the track.

Specialized and advanced #

• Roberts, Yaida and Hanin, The Principles of Deep Learning Theory (arXiv). Signal propagation, criticality and initialization in wide networks, from a physics point of view. Advanced; read with lessons 04 and 17.
• Bronstein, Bruna, Cohen and Veličković, Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges (arXiv). Equivariance as the design principle behind CNNs, GNNs and transformers. Lessons 07 and 13.
• William L. Hamilton, Graph Representation Learning (free PDF). Message passing, GCNs, expressivity and over-smoothing in one short book. Lesson 13.
• Kevin P. Murphy, Probabilistic Machine Learning: An Introduction and Advanced Topics (free PDFs). Broad references with chapters on neural networks, self-supervised learning and generalization. For lookup.
• Francis Bach, Learning Theory from First Principles (free PDF). Rademacher complexity, kernels, implicit bias and double descent, with proofs. Lesson 17.
• Jurafsky and Martin, Speech and Language Processing, 3rd edition draft (free online). Clear chapters on RNNs, transformers and language models. Lessons 08 and 11.

Short reads and tools #

• Andrej Karpathy, "A Recipe for Training Neural Networks" (blog). A practical checklist: start simple, overfit one batch, then regularize. Lessons 02 and 06.
• Christopher Olah, "Understanding LSTM Networks" (blog). The gates of an LSTM, one figure at a time. Lesson 08.
• Jay Alammar, "The Illustrated Transformer" (blog). The shapes inside attention and a transformer block, as pictures. Lessons 09 and 11.
• Rush et al., "The Annotated Transformer" (online). The original paper, line by line, with PyTorch code. Lessons 09–11.
• EleutherAI, "Rotary Embeddings: A Relative Revolution" (blog). RoPE derived and tested. Lesson 10.
• Google Research, Deep Learning Tuning Playbook (GitHub). How to choose the batch size, the optimizer and the learning-rate schedule in a real project. Lesson 12.
• Kipply, "Transformer Inference Arithmetic" (blog). FLOPs, memory and the KV cache with real numbers. Lesson 11.
• Sanchez-Lengeling et al., "A Gentle Introduction to Graph Neural Networks" (Distill). Message passing with interactive figures. Lesson 13.
• Lilian Weng, "Contrastive Representation Learning" (blog). A survey of contrastive losses and methods. Lesson 14.
• Hugging Face, "Mixture of Experts Explained" (blog). Routing, load balancing and capacity in plain words. Lesson 15.
• Rush and Karamcheti, "The Annotated S4" (online). The S4 layer with JAX code: discretization, the convolution kernel and the recurrence. Lesson 16.
• Odena, Dumoulin and Olah, "Deconvolution and Checkerboard Artifacts" (Distill). Why transposed convolutions make grid patterns. Lesson 07.

Which lesson pairs with what #

The first choice for each lesson. The Watch and read section at the end of each lesson gives more, with exact section numbers. "UDL" is Prince's Understanding Deep Learning; "D2L" is Dive into Deep Learning.