17 lessons · about 21 hours · 75 figures
Deep Learning
Build a neural network from one ReLU unit, train it with backpropagation, and learn why deep networks train at all: initialization, normalization and residual connections. Then climb through convolutional and recurrent networks to attention and the transformer, and on to graph networks, self-supervised learning, mixture of experts, state-space models and generalization.
For students, engineers and researchers who know linear and logistic regression, matrix multiplication and the chain rule, and want to derive, implement and debug modern deep networks, from a first course to research level. Notebooks use NumPy and small PyTorch models on a CPU.
Quick recap
The key takeaways of all lessons on one page
All interactive figures
Every figure of the track on one page
Videos and books
The best free courses
Glossary
Every technical term of the track with a short definition
Concept map
How the lessons of this track connect to each other and to other tracks.
Keeps every lesson page and figure of this track in this browser, so you can read it without internet.
Part 1
Networks and how they learn
The multilayer perceptron, what it can represent, the training loop, and backpropagation.
-
01
MLPs and universal approximationA multilayer perceptron is a stack of affine maps and nonlinearities. One hidden layer can approximate any continuous function on a compact set, but this is an existence result only; depth can be exponentially more efficient than width.
65 min
3 figures
Beginner -
02
Training a network: losses, mini-batches and the training loopTraining minimizes the average loss over the training data with small gradient steps on random mini-batches. Losses come from maximum likelihood, are computed from logits for stability, and the learning rate must stay below a limit set by the curvature.
65 min
4 figures
Beginner -
03
Backpropagation on computational graphsHow a network computes all its gradients in one backward pass. Adjoints and the chain rule on a graph, the three local rules of an MLP, what backprop costs in time and memory, activation checkpointing, and how to check a gradient with finite differences.
65 min
5 figures
Intermediate
Part 2
Making deep networks trainable
Activations and initialization, normalization and residual connections, and regularization.
-
04
Activations and initialization from variance propagationWhy the choice of nonlinearity and the scale of the random initial weights decide whether a deep network trains. ReLU, GELU, SiLU and SwiGLU; the variance recursion and the LeCun, Xavier and He initializations; symmetry breaking; and a first look at μP.
65 min
4 figures
Intermediate -
05
Normalization layers and residual connectionsBatchNorm, LayerNorm and RMSNorm (axes, train and eval mode, the backward pass as removing two components), and residual connections (the identity path, pre-LN and post-LN transformer blocks).
75 min
5 figures
Intermediate -
06
Regularization: dropout, weight decay, label smoothing, augmentation, early stoppingFive ways to lower the test error of a network, each with its derivation and its classic bug. Inverted dropout, L2 versus weight decay (and why AdamW exists), label smoothing, data augmentation as invariance, and early stopping as ridge.
55 min
4 figures
Intermediate
Part 3
Convolutional and recurrent networks
Weight sharing in space and in time, receptive fields, ResNets, and vanishing gradients through time.
-
07
Convolutional networks: filters, arithmetic, im2col and ResNetA convolution layer slides one small filter over the whole input. Learn to compute its shapes, parameters, cost and receptive field, write it as one matrix product (im2col), derive its backward pass, and read the ResNet blocks.
85 min
5 figures
Intermediate -
08
Recurrent networks, LSTMs and vanishing gradientsTokens, embeddings and next-token prediction; the recurrent network and backpropagation through time; why gradients vanish or explode over many steps; gradient clipping; the LSTM cell; and why attention and state-space models replaced RNNs for long sequences.
70 min
4 figures
Intermediate
Part 4
Attention and transformers
Attention as a soft lookup, positional encodings, the transformer and the vision transformer, and how large models are trained stably.
-
09
Attention: queries, keys, values and masksAttention is a soft dictionary lookup. Scaled dot-product attention, why we divide by the square root of the head size, causal and padding masks, multi-head and cross-attention, and why attention without positions treats its input as a set.
70 min
5 figures
Intermediate -
10
Positional encodings: sinusoidal, learned, RoPE and ALiBiAttention without positions sees a set. Learned and sinusoidal encodings add position to the input; RoPE rotates queries and keys so that scores depend only on the offset; ALiBi adds a distance penalty. How each scheme behaves on inputs longer than in training, and how RoPE contexts are extended.
65 min
4 figures
Advanced -
11
The transformer: the block, model families and vision transformersHow attention, MLPs, LayerNorm and residual connections make the transformer block, and how blocks make GPT, BERT, T5 and the vision transformer. Count parameters and FLOPs, generate text with a KV cache, and turn an image into patch tokens.
95 min
4 figures
Intermediate -
12
Training dynamics: warmup, clipping, loss spikes, large batches, mixed precisionWhy large networks diverge and how practitioners keep them stable. Learning-rate warmup and decay, global-norm gradient clipping, the causes of loss spikes (attention-logit growth, output-logit drift, an out-of-date Adam second moment) and their fixes, large-batch training, and mixed precision with fp32 master weights and loss scaling.
80 min
5 figures
Advanced
Part 5
Modern architectures and theory
Graph networks, learning without labels, sparse experts, state-space models, and why large networks generalize.
-
13
Graph neural networks: message passing, GCN and expressivityA graph neural network updates every node from its neighbours, with the same weights everywhere. Message passing and the GCN layer, why the output does not depend on the node numbering, what GNNs cannot tell apart (the 1-WL test), why sum beats mean and max, and why deep GNNs over-smooth and over-squash.
85 min
5 figures
Advanced -
14
Self-supervised learning: contrastive, non-contrastive and masked modellingSelf-supervised learning makes its training signal from unlabelled data. The InfoNCE and CLIP losses as cross-entropy over a similarity matrix, why they bound mutual information by log N, how SimCLR, BYOL, DINO and MAE avoid collapse, and how to use the learned features (linear probe, k-NN, fine-tuning, zero-shot).
75 min
4 figures
Advanced -
15
Mixture of experts: routing, load balancing and capacityA mixture-of-experts layer replaces one feed-forward network by many experts and sends each token to only a few of them. The top-k router, total and active parameters, router collapse and the Switch load-balancing loss with its gradient, expert capacity and token dropping, the router z-loss, expert parallelism, and balancing without an auxiliary loss.
75 min
4 figures
Advanced -
16
State-space models: from linear recurrences to S4 and MambaA state-space model is a linear recurrence that comes from a continuous-time system. It trains in parallel as a convolution or a scan and generates with a fixed-size state. Discretization, the convolution kernel, stability, S4, Mamba's selection and the associative scan.
80 min
5 figures
Advanced -
17
Generalization: double descent and implicit regularizationWhy a network with more parameters than training examples can still generalize. Gradient descent picks the minimum-norm solution, the test error peaks where the model can just fit the data, and bounds that look only at the model class cannot explain deep learning.
100 min
5 figures
Advanced