15 lessons · about 23 hours · 62 figures
Generative Models
A generative model learns a probability distribution from data and then draws new samples from it. Start from maximum likelihood and the ELBO, see every model family on small 2D examples in the browser, and build up to diffusion, flow matching, guidance, evaluation and generative models for science.
For students, engineers and researchers who know probability (Gaussians, Bayes' rule, maximum likelihood), the KL divergence and the basics of neural networks and gradient descent, and want to understand modern generative models well enough to derive, implement and judge them.
Quick recap
The key takeaways of all lessons on one page
All interactive figures
Every figure of the track on one page
Videos and books
The best free courses
Glossary
Every technical term of the track with a short definition
Concept map
How the lessons of this track connect to each other and to other tracks.
Keeps every lesson page and figure of this track in this browser, so you can read it without internet.
Part 1
Likelihood and latent variables
What a generative model is, maximum likelihood as a KL divergence, the ELBO and variational autoencoders.
-
01
Generative models, latent variables and the ELBOA generative model is a probability distribution that we can sample from. Maximum likelihood fits it by minimizing the forward KL divergence. Latent variable models are easy to sample but hard to evaluate, so we train them with a lower bound, the ELBO, whose gap is a KL divergence to the posterior; EM, amortized inference and the IWAE bound follow.
75 min
4 figures
Intermediate -
02
Variational autoencoders: reparameterization, posterior collapse, β-VAE and VQ-VAEA VAE is a latent variable model with a neural encoder and decoder, trained on the ELBO. The reparameterization trick gives low-variance gradients, the Gaussian KL has a closed form, and the decoder variance sets the balance between rate and distortion. Posterior collapse, the β-VAE and VQ-VAE follow from this balance.
95 min
4 figures
Intermediate
Part 2
Models with an exact likelihood
Autoregressive models factorize the density with the chain rule; normalizing flows transform it with invertible maps.
-
03
Autoregressive models: the chain rule, MADE, PixelCNN and transformer language modelsThe chain rule of probability turns density estimation into a sequence of classification problems. Exact likelihood in one pass, sampling in D passes, and the masks (MADE, PixelCNN, causal attention) that make it work.
70 min
4 figures
Intermediate -
04
Normalizing flows: change of variables, coupling, autoregressive and continuous flowsAn invertible network maps data to a Gaussian, and the change-of-variables formula gives the exact likelihood. Coupling layers (RealNVP, Glow), autoregressive flows (MAF, IAF) and continuous flows (FFJORD) are three ways to keep the log-determinant cheap.
90 min
5 figures
Intermediate
Part 3
Adversarial and energy-based models
GANs learn from a discriminator; energy-based and score models learn an unnormalized density or its gradient and sample with Langevin dynamics.
-
05
GANs and Wasserstein GANs: the optimal discriminator, training dynamics and A generator learns to sample by fooling a discriminator. The optimal discriminator reads the density ratio, the game minimizes the Jensen–Shannon divergence, and plain gradient play can cycle. The Wasserstein-1 distance with a Lipschitz critic gives useful gradients where JSD gives none.
95 min
5 figures
Advanced -
06
Energy-based models, score matching and Langevin dynamicsAn energy-based model writes a density as with an unknown constant . Langevin dynamics samples it with the score , maximum likelihood trains it with model samples, and denoising score matching learns the score directly. Many noise levels fix the failures of a single score, which leads to diffusion models.
100 min
5 figures
Advanced
Part 4
Diffusion and flow matching
Add noise step by step, learn to remove it, and sample by solving a reverse SDE or ODE; flow matching learns the ODE directly.
-
07
Denoising diffusion (DDPM): forward process, forward posterior, ELBO and the noise-prediction lossDDPM adds Gaussian noise in many small steps and learns to remove it step by step. The forward process has closed forms, the ELBO splits into one Gaussian KL per step, and each KL is a weighted error of a noise prediction; dropping the weights gives the simple loss used in practice.
95 min
4 figures
Advanced -
08
Diffusion parameterizations: , , and the score; SNR weighting; EDM preconditioningA diffusion network can predict the noise, the clean sample, the velocity v or the score. At the optimum these are one function seen through linear maps (Tweedie's formula). The real differences are the loss weight over the log-SNR, the scale of the target and how errors grow; EDM preconditioning and zero terminal SNR follow from this.
100 min
5 figures
Advanced -
09
Reverse SDEs, the probability-flow ODE and samplersA diffusion model generates data by solving a differential equation backward from noise to data. The reverse-time SDE and the probability-flow ODE have the same marginals; DDPM, DDIM, Euler, Heun and DPM-Solver are ways to discretize them. Learn why DDIM is Euler's method on the ODE, how to count network evaluations, and why the noise levels of a good step schedule crowd near zero.
115 min
5 figures
Advanced -
10
Flow matching and rectified flowFlow matching trains the velocity field of an ODE from noise to data by plain regression on straight lines between data and noise pairs, with no ODE solve during training. The network learns the average of these straight-line velocities, which bends the paths. Learn why the conditional loss has the same gradient as the true one, how flow matching relates to diffusion, and how reflow straightens the paths so that a few Euler steps are enough.
100 min
4 figures
Advanced
Part 5
Control, speed and scale
Steer samples with guidance, generate in one or a few steps, and run diffusion in a latent space with transformers.
-
11
Guidance: classifier guidance and classifier-free guidanceGuidance steers a diffusion model toward a condition at sampling time. Bayes' rule splits the conditional score into the unconditional score plus a classifier gradient; classifier guidance trains that classifier on noisy inputs, and classifier-free guidance gets it from one network trained with condition dropout. Guidance trades diversity for fidelity, and the guided samples do not follow the "tilted" distribution one might expect.
75 min
3 figures
Advanced -
12
Few-step generation: progressive distillation, consistency models and adversarial distillationA diffusion sampler needs many steps because its ODE paths bend where the data has several modes. Few-step methods learn longer steps (progressive distillation), learn the end point of every path (consistency models), straighten the paths (reflow), or give up the path and match only the output distribution (adversarial distillation). Each one fails in its own way.
80 min
3 figures
Advanced -
13
Latent diffusion and diffusion transformers: autoencoders, the scale factor, cross-attention and DiTLatent diffusion first compresses the data with an autoencoder and then runs diffusion on the small latent, decoding once at the end. This lesson covers the downsampling factor, the latent scale factor and its effect on the SNR, three ways to feed a condition (concatenation, adaLN-Zero, cross- or joint attention), and the diffusion transformer with its token and FLOP counts.
70 min
3 figures
Intermediate
Part 6
Evaluation and science
Measure generative models with likelihoods, sample-based distances and proper scoring rules, and use diffusion priors to solve inverse problems.
-
14
Evaluating generative models: bits per dimension, FID, KID, precision and recall, CRPS and calibrationA generative model has no single accuracy. This lesson covers three kinds of evaluation and their traps - likelihoods in bits per dimension, sample-based distances in feature space (FID, KID, precision and recall), and proper scores and calibration checks for forecasts (CRPS, energy and variogram scores, rank histograms, the spread-skill ratio).
110 min
5 figures
Advanced -
15
Diffusion models for inverse problems and data assimilationA diffusion model can serve as a learned prior for inverse problems. The posterior score splits into the prior score plus a noisy likelihood score, which is intractable in general. DPS and covariance-corrected methods approximate it, with known biases; the linear-Gaussian case is exact and shows them. Score-based data assimilation applies the same idea to whole trajectories, and calibration tests check the result.
100 min
3 figures
Advanced