Interactive figures
Every figure in the track. Each one is a small experiment: drag, slide, and watch the numbers. The link under each figure takes you to the lesson that explains it.
Lesson 1 · Latent variables and the ELBO
Choose "forward KL", press "Find the best q", then do the same for "reverse KL". The blue curve is the data density ; the orange curve is one Gaussian . Forward KL (what MLE minimizes) gives a wide that covers both bumps and puts mass in the empty gap. Reverse KL gives a narrow on one bump. Then press "track toy ±1": the bumps overlap, has only one mode, and both fits are almost the same.
Click each family, then drag the yellow probe. The data are the track's "8 Gaussians" (eight bumps on a circle of radius 2). The figure shows what each family gives you: a density (blue shading), samples (orange) and the score (violet arrows). The readout says what the family can tell you about the probe point. All values are exact, computed from the known mixture; nothing is trained here.
Drag the observation (yellow), then move the slider . The bottom panel shows the ELBO for every guess under the ceiling . The yellow segment is the gap; the readout shows that it equals the KL divergence from to the exact posterior (the blue bars). The gap closes only at , the vertical blue line. The start is the hard assignment .
Move the proposal (orange) and watch the bound. The model is , , with the observation ; so is known exactly (blue line), and the posterior is . Each orange point is a Monte Carlo estimate of with standard errors. With = the prior, the ELBO () is nats too low, but is within nats. With = the posterior every weight equals , and all are exact. Try "too narrow": the weights have infinite variance and the bound improves very slowly.
Lesson 2 · Variational autoencoders
Move β (or press the "Try" buttons) and watch the blobs. Right: the data (blue), a 2D Gaussian cloud with variances and along two directions, and eight chosen points (yellow). Left: the latent space with the prior (aqua circles at 1 and 2 standard deviations) and the encoder blob of each chosen point (orange). At both latent directions are used. At the threshold exceeds : the second direction collapses, every blob becomes as wide as the prior in , and the model (orange dashed) keeps only the first direction. At both collapse.
Compare the two histograms, then change . Each histogram shows 500 independent gradient estimates; each estimate averages samples. The pathwise estimator averages ; the score-function estimator averages . Both are centred on the exact gradient (dark line). For at the per-sample variances are 4 and 87. For the order flips. For the step function the pathwise estimate is always 0 and wrong: has no useful slope.
Move β, or press "Sweep β". The magenta curve is the set of optimal (rate, distortion) points of the linear VAE of section 2, one for each β. The orange point is the optimum for the current β, and the dashed line through it has slope : the line $D + \beta R = $ const touches the curve there. The yellow ring is , the ordinary ELBO. Large β slides the point to rate 0: both directions collapse (at , here 2 and 6).
Press "Codebook step" a few times, then "Reset dead codes". The blue points are encoder outputs (the track's 8-Gaussians data); the orange dots are the codes, and the thin lines are the Voronoi cells: every is replaced by the code of its cell. From the bad start in the corner, one code wins every point; the others (magenta) get no gradient and never move. Resetting dead codes to random encoder outputs brings them back. Then try "codes from data": every code is used, but two codes can share one cluster. The violet arrows are the commitment pull on .
Lesson 3 · Autoregressive models
Move the sliders to change the seven conditional probabilities of three binary variables. The bars are the eight joint probabilities (one per leaf); their sum stays exactly 1. In evaluate mode, pick an : its three factors light up together in yellow, because all of them can be read from the data at once. Switch to sample and press Next step: now each factor needs the result of the step before. Last, switch on the leak and watch the sum become 2.
Click a column of the two-moons picture. The top chart is , one bar per column; the bottom chart is the conditional of the selected column. Press Sample: each orange point takes two sequential steps, a column from , then a row from . With smoothing the model gives probability 0 to every empty cell; a larger spreads some mass there.
Click a hidden unit to change its degree, and choose an output to see all paths into it. The table of path counts (the product of the masks) must be zero on and above the diagonal; the shaded cells are the forbidden ones. Change the order: the same rule works for any order, with "degree" meaning "position in the order". Switch on the leak (output rule ): the diagonal fills, and each output sees its own input.
Increase the number of layers . The blue region is everything the target pixel can see. The orange pixels come before the target in raster order and lie within steps, but no path reaches them: the blind spot. It grows with . Then switch on the two stacks: the blind spot disappears.
Lesson 4 · Normalizing flows
Drag the yellow knots of the map . The aqua curve on the left is the base density along the axis. The top chart shows the model density in orange, and a histogram of samples with . The yellow probe interval and its image on the axis hold the same probability. Notice that the area under stays 1, and that the bumps appear where the map is steep.
Add layers and move the sliders of and . The orange heat map is the exact model density . The aqua curves are the straight grid lines of the base space, carried to data space by the sampling map : where they crowd together, the density is high. Drag the yellow point: each layer adds its own log-determinant, which is just of the kept coordinate. The grid sum of the density stays 1.
Choose a flow and a task, then press Next pass. One MADE pass computes and for all at once, but entry is valid only when the network inputs are known. Entries that became valid in the last pass are yellow. Try all four cases: MAF density and IAF sampling finish in one pass; MAF sampling and IAF density need passes.
Move the time , or press Play. A linear field (violet arrows) moves samples of ; the aqua points and the grey ellipses (1 and 2 standard deviations) show , which stays Gaussian, so everything is exact. The readouts compare the change of log-density along the magenta path, computed from the Gaussian formula, with . Drag the start point and try the presets: the two numbers always agree, and a rotation () keeps the density along every path.
Choose a Jacobian and a number of probes . The histograms show 2000 estimates of the trace with Gaussian and Rademacher probes; the black line is the true trace. The lower chart shows the variance against on log scales, with the formulas as dashed lines. Notice that both estimators are centered on the truth, that the variance falls like , and that Rademacher probes are exact for a diagonal Jacobian.
Lesson 5 · GANs and Wasserstein GANs
Drag the orange peak (or use the sliders) to move and widen the generator density . The data density is fixed. The black curve is the optimal discriminator : it is where the two densities are equal, close to 1 where only the data have mass, and close to 0 where only the generator has mass. Move the yellow probe and compare with . Choose "far generator": JSD gets close to and is flat near the generator.
Move the logit of a fake sample. Top: the saturating and the non-saturating generator losses. Bottom: the size of their gradients on a log scale. At (the discriminator is sure the sample is fake) the saturating gradient is and the non-saturating one is almost 1. At the two are equal.
Press "Run 1000 steps" and watch the iterates spiral out from the start point along the game vector field. Then switch to "alternating": the path closes on itself. Then choose "simultaneous + R1": the path spirals in. The readouts give the eigenvalues of and the growth factor per step near ; the chart below shows the distance to on a log scale. Drag the start point to try other starts.
Press Play. The real data are the track's 8 Gaussians. Each generator sample moves along of the discriminator logit, the only signal a generator gets. The background shows where says "real" (blue) or "fake" (orange). Then choose the "Gaussian" generator , which is too simple for 8 modes: it collapses to a small blob that travels around the ring. "Freeze D" lets the generator answer a fixed discriminator: the samples pile up at a few maxima of .
Move the shift . The data segment stays at ; the model segment moves. The magenta arrows are the transport plan. In the chart, JSD stays at for every , while shrinks as the segments get closer. Add instance noise : the supports now overlap and JSD becomes smooth too, but it is still almost flat far from .
Lesson 6 · EBMs, score matching, Langevin
Press Play. 2000 particles start at and move with Langevin steps. The top chart compares the orange histogram of the particles with the exact blue density . The bottom chart shows the energy and 80 of the particles on it. Then choose the Gaussian target and make large: with ULA (plain Langevin steps) the histogram is too wide, and the readout gives the theory . Switch to MALA: the bias disappears, but fewer steps are accepted.
Press "1 update" a few times. The energy is , and the blue data come from two bumps. At the start () the model has one bump at 0. The orange chain samples go there, and the arrows show that the update pushes the energy up where there are more chain samples and down where there are more data. Run 100 updates: a double well appears, the moments of data and chains become equal, and reaches the exact maximum-likelihood value (dashed lines in the small chart).
Move the σ slider. At small σ, each violet arrow points to the nearest blue mode; between modes the arrows are almost zero. At large σ, all arrows point to the centre. Now compare two runs from the same uniform start: Langevin at this σ (with σ small) and Annealed Langevin (σ goes from 3 down to 0.05). Mode 1 has weight 0.4 and the others 0.086. The bar chart shows the fraction of particles near each mode: at fixed small σ every mode gets about 1/8; the annealed run gets close to 0.4 for mode 1.
Add samples. Each aqua dot is one noisy point with its DSM target . The targets are very noisy, but their average in each small bin settles on the exact score of . The last readout shows that even the exact score has a positive DSM loss, equal to the constant . Make σ small: the targets spread more (their standard deviation is ), and you need more samples.
Raise the barrier. The energy is , so the barrier height is . The trajectory hops between the wells often for small and rarely for large . The lower chart has a log scale: the exact mean crossing time is close to a straight line, so it grows exponentially in , and Kramers' formula (dashed) is close for high barriers. Press Simulate at a few heights: the simulated averages fall on the exact curve.
Lesson 7 · Denoising diffusion (DDPM)
Move the step (or press Play). The 8 modes shrink toward the center and blur into one Gaussian. The shaded heat map is the exact density of . Switch "How is made" between one jump and running the chain for steps: the path of the yellow point changes, but the cloud of points and the variance readouts agree. This is Theorem 1 below.
One reverse step in 1D, for data with two points . The blue curves are the forward posteriors for each data point: always Gaussian. The black curve is the true reverse step , a mixture of the two blue curves. The orange dashed curve is the best single Gaussian. With "large step" the black curve has two bumps and the KL readout is large; choose "small step" and it is almost Gaussian. Move the observed to see the weights change.
Compare the two schedules. Show , or against . The linear schedule loses half of the signal variance () after about a quarter of the steps (); the cosine schedule keeps it until the middle (). Change : the dashed curve shows what happens if you keep the values of with fewer steps.
Press "Play the reverse process". 1000 samples start as Gaussian noise and move to the 8 data modes; the path of one sample is drawn. The noise prediction is the exact best for this data (computed from the mixture, not learned), so any error comes from the sampler. Choose : with few large steps the Gaussian reverse step is wrong, and samples land between the modes. With or both variance choices work.
Lesson 8 · Diffusion parameterizations
Drag the orange prediction along the dashed line. All points on that line have the same . The blue point is the true pair ; the violet arrow is . The distance between the two points along the line is the error . Its horizontal part is the -error and its vertical part the -error. Move the angle to high noise: the line becomes almost horizontal, so a small -error is a large -error (the orange bar grows). The two bars never exceed 1.
Drag the yellow observation and change the noise level . The magenta curve is the posterior over the clean value, computed on a grid with Bayes' rule. Its mean is the orange line. The violet arrow is the Tweedie step , computed from the exact score. It ends exactly at the posterior mean (compare the two bold numbers). At high noise the posterior covers both modes, and lies between them: the best guess in squared error is an average, not a plausible sample. The lower plot shows the whole denoiser for this . Data: ; the score is exact, not learned.
Pick a target and the way is sampled. The orange curve is the weight of the resulting loss. It is the product of the aqua density (how often training sees each noise level) and the violet target factor (Theorem 2: for , for , for ). The grey line is the ELBO, . Notice that and tilt the weight toward high noise (left), and that min-SNR-5 cuts the weight at low noise (right). The badge says whether the weight decreases in . The linear schedule is drawn in its continuous form, from to .
Move the noise level from left to right. Top: falls from 1 to 0 around , rises like and then stops at , while -prediction (grey dashed, ) keeps growing to 80. Bottom: the effective target has a noise part and a signal part; their shares always add up to 1. The Monte Carlo readout checks with 2000 samples. Switch the data to two points: the variance is still 1, because only matters.
Compare the schedules and change the latent size. Blue and magenta: the channel mean of in training, for a bright image (clean mean ) and a dark one (). Aqua dashed: the channel mean of pure noise, which is where sampling starts. With the Stable Diffusion schedule at the two training curves are clearly apart, so the network learns to read the brightness from . At inference the start is always near 0. Increase : the leak gets worse, because the noise average shrinks like but the signal does not. Zero terminal SNR makes all three curves equal.
Lesson 9 · Reverse SDEs, PF-ODE and samplers
Move the time slider and compare the two histograms with the dashed exact density . Both the ODE particles (magenta) and the SDE particles (orange) follow at every time, and their standard deviations agree with the exact value in the panel. But single paths differ: each magenta path is fixed by its start point, while the orange paths are random. Two orange paths from nearby start points can end at different modes. Press New noise to draw new start points. The score is exact (computed from the known mixture), not learned.
Choose a sampler, an NFE budget and a step schedule. The blue dots are data, the orange dots are 500 samples, the magenta lines are a few sampling paths (drawn in VP coordinates, which stay bounded). Start with Euler at 10 NFE on the Karras schedule: about 85% of the samples land on a mode. Now switch the schedule to uniform: fewer than 15% of the samples reach a mode even at 50 NFE. Try the reverse SDE at 10 NFE (about 60%), then at 50 NFE. Try Heun: below about 15 NFE its second slope is taken where the denoiser changes fast, and it fails; from 20 NFE it works. The score is exact, not learned.
Move , and the two noise levels. The yellow point is the current state in the coordinates . The violet dashed line joins it to (the blue point on the axis ). The Euler step moves along this line to the next level (magenta ring); the DDIM step (orange dot) lands inside the ring, and the panel shows a difference of up to rounding. Now raise : the step becomes random, and its mean leaves the line. Switch to exact, Gaussian data to see the true ODE path (thin magenta curve) and the Euler error.
Compare the two lines on log–log axes. Each point integrates the exact Gaussian-toy ODE from to with Euler (orange) or Heun (violet) and plots the relative error against the NFE used. At large NFE the slopes approach and (dashed reference lines). The yellow rings mark about 30 NFE: there the two errors are still close. At about 250 NFE Heun is more than ten times more accurate: halving its steps divides its error by 4, not by 2. Switch to the uniform grid: both methods need hundreds of NFE before their order shows, because almost all steps are spent at high noise.
Move and . The top panel shows the levels for , for your (orange) and for , on a log axis; the yellow line is . The bottom panel shows the local error that one Euler step makes on the Gaussian toy, at the level where the step starts. With one long last step from to almost 0 makes nearly all of the error. Larger moves levels to low noise, where the path bends, and the errors become more even. Press EDM for the default 18 levels with : 7 of them lie below .
Lesson 10 · Flow matching and rectified flow
Move the time slider in the conditional paths view. Each yellow point moves on a straight magenta line from its noise point (green) to its data point (blue); the violet arrows show its velocity in the sampling direction (toward ). Near the center, close points move in very different directions. Now switch to marginal flow: one smooth velocity field moves every point, the paths bend, and the orange end points land on the modes. Compare the transport costs: about 5.6 for the random pairs, about 1.05 for the start and end points of the marginal flow. Then try minibatch OT for the pairs. The velocity of the marginal flow is computed exactly from the known mixture, not learned.
Drag the yellow point in the plane. The two dashed lines are the conditional paths through it; the darker one has the larger posterior weight. The grey arrows are their velocities, the violet arrow is the weighted average, the marginal velocity. Between the modes (try between the modes) the two lines have equal weight and their velocities cancel. Near a mode at little noise, one line has all the weight and the marginal velocity equals its velocity. The magenta curves are paths of the marginal ODE: they never cross, and they bend where the weights change.
Choose the number of Euler steps. With 1 step every sample (orange) lands exactly at the center, the mean of the data. With 2 steps the samples form a ring of radius about 1.4, inside the data. With 4 steps about 83% are on a mode, and with 8 steps about 97%. The magenta lines are Euler paths. The velocity is exact (computed from the known mixture), so every error you see comes from the step size.
Compare the two flows and move the number of Euler steps. Before reflow, the exact magenta paths from the noise points (green) first head toward 0 and turn late; the orange Euler polylines with 1 step all end at 0 (error 1.0), with 2 steps at (error 0.44). Switch to after one reflow: the paths are straight lines, and Euler is exact with any number of steps. In 1D a flow keeps the order of points, so the rectified pairing sends negative noise to and positive noise to : it is the optimal transport pairing, with cost instead of .
Lesson 11 · Guidance
Move the noise level and watch the noisy classifier (top) flatten as the noise grows; the dashed grey curve is the clean classifier, which is the wrong function to use on noisy inputs. Then move the classifier scale (bottom). The orange guided density equals the blue class density at . For it gets narrower and moves right, and approaches 1. Everything is computed exactly, not learned.
Move the guidance weight . The 8 Gaussians are split into a target class (blue) and another class (grey). The scores are exact, not learned, and the orange samples come from the guided probability-flow ODE. At about half of the samples are in the wrong class; at (exact conditional model) all are right. For the samples move outward past the modes (the mean radius grows above 2). Switch to "two halves" and watch the middle modes of the class take more samples than the edge modes. Switch to the weak conditional model (80% on the class) to see guidance raise the accuracy above what gives.
Move the interval ends and . Both panels use the same weak conditional model (80% of its mass on the class) and the same starting noise; the scores are exact. Left: guidance at every level. Right: guidance only for in the yellow band, and the plain conditional model elsewhere. Notice that guidance at high noise (move above 2) mainly moves samples to the two middle modes of the class: the effective number of modes drops. Guidance only in the middle band keeps the modes balanced and still gives high accuracy, at a lower NFE.
Lesson 12 · Few-step generation
Change the number of Euler steps and the start value. The magenta curves are exact PF-ODE paths, from noise on the right to the blue data points on the left. The orange polyline is the Euler sampler with equal steps in , from the yellow start. With two data points and , every sample lands near : the share of samples within of a data point is . With it is close to the exact ODE. Switch to one point: the paths are straight lines, and one step is already exact. The denoiser is exact, not learned.
Choose the number of student steps , from 64 down to 1. Top: the sampler map from the start noise to the output . The magenta band is the 64-step DDIM teacher. The orange dots are the distilled student with steps. Here each round fits its target exactly, so the student reproduces the teacher with halving rounds. The grey dashed curve is plain DDIM with the same steps and no distillation. At it outputs for every start. Bottom: the Wasserstein-1 distance between the output and the data. Plain DDIM gets worse as falls; the ideal student stays at the teacher's value. Look at the readouts: the student's first prediction is not the posterior mean, it is the teacher's end point. The denoiser is exact, not learned. A real student adds a small fitting error in each round.
Move the yellow point with the two top sliders. The bold magenta path passes through it. The violet dashed lines show the consistency map: the point and the yellow rings on the same path all map to one end, at . Now change the step . In CD mode the teacher takes one Euler step to and lands at the orange point, slightly off the path; its own end differs, and the squared gap is the CD loss term. Halve : the gap falls by a factor of 4 or more, because the local error of Euler is . Switch to CT mode: there is no teacher, and the partner with the same noise depends on the unknown (the bar shows standard deviation; with two data points, the ring size shows the probability). Its mean equals the Euler point. Bottom: and ; at they are exactly 1 and 0. The paths use the exact denoiser, not a learned one.
Lesson 13 · Latent diffusion and DiT
Change the image size, the factor , the latent channels and the patch size . The blue square is the image, cut into tokens (one yellow token is marked). The bars compare the current setting with one token per pixel (grey), on a log scale. Start from SD-like 512: , so tokens. Now double : the tokens grow 4 times and the attention cost grows 16 times. Halve : the same happens. Notice how small the attention share is for DiT at 256 tokens, and how it takes over at high resolution.
Change the latent standard deviation and the step . Top: the effective log-SNR is the designed one (grey dashed) shifted up by . Bottom: the same latent pattern noised at step , for unit variance (left) and for (right); blue is positive and orange negative. With the pattern on the right is still clear at , where the left one is almost pure noise. Try the preset : the two panels agree. The schedule is Stable Diffusion's "scaled linear" schedule with .
Switch between cross-attention and joint attention, and pad the last text token. Each row is a query, each column a key, and each cell is an attention weight; every row sums to 1. In cross-attention the rows are the four image tokens and the columns the three text tokens : the matrix is . Move the key scale of : every image row puts more weight on . The padding mask sets the last column to 0, and the other weights grow so that rows still sum to 1. In joint attention the matrix is , with four blocks; the text rows now also read the image. The readout gives the cost of at the level of Stable Diffusion v1.
Lesson 14 · Evaluating generative models
Pick a model and compare its test NLL with its samples. The blue area is the data density, the orange curve the model, and the orange ticks below are 60 model samples. Junk mixture: only a share comes from the true model and the rest from a broad "junk" Gaussian. With only about a quarter of the samples look like data, but the NLL rises by much less than nats. In dimensions the bound is bits per dimension: invisible in a likelihood table. One mode: perfect samples, huge NLL. Broad Gaussian: the maximum-likelihood fit covers both modes, and many samples fall between them. "Good samples" counts model mass where the data density is at least 5% of its peak; "data covered" the reverse.
Drag the orange mean and change the generated covariance. The blue and orange ellipses show 1 and 2 standard deviations of the real and the generated Gaussian; the dots are samples of each. The exact Fréchet distance splits into a mean term and a covariance term. Now press Same as real: the exact distance is 0, but the estimate from samples is positive, and its mean over many repeats is close to the second-order bias printed below it. Increase : the bias falls like .
Pick a generated set and change . Real data (blue) is the 8-Gaussians toy; generated points are orange. A filled dot lies inside the other set's support, a ring lies outside. Mode collapse covers only three modes: precision stays high, recall drops to about . Too wide does the opposite. Choose outliers and show the balls around the generated points: six far outliers get huge balls, which cover real points and inflate recall.
Move the forecast mean , its spread and the outcome . The magenta area is the squared gap between the forecast CDF and the outcome step; its area is the CRPS, and it matches the closed form and the kernel form. Shrink toward 0: the CRPS becomes . The violet dashed steps are the CDF of one ensemble of members. Over 2000 ensembles the standard estimator is too large on average by about , while the fair estimator matches the exact CRPS. Try : the bias is largest.
Change the spread factor and the bias of the members, or use the presets. The orange bars are the observed ranks of the outcome over many cases; the magenta dots are the exact probabilities, and the grey line is the flat value . Too narrow gives a U, too wide a dome, a bias a slope. The corrected spread–skill ratio is near 1 only for , ; there the uncorrected ratio is about . Use 200 cases: even a calibrated ensemble then gives a bumpy histogram.
Lesson 15 · Inverse problems and assimilation
Move the noise level . Top: the exact noisy likelihood of the observation as a function of , and its DPS and ΠGDM approximations. At high noise (small ) the exact curve is wide and flat, but the DPS curve stays as narrow as the measurement noise allows. Bottom: how many times too large the DPS and ΠGDM likelihood scores are; the ratio is for DPS. Make small (precise sensors) and the DPS error at high noise grows. Switch to : ΠGDM becomes exact. Everything is computed in closed form.
Switch the sampler between the closed-form posterior, the diffusion sampler with the exact conditional score, and DPS. The prior is the 8 Gaussians (blue, exact scores, nothing learned); we observe only the first coordinate, , shown as the yellow band . The exact posterior is again a Gaussian mixture, and the exact-score sampler reproduces it. The DPS samples are too concentrated: compare the standard deviations in the panel. Move down to 0.1: DPS pulls too close to . Set and : the DPS weights of the four plausible modes are wrong.
Switch the sampler. The prior over the 48-step trajectory is a stationary AR(1) process, so the exact posterior is known: its mean (dashed grey) and 90% band (grey) come from the Kalman smoother. The true trajectory is blue and the observations are yellow. The orange lines are samples of the chosen method and its 90% band (dashed). The diffusion sampler with the exact noisy likelihood matches the Kalman smoother. DPS is over-confident: its spread is too small, most at the observations, and its band misses the truth more often than 10% of the time. Change the observation spacing and and compare the coverage numbers.