Interactive figures
Every figure in the track. Each one is a small experiment: drag, slide, and watch the numbers. The link under each figure takes you to the lesson that explains it.
Lesson 1 · Hashing, two pointers, windows
Move the slider for (the input size, on a log scale). Each bar is the number of steps of one complexity class; the dashed line is about one second of Python ( steps). Bars past the line are too slow. Notice how fast crosses the line: at it is a thousand times over the budget, while is far below it.
Press Next step to walk through "count the subarrays with sum ". The row holds the prefix sums. At each step the code looks up in the dict of earlier prefix counts; every earlier match is one subarray with sum , drawn as a bracket. Then switch off "start with {0: 1}" and run again: the subarrays that start at index 0 are lost.
Press Next step to run the window for "shortest subarray with sum target". The right pointer adds one element per step; while the window is valid, the code records its length and the left pointer removes one element. The counters show that each index enters once and leaves at most once. Then choose the preset with negative numbers. Before you step, predict: will the window find the true shortest subarray?
Set the tail length and the cycle length ( means no cycle), then press Next step. In phase 1 the tortoise moves 1 node per step and the hare moves 2. After they meet, phase 2 puts the tortoise back at the head and moves both by 1: they meet at the first node of the cycle. Check the readout: the first meeting is at the smallest multiple of that is at least , and phase 2 takes exactly steps.
Lesson 2 · Stacks, binary search, heaps
Press Next step to run Daily Temperatures. The bars are the temperatures; the blue day is the current one, and the stack (right) holds the days that still wait. When the current day is warmer than the top, the top is popped and gets its answer. Switch on Predict to guess how many days each new day pops before you see it. Watch the counters: pushes plus pops never exceed .
Press Next step. Each cell shows the value of the predicate (F or T). The search keeps and : everything left of is known to be F and everything from on is known to be T. The cell is the one being tested. Move and check that the result is always the first T. Then choose custom : its switches several times. The search still stops after at most calls, but it returns a switch point that is not the first T. Click cells to make monotone and run it again.
Push and pop values. The same heap is drawn as a tree (top) and as a list (bottom); the colours link each node to its list cell, and index has children and . A push appends the value and sifts it up while it is smaller than its parent; a pop moves the last value to the root and sifts it down towards its smaller child. Count the swaps: never more than the height .
Choose merge or rooms, then press Next step. In merge mode the intervals are sorted by start; the current block grows while the next start is at most its end, and a gap closes it into the output. In rooms mode a vertical line sweeps over the start and end events; the running count is the number of rooms in use, and its maximum is the answer.
Pick a greedy rule and compare it with the optimum. The chosen intervals are aqua; the rejected ones are grey. The readout compares the size of the greedy set with the optimum found by trying all subsets. The presets are the counterexamples for "earliest start" and "shortest first"; press Random a few times and notice that only "earliest end" always matches the optimum.
Lesson 3 · Trees and graphs
Choose an order and press Step (or Play). The blue node is the current one, the orange nodes are on the stack (for DFS this is the current path from the root) or in the queue (for level order), and the output grows below. Switch to Predict mode and click the node that you think comes next. Then try the trees "path" and "complete": the stack never holds more than nodes, while the queue of level order holds a whole level.
Pick the bag and step through the same loop. Blue marks the node just taken out, aqua nodes are visited, and the thick edges form the search tree (each node is joined to the node that put it in the bag). The panel shows the bag and the order. Compare the depth of node in the three trees: DFS reaches it after 6 edges, BFS after 3, and Dijkstra finds the cheapest path, of weight 8. "Skipped" counts entries that were taken out but were already visited.
Run Kahn's algorithm on an ML pipeline. The number above each task is its current in-degree; tasks with in-degree 0 wait in the orange queue, and output tasks turn aqua. Then switch on the edge report → preprocess: it closes a cycle, the in-degrees on the cycle never reach 0, and the output stops after 1 of 6 tasks.
Step through the script, or click two nodes to union them (click one node and press find to
run find on it). The arrows are parent pointers; roots have a ring. Switch the two rules off and
run the script "chain": the tree becomes a path and find(0) follows 7 pointers. With union by rank
the same unions build a flat tree; with path compression, every find also flattens its path.
Compare the counter of pointer steps.
Step through Dijkstra. The aqua nodes are settled (taken out of the heap, distance final); the number in each node is its current distance. Click an edge label, then move the slider to change its weight. The preset "negative edge" gives the weight : Dijkstra settles too early, and the panel compares its result with Bellman-Ford, which is correct.
Lesson 4 · Backtracking and DP
Step through the search. On the board, the blue queens are the current
partial solution and the magenta squares are attacked. In the tree, each node
is a partial solution (the edge label is the column of the new queen); the path from the root to
the blue node is cur,
aqua leaves are solutions, and the counters show how many squares were tried
and pruned. Switch to "subsets of {1, 2, 3}" to see the plain template without pruning: 15 nodes,
8 leaves.
Move the slider and switch the view. Node is the call , and nodes with the same argument have the same colour. "With a memo" shows the calls that really run: repeated calls return from the cache (dashed). The tree has calls ( = Fibonacci numbers), which grows like ; the DAG has only states and edges. A memo turns the tree into the DAG.
Fill the table cell by cell. Each cell shows , and its colour says which case won: aqua = match (diagonal, free), orange = substitute (diagonal, +1), blue = delete (from above, +1), violet = insert (from the left, +1). Turn on Predict to type each value before it is shown. When the table is full, Traceback draws the path of one optimal alignment. Edit the two words to try your own.
Predict where each new value goes, then check. Click the slot of tails that the new value
replaces, or the empty slot at the end to append it. The
orange cells are the entries that binary search compared with , and the
aqua cell is the entry that changed. Notice that a small value can
replace an early entry without changing the length: it only makes future extensions easier.
Lesson 5 · Practical multi-part coding
Press "Next" to apply one operation at a time. The violet node is the one
the last operation touched; it always moves to the most-recently-used end (right). When the size
becomes , the node right after head is evicted (magenta). Turn on
Predict first to guess each result before you see it, and use the examples or edit the operations.
Step through the script with "Next". The data dict on the left is always the current view. In the undo stack, orange entries were just recorded (first touch). At a nested commit, aqua entries are copied into the parent and magenta ones are dropped, because the parent already has the key. Switch to commit rule for the shortest case, or to free play and try your own sequence.
Click the time line to add requests, and drag the orange handle to move the window. The blue line is the token level : it rises with slope up to and drops by one for each allowed request; rejected requests are crosses. The panel counts the allowed requests in the window and in the worst window of the same length. Try to beat the bound : you cannot. Bound reached shows a case where the count equals it.
Lesson 6 · ML coding in NumPy
Move the logits, or pick an example. The left half of the table is the naive computation
in float64; the right half first subtracts . With large logits the
naive form gives inf/inf = nan; with very negative logits it gives 0/0 = nan. The shifted form never
fails. Try with class 2: the naive cross-entropy is inf, but
gives the right value, about 1000.
Press +10, +100 and +1000 to run gradient descent. The violet line is the decision line ; the dashed lines are where (blue) and (orange). On separable data with the loss keeps falling and keeps growing, so the dashed lines close in on the decision line. Set and run again: stops growing. Switch to overlapping data: now a finite minimizer exists even with .
Press "Next" to run one half-step at a time, and watch in the chart. Each point takes the colour of its cluster; thin lines join it to its center. goes down or stays the same after every half-step. The bad start puts one center between two blobs and two centers in the third blob: Lloyd's algorithm stops in a poor local optimum. Try k-means++ seeds, or drag a center yourself. The worked example is the line example of section 4.
Drag the scores and watch three things agree. The grid has one cell for each (positive, negative) pair: 1 if the positive scores higher, ½ on a tie, 0 otherwise. The shaded area under the ROC curve, the grid sum divided by , and the rank formula always give the same number. Move a positive onto a negative to make a tie: the ROC curve gets a diagonal step.
Lesson 7 · Training loop, AdamW, BPE
Move the gradient scale from to . The loss is : the same valley, only steeper or flatter. The orange path is SGD with a fixed learning rate: it barely moves for small and explodes for large . The blue path is Adam: it is the same for every . Then turn bias correction off: the first Adam steps become several times too long (3.2 times at step 1).
Change the token counts of the micro-batches. Each box is one target token; its width is the weight that token gets in the loss. The top row is the mean of the micro-batch means; the bottom row is the correct rule, sum of token losses divided by the total number of tokens. With equal counts the two rows agree. With unequal counts, tokens in the small micro-batch get too much weight, and the loss (and its gradient) is wrong.
Set and look at the bf16 row. The gap between neighbouring bf16 numbers near 1 is . An update of to a weight of 1 is less than half that gap, so in bf16 it is rounded away: the weight never changes. This is why the optimizer keeps fp32 "master" weights. Then try 1e−8: fp16 underflows to 0, bf16 does not. (Colours: bf16, fp16, fp32.)
Press Merge to learn one merge at a time. The table counts each neighbouring pair, weighted by how often its word occurs. The top pair becomes a new token; a tie is marked and broken by the rule "the lexicographically greater pair wins". Then type a new word, such as slots or owl: it is encoded by applying the learned merges in the order they were learned.
Change the gradient size of weight 2. The chart shows how much the decay term alone has shrunk each weight (the run with decay minus the run without; the figure writes the learning rate as ). With AdamW, both weights shrink by about the same amount, . With Adam + L2, the weight with no gradient (solid) is pulled down fast, and the weight with large gradients (dashed) is decayed much less.
Lesson 8 · Transformer decoder from scratch
Click a row to pick a query; then turn the causal mask off and on. Left: the scaled scores; right: the weights (each row sums to 1). The output of the selected query is drawn as the weighted average of the value vectors. With the causal mask, row has zeros right of the diagonal: token cannot see later tokens. The first token can see only itself, so its output is exactly its own value vector. Move the scale to 0: every visible weight becomes equal.
Switch between the two head splits. Each cell is one feature, coloured by its token
(token 1, token 2, token 3)
and labelled with its feature index. With the bug, view(B, H, T, d_h), the memory is cut into the
right shape but the wrong pieces: rows (magenta frames) hold the wrong token, often a later one. The
shape is the same, so only a test against a reference finds this bug.
Press "Shift both by +1" several times. The query and the key turn together, the angle between them stays, and the score in the log stays the same. Then press "Swap m and n": the offset changes sign and, for most and , the score changes, so RoPE can tell "before" from "after". The slider is the frequency of this pair.
Press Next step and compare the two masks. Rows are query positions, columns are key
positions. Blue cells are computed in this step; faint cells were computed
earlier and are not recomputed; the aqua band marks the cached keys. With
is_causal=True the triangle starts in the upper-left corner of the score matrix,
so the new token sees only key 0, and the keys it must see are hidden (magenta ✕).
Switch to tril(diagonal=P): the new rows match the full forward pass. Try 2 new tokens per step.
Predict first: with and , which tokens survive top-p? Then check the table. Faint blue bars are ; solid orange bars are the final distribution after the filters, renormalized. Lower the temperature: the nucleus shrinks to one token. Pick flat and peaked and compare top-k with to top-p with : top-p keeps many tokens for the flat row and one for the peaked row.
Press Next step until the search stops, then switch . The toy model has the tokens , and the end token . Blue = kept and live, aqua = kept and finished, faint = pruned, orange = the winner. Greedy decoding takes first and never sees , the best prefix of length 2. With the short "" wins; with the longer "" wins. Try beam width 1: it is greedy decoding.
Lesson 9 · Classic ML and A/B tests
Move the degree slider and watch the 20 faint fits, each trained on a new random training set of 15 points. The thick orange curve is their average, and the dashed curve is the true function. At degree 0–1 the average is far from the truth (bias). At degree 9–10 the average is close to the truth, but the single fits swing wildly (variance). The lower chart shows bias², variance and the noise for every degree; their sum is the expected test error, a U-shaped curve. The average training error only falls as the degree grows.
Drag the centre of the ellipses (the unregularized solution) and shrink the radius. The L1 solution (diamond) often sits on a corner, so one weight is exactly 0. The L2 solution (disk) almost never lands on an axis. Ask yourself: why does the corner win so often?
Move the prevalence slider from 50% down to 0.5% (the fraud preset) and keep the threshold. The ROC curve and its AUC do not move: TPR and FPR are each computed inside one class. The PR curve, the precision and the number of false alarms per caught positive change a lot, because precision compares the two classes in absolute numbers. Then move the threshold and watch precision and recall trade against each other. The areas right of the threshold are the true positives and the false positives.
Find the number of users that gives 80% power. The blue curve is the distribution of the observed difference when there is no effect; the orange curve, when the true lift is the MDE. The test rejects outside the dashed lines. Power is the orange area outside them. Move until the power reads 0.80, then halve the MDE and see how far must move.
Press "Run 2,000 A/A tests" with 1 look, then with 10 and 20 looks. Both arms are identical, so every rejection is a false positive. A path turns orange when it crosses at any look, and the experimenter stops there. The lower chart compares the simulated false-positive rate with the planned 5%.
Lesson 10 · Deep learning breadth
Move the gain below and above 1. The plot shows the expected size (RMS) of the signal after each block, on a log scale. In a plain stack it is : it vanishes or explodes. In a residual stack the branch adds to an identity path, so the signal never vanishes, but it grows like . Scaling each branch by (GPT-2 style) or normalizing its input (pre-LN) keeps the growth slow.
Start with the preset of predict 1: width 512, 12 layers, weights with variance , ReLU. The blue line is the second moment of the activations in each layer (from a real random network), the dashed line is the theory. Switch to He () and the line becomes flat. The orange line is the second moment of the gradient in the backward pass: it shrinks or grows in the same way.
Start with 1 Euler step: every sample lands on 0, the mean of the data, and none on . Increase the number of steps and watch the samples split into the two data points. The faint lines are a few straight training paths from noise to data; they cross, so the learned field (arrows) is their average and the exact ODE paths curve. Try the uneven data: one step lands on the mean 0.4.
Switch between cosine and WSD, then press "extend the run by 50%". A cosine run must decay again from a high learning rate, because its decay depends on the planned length. A WSD run continues its stable phase and only moves the final decay, so the old checkpoints stay useful. Move the step slider to read .
Read the loss curve and pick the most likely cause before the figure explains it. Go through all six cases. The blue curve is the training loss and the orange curve is the validation loss.
Lesson 11 · LLM breadth
Move the model size along one budget. The blue curve is the Chinchilla loss fit with tokens, so every point costs the same compute . The orange dot is the minimum of the fit, and the aqua dot is the "20 tokens per parameter" rule. The preset is predict question 1: 10 B parameters on 20 B tokens. Notice how far up the U it sits, and how flat the bottom is.
Shorten and lengthen the leash with . Faint blue bars are the reference policy ; solid orange bars are the optimal policy . Edit the rewards and the reference probabilities. Try "reward hack": one answer has a high proxy reward but a tiny reference probability. Watch how small must be before moves to it, and how large the KL becomes.
Choose the rank budget . The target update (left) and its best rank- approximation . Move and watch the error and the parameter count against . When the singular values decay fast, a small already captures most of ; this is the bet that LoRA makes.
Click an answer to flip its reward between 1 (correct) and 0 (wrong). The bars are the advantages: blue pushes the answer up, orange pushes it down. Make all answers correct (or all wrong): every advantage becomes 0, and the prompt gives no learning signal. Switch to "no std" (Dr. GRPO) to see that only the scale changes.
Change , and . The blue curve is the unbiased estimator for every , and the orange curve is the plug-in . At they agree. For larger the plug-in is lower, and at the unbiased estimator is exactly 1 whenever .
Lesson 12 · Training at scale
Make a 7B model fit on 80 GB GPUs. With DDP (stage 0) the model states alone take 112 GB: the bar crosses the 80 GB line. Switch to ZeRO stages 1, 2 and 3 and watch the optimizer states, then the gradients, then the parameters shrink by the number of GPUs. Then try "standard attention" at : the activations explode, and ZeRO does not help them. Tensor parallelism () and full checkpointing do.
Step through a ring all-reduce on 4 GPUs. Press Next step. In the first 3 steps (reduce-scatter) each GPU sends one chunk (orange outline) to the next GPU, which adds it: the small counts the GPUs already in the partial sum. After step 3 each GPU holds one complete chunk. The next 3 steps (all-gather) copy the complete chunks around the ring. Watch the counter reach times the vector size.
Shrink the bubble. Each row is a stage; blue boxes are forward passes, orange boxes backward passes (twice as long), numbered by micro-batch; the grey gaps are the bubble. With , the bubble is . Raise to 16: it falls to . Then switch to 1F1B: the step time stays the same, but stage 0 now holds the activations of at most micro-batches instead of .
Move a matmul along the roofline. The blue roof is the best possible FLOP/s at each intensity. The orange point is a matmul. With it sits at , far left: memory-bound, less than 1% of peak. Slide up: the point climbs the slanted part and reaches the flat roof between and on an H100 (the dashed ridge, ). Compare A100 and H100: the H100 needs a larger .
Find the best checkpoint interval. The writing cost falls with , the lost work grows with ; the total has its minimum where they cross, at the violet dot . Press Simulate failures: the magenta dots come from a Monte Carlo run with random failures and follow the formula. Now move from 1,024 to 16,384 GPUs (): the part of the overhead grows by , the square root of 16 (the restart part grows ).
Lesson 13 · LLM serving
Find the batch size that pays. At the throughput is low: the step reads 26 GB of weights for one token. Slide up: the throughput grows almost in proportion, while the time per output token grows slowly. Then the curves bend, because every sequence adds its own cache reads. The magenta wall is where the KV memory is full. Switch to the 8B GQA model on an H100: 5 times less cache per token moves the wall far to the right, and with a short context a violet line appears where the step becomes compute-bound.
Compare the two schedulers. Each bar is one request in a batch slot; grey is an idle slot. With static batching, a short request leaves its slot idle until the longest request of its batch ends, and the next batch waits. Switch to continuous: every freed slot takes the next request at once. Watch the total steps, the slot utilization and the mean wait before a request starts.
Fit more requests in the same memory. Each square is a block of 16 token slots; a colour is one request, and the filled part of a square holds real tokens. With "reserve max length" each request takes 32 blocks, mostly empty, and only a few requests run. Switch to paged blocks: many more requests fit, and each one wastes less than one block. Then share the system prompt: its grey blocks are stored once for all requests.
Choose the draft length. The bars are the expected tokens per target pass; they approach as grows. The speed-up line includes the cost of the draft passes, so it has a best . Lower to 0.5: long drafts stop paying. Press Simulate to check the formula with 10,000 random passes.
Push the server toward full load. The service time is 20 ms, so the server can do at most 50 requests/s. At 40 requests/s () the M/M/1 mean latency is already 100 ms, five times the service time. Move to 45 and 48 and watch the curve explode near . Then pick "10 ms" (a model twice as fast): the latency at 40 requests/s falls 6 times. Press Simulate to check Little's law on 20,000 random requests.
Lesson 14 · ML system design
Move the sliders and watch the compute of each stage. With the defaults, the funnel needs a few GPUs, while ranking every item for every request needs hundreds of thousands (orange bar). Set the search to exact scan: retrieval without an approximate index becomes the most expensive stage. Raise the candidates : the ranker cost grows linearly with .
Reorder the list with the arrows and watch DCG, NDCG and recall. The bars show what each position adds to DCG: this order against the best order (grey). Swap the top two items: the change is large. Swap the last two: the change is small, because of the discount . Switch the gain to "rel" and compare NDCG with the exponential gain.
Compare the two training losses. Users (violet arrows) and items (aqua dots, larger = more popular) live on a circle; a user likes nearby items and popular items. The figure trains the scores with SGD on the in-batch softmax. With the plain loss, the learned probabilities (orange) are too low for the popular items 1 and 2 and too high for rare ones. Switch on the log-Q correction: the learned bars (blue) match the true preferences (grey). Set the skew to 0 (all items equally popular): both losses agree.
Simulate a click log. The old ranker almost always shows items in the same order (A first, E last). Compare the true relevance, the naive click rate and the IPS estimate. The naive rate calls A the best item; IPS finds E. Set the position bias η to 0: both estimates agree. Switch to random order: the naive rate keeps the right order but is too small for every item. Try 2,000 sessions: IPS is unbiased but noisy, most of all for items that sit low on the page.
Keep the shift at 0 and look at the violet histogram: the PSI of 400 windows without drift. With , more than a third of them cross 0.1. Raise the window size and set : the whole histogram moves towards 0. Then add a small true shift (0.2σ) and compare the PSI of this window (orange line) with the no-drift histogram. With large samples, the KS p-value is tiny even for small shifts.
Lesson 15 · GenAI system design
Find the stage that dominates TTFT. With the defaults (a 14.8 B model on one H100), the prefill and the query rewrite are the largest parts. Raise the prompt tokens from 3,750 to 30,000 (many chunks or a long history): prefill grows faster than linearly, because its attention part grows with the square of the length, and it passes the 1.5 s target. Switch the retrievers to in series: the change is small. Compare the first token with the last one: decode time is the output length times the TPOT.
Move the answer (magenta) along the document and watch the chunks. A chunk that contains the whole answer is aqua; a chunk that cuts it is orange. With overlap 8, an answer of 9 tokens always fits in one chunk (Result 1); make it 12 tokens long and find a position where every chunk cuts it. Raise the overlap: the answer fits again, but the number of chunks and the index size grow.
Change and watch the fused list. Document A is first for BM25 and tenth for dense retrieval; document B is third in both. At , B wins: agreement beats one strong vote. Lower to 1: C (first for dense) and A jump to the top. Switch to sum of raw scores: the fused list is almost the BM25 list, because BM25 scores are ten times larger than cosines.
Start with no checks (verifier recall 0): the orange curve is . With and 10 steps, only 60% of the tasks succeed. Now set the verifier recall to 0.8 and allow 2 retries: the blue curve rises close to 1 for 10 steps, and the cost is only about 4% more attempts. Lower to 0.85: even a good verifier cannot save a 40-step task.
Raise the guidance scale for class B (orange). At the samples cover all four components (unconditional). At they cover the two B components (conditional). At and they crowd into the B component that is farthest from class A: almost all samples are "on target", but one of the two modes is lost and the spread shrinks. Every step costs two denoiser evaluations.
Lesson 16 · Behavioral and research talk
Drag the four boundaries (or focus a handle and use the arrow keys) to split a STAR answer into Situation, Task, Action, Result and Reflection. The preset is a common mistake: one minute of context. Move time into the Action until every part is inside its recommended range. Then try a 2-minute and a 3-minute answer.
Tick the themes that each story can answer. The bottom row counts the stories for each theme; a theme with fewer than two stories is marked. The right column counts the themes of each story; more than three is marked. The preset is a draft bank with two problems. Fix both by changing as few cells as possible.
Reorder and resize the blocks of a talk. Select a block, then use ↑ ↓ to move it and − + to change its length. The bar shows the talk; the orange line marks the minute when the audience first hears the main result. The preset is a common mistake: 15 minutes of background and the result at the end. Fix it, then switch to the 5-minute talk.