Interactive figures
Every figure in the track. Each one is a small experiment: drag, slide, and watch the numbers. The link under each figure takes you to the lesson that explains it.
Lesson 1 · Reproducible runs and data contracts
Click a stage of the loop (or focus it with Tab and press Enter). The panel says what can go wrong there without any error message, and which lesson of this track covers it. The orange stage is the selected one. The small box in the centre is the model code: one part of the system among many.
Press "Sum in 200 new orders": the same numbers are added in 200 random orders. Each bar counts how many orders gave a result at that distance from the median result, measured in units in the last place (ulp) of the result. Notice: the results differ in the last digits. Switch to float64: the relative spread (in the panel) becomes hundreds of millions of times smaller (float64 keeps 29 more bits), but it does not become zero.
Choose a method, then press "Add 20% more keys". Each square is one key, coloured by its split: train, validation and test. The strip below shows the number that decides the split: for the hash split, and the position in a random permutation for the seeded random split. A magenta frame marks an old key that changed its split. With the hash split no old key moves; with the random split most old test keys move. "New salt" draws a new hash split and compares it with the old one.
Change the contract with the sliders and watch the violations. Cells that break a rule are
magenta. Then switch the unit of pga_g from to m/s². The 8-row table
shows the same six violation groups, and in a batch of 400 records almost every row still passes
the range check (its maximum is the aqua line). Only the histogram of
values against the reference batch (blue) shows that the
current batch (orange) moved by about 0.99.
Lesson 2 · Testing ML systems
Draw a few suites of 20 random cases, then show where the test fails. The map is the model's probability of rain (darker blue = higher). Each case is a short arrow from to ; it fails (magenta) when the probability drops by more than 0.01. The "network" has a small dip near , °C that covers about 2.4% of the inputs. A suite of 20 cases finds it only about 38% of the time. Switch to "logistic": there the sign of settles the question for all inputs.
Move and the true failure rate , then run 1,000 suites. The violet curve is the exact bound ; the dashed line is . The two agree for . When the magenta line of lies above the bound, fewer than 5% of suites pass (blue squares). With and , about 4.9% of suites still pass: this is the 95% level at work.
Start with the worked example, then try the other presets. Blue bars are the old model, orange bars the new one; the aqua dashed line is the old alpine accuracy minus . Overall the new model wins, while the alpine slice loses 10 points. Move the slider "disagree (b + c)/n": the drop stays the same, but McNemar's p-value changes. Fewer disagreements mean a clearer verdict.
Click tests to change their state and watch the final score. Each test cycles through not done, manual (0.5) and automated (1). The section with the lowest sum has a magenta frame: it alone sets the score. Raise any other section and nothing changes. Compare the final score with the mean.
Lesson 3 · Containers and CI
Predict, then press "Build". Choose what changed since the last build and the order of the
Dockerfile. Before you build, guess how many steps run again. After the build, reused steps are
blue (cached) and steps that run again are orange
(rebuilt); the magenta mark is the first cache miss. Notice: with the
lockfile copied first, a code change rebuilds only the last copy; with COPY . . first, it
reinstalls every package.
Choose the host driver and the CUDA version of the image. The stack shows where each part comes from: blue = host, orange = image. The verdict below applies NVIDIA's compatibility rules. Try a CUDA 12.8 image on driver 535, then a 13.0 image on driver 570, then turn on the forward-compatibility package and switch the GPU type.
Choose a trigger, then make one job fail. Boxes are jobs; arrows are needs: edges.
Blue jobs pass, the magenta job fails, grey jobs are
skipped or do not run on this trigger. The bars below show when each job runs; the dashed
aqua line is the wall time, the end of the longest chain of needs. Turn the
caches off and see how much time they save.
Lesson 4 · Kubernetes and the cloud
Press "2. Retry" in each write mode, then move the crash point. The next task should read the 1,200 rows of the day once. With append, the parts of the failed attempt are read again (magenta: duplicates). With overwrite parts by name, the retry writes only 6 parts, so parts 6–8 of attempt 1 stay and are read (stale data). Only with attempt prefix + manifest does the reader see exactly the blue parts of the successful attempt; the manifest is written last and names them.
Watch the Pods, then change the probes. Each container loads its model for 90 s. When the liveness probe checks the model and there is no startup probe, the kubelet restarts every container after 30 s of failed checks (magenta: back-off before the next restart), so no Pod ever becomes ready. Switch liveness to "only the process", or add a startup probe with a 120 s budget: Pods load, become ready and get traffic from the Service. A 60 s budget is too short and gives a restart loop again. Press "Kill a Pod" to see the control loop replace it.
Pick a load curve and change the tolerance and the window. The blue line is the replica count; the grey dashed line is , what a perfect autoscaler would run. The lower chart shows the per-Pod metric against the target and the tolerance band. In "step", notice that the HPA stops at 5 Pods with : is inside the tolerance. Set the window to 0 with the "noisy plateau" to see the replica count change much more often.
Move the sliders, then press "Simulate". The violet curve is the formula (plus ), and the dashed grey curves are its two terms. The green line marks . The orange dots are a Monte Carlo simulation with random interruptions. Notice that the minimum is flat: intervals from about to cost only a little more. Add nodes and see and shrink.
Lesson 5 · Serving and queueing
Watch the queue, then move . Orange dots are requests waiting in front of the replica; the chart shows the number of requests in the system over time, with the theory mean . At , () long queues build up in bursts. Set (): the queue is short and empties often. The table compares 400,000 simulated requests with the M/M/1 formulas of section 3.3.
Move the operating point and read the latency. The blue curve is the mean latency, the orange curve the p99 (for exponential service times, M/M/1), and the green line the SLO. Notice the factor in the readout: from to every latency doubles. Set (every request takes exactly the same time): the mean waiting time halves, but the wall at stays.
Switch the sample, then move the fan-out. Left: the latency histogram with the p50, mean, p95 and p99. In the 97/3 example the mean sits between two groups and describes no request. "Two replicas" shows that the mean of two p99 values is not the p99 of all requests. Right: the curve ; notice how fast it rises for .
Use the presets, then change and the delay. The chart shows the p99 latency against the arrival rate, for your batcher and for no batching (one request per call). The grey dashed line is the capacity , the green line the SLO. Without batching the replica saturates near 95 requests/s; with it handles 400/s at a p99 under 30 ms. Compare the simulated mean batch size with the fixed point . At "night", set the delay to 0: the latency drops by about the delay.
First pick your guess, then read the bars. Each bar is the p99 latency of one replica when the peak traffic is split over replicas: blue meets the SLO, magenta misses it. Your guess is the orange dot. Then change the SLO or the service time, predict again, and check. Notice how much the answer changes when the SLO comes close to .
Lesson 6 · Registry and canary releases
Run the gate on v4, then on v5, then roll back. The blue row is the champion that serves users; orange rows are candidates in staging; grey rows are archived; magenta rows were rejected or rolled back. The table shows each gate rule. v4 is better on the main metric but fails on heavy rain, so the pointer does not move. Then set "Old artifacts" to "deleted" and roll back: without the old artifact there is no way back.
Switch between strategies and move . The orange line is the share of traffic served by the new version. A bug starts at ; the aqua line is its detection; the magenta area is the harm (requests served by the buggy version); set the vertical axis to 0–10% to see small shares. With "alert after a fixed delay", a 1% canary hurts 100 times fewer requests than blue-green. Now choose "after N requests": if detection needs data, every strategy serves about bad requests, and the canary's advantage is the small blast radius.
Raise the canary share 1% → 5% → 25% → 100%. Each square is one user, coloured by the version that served the last request: blue stable, orange canary. With hash routing, orange squares stay orange. Now switch to "random per request" and press "Next request" a few times: users flicker, and a magenta ring marks users who went back from the canary to stable. "New salt" draws a new, unrelated canary group.
Move and watch the two curves separate. Blue: under ; orange: under ; aqua: the cut-off ; magenta area: false alarms ; orange area: power. At the power is 0.80, as the theorem below says. Change the canary share to see the run time, and press "Simulate" to check the curves with random canaries.
Change the number of looks, then the rule. Each line is the z-statistic of one canary that is exactly as good as stable. Magenta paths cross the aqua threshold at some look: a false rollback. With 10 looks at 1.645 about 17% of identical models are rolled back. "One look at the end" gives 5% for any number of batches. "Strict constant" uses a higher line at every look and also gives 5%. The small chart shows the rate against the number of looks.
Lesson 7 · Drift statistics
Choose a kind of shift and move the strength. Blue is the reference, orange the current data, aqua the model's fixed rule. Top: the input density . Middle: the true rule and the model's prediction. Right: how often the model predicts 1, how often , and the accuracy. Notice: under covariate shift the accuracy drops only when the inputs move to , where the reference had almost no data and the model is wrong. Under concept drift the input density and do not move at all, but the accuracy falls.
Move the mean shift to 0, then make the window small and the bins many. Blue bars are the reference proportions (all ), orange bars the current , and the lower chart shows each bin's term. The gauge compares PSI with the noise floor (violet) and the levels 0.1 and 0.25 (aqua). Notice: with no shift, and , PSI is far above 0.1 from noise alone. Press New current window a few times to see how much it varies.
Move the shift and the sample size. The blue and orange steps are the empirical CDFs of the reference and current samples; the magenta bar is the largest gap . Notice: at a fixed shift, stays near its population value as grows, but grows like and the p-value goes to 0. Press New samples to see how much varies at small .
Slide the sample size from to , then press Simulate. The shift is fixed (here standard deviations). Top: PSI (orange) and (violet) approach their population values and then stay flat. Bottom: the chi-square p-value (orange) and the KS p-value (violet) fall far below (aqua). Notice: at both tests say "significant" with p-values below , while PSI is , far below any alert level.
Lesson 8 · Monitoring and skew
Switch between naive rule and hysteresis, then try the other streams. Top: the stream and its true mean (orange). Bottom: the rolling PSI (orange), the warn, alert and clear levels (aqua) and the pages (magenta). Notice: in "near the threshold" the score crosses the alert level many times during one incident, so the naive rule pages 5 times and hysteresis once. In "no drift", make the window small: the noise floor of lesson 07 rises and false alarms appear. In "seasonal", every season looks like drift because the reference holds one season only.
Move the label delay and the window, then switch the cause to concept drift. Blue is the true accuracy, orange what the accuracy monitor can measure, violet the PSI of each day's inputs, aqua the alert levels and magenta the first alert of each monitor. Notice: the accuracy alert comes about days after the drop or later; the input monitor reacts on day 100. With concept drift the inputs do not change, so only the late accuracy monitor sees the problem. Make the window shorter than the delay: the accuracy monitor has no data at all.
Move the TTL, switch strict and inclusive, then switch to the plain join. Blue dots are the feature rows of station S1, orange dots the label rows, aqua bands each label's look-back window , and a line shows which feature row each label gets. Notice: the label at 12:00 gets 0.25 only with the inclusive rule. With the plain join every label gets 0.40, and the three labels before 18:00 see a value from their future (magenta).
Choose a cause of skew. Each dot is one logged request: the value the training pipeline computes (x) against the value the server used (y). Blue dots match, magenta dots do not; the dashed line is online = offline. Notice: client-side scaling breaks every humidity value; a stale value breaks a third of the temperatures; float32 breaks most values under the exact test and none under a small tolerance, so state the tolerance with the match rate.
Lesson 9 · LLMOps and evals
Move and and watch the two gates. Each cell is one golden case: blue = only the baseline passes (), orange = only the candidate passes (), light = both pass, grey = both fail. The number line shows the drop in pass rate (orange dot) and its one-sided 95% upper bound; the aqua line is the margin. With , the significance gate passes (), but the upper bound (6.85 points) is above the 5-point margin. Now set : the same rates give a tighter bound.
Try the presets, then move the sliders. The table shows the expected counts on 200 expert-labelled traces (errors in magenta). The bars compare the true pass rate , the pass rate the judge reports, and the corrected rate from Result 2. "Always pass" has 75% agreement but , and the correction is undefined. Press "Draw a labelled set" several times: TPR and TNR estimated from 200 labels are noisy, and so is the corrected rate.
Lower the attack rate and watch the right panel. Left: 1,000 requests; orange squares are attacks (solid = blocked, outlined = missed), magenta squares are valid users who are blocked. Right: 100 blocked requests. At 0.5% attacks, a detector that catches 95% of attacks and flags 2% of valid requests has a precision of 19%: about four of five blocks hit a valid user. With "common attacks" the same detector is precise.