Model registry, CD and safe deployment: shadow, canary, A/B

Before this lesson: lesson 01 (lineage and hash-based splits), lesson 02 (slice tests), lesson 03 (CI and image digests), lesson 05 (model servers). From probability and statistics: the CLT, hypothesis testing and power and A/B testing. Time: about 80 minutes, plus the notebook.

---

Why this matters #

Most failures of a live system follow a change. The Google SRE book says: "SRE has found that roughly 70% of outages are due to changes in a live system." In ML a change can be new code, new data, a new model or a new config. A model can also be correct as code and still be worse on today's inputs.

So every change should be small, tested and reversible:

• the registry answers "which model serves now, and what was it built from?";
• the gate stops a model that is worse on a slice that matters;
• a staged rollout limits the blast radius: the share of users that a bad change can hurt;
• the sample-size formula tells you how long each stage must last.

A weather service that replaces its forecast model needs all four, and so does a fraud or ranking team. "How do you deploy a new model safely?" is also a standard interview question.

1. Predict first #

2. The picture #

The path of a change #

Every change moves along one path:

commit → CI tests → artifacts (an image, a model) → registry → gate → small share of traffic → watch → ramp up or roll back.

Lesson 03 covered the first half: CI tests the code and builds an image tagged with its commit. ML adds a second loop. The code loop starts with a commit. The model loop starts with new data or a schedule: a training pipeline produces a new model. Both loops end in the same place: the registry.

The registry is fixed versions plus a pointer #

A registry never changes a stored version. It only adds new versions. One name, the alias (here champion), points to the version that serves. The model server reads the alias. So:

• promotion moves the pointer forward;
• rollback moves the pointer back. Nothing is rebuilt, so a rollback takes seconds.

Interactive figure: loading…

A ladder of risk #

A new version climbs a ladder. Each step shows it to more real traffic:

1. offline evaluation: no users;
2. shadow: live inputs, outputs logged but not used;
3. canary: a small share of users really served, compared with the rest;
4. full rollout (and, if the team needs a business decision, an A/B test).

Each step catches failures that the step before cannot see. The cost of a bug depends on two things: how much traffic the new version serves, and how long the bug runs before someone detects it.

Interactive figure: loading…

Who is in the canary? #

We must decide, for each request, which version serves it. The rule is to hash the user id and compare the hash with the share. Then a user always sees the same version, and raising the share only adds users. The other way, a random choice per request, mixes both versions for the same user.

Interactive figure: loading…

3. Definitions and main results #

3.1 Registry, lineage and aliases #

A model registry stores, for each model name, versions $1, 2, 3, \dots$ Each version has:

• the artifact (the model file) and its address (URI);
• its lineage: the code commit, the image digest, the data version and the config hash (lesson 01), so the version can be rebuilt;
• its offline metrics, overall and per slice;
• tags (key–value notes) and documentation.

An alias is a mutable name that points to one version, for example champion. In MLflow the URI models:/nowcast@champion loads the version that the alias points to. Some registries use fixed stages (Staging, Production, Archived) instead; the idea is the same: one pointer, plus a history.

Promotion moves the pointer to a new version. Rollback moves it back to the previous one. A rollback works only if two things are true: the old artifact still exists, and the features it needs are still produced (the input schema did not change under it).

A model card (Mitchell et al., 2019) is a short document stored with the version: intended use, training and evaluation data, metrics per group or slice, and known limits. It answers "what is this model for, and where does it fail?" for people who did not train it.

3.2 The promotion gate and continuous training #

A promotion gate is a set of rules in code. The candidate must pass all of them:

1. absolute bounds: limits that hold even without a production model (for example CSI $\ge 0.35$, p99 latency $\le 100$ ms);
2. primary metric: the candidate beats production by at least a margin $\delta$;
3. guard metrics: key slices, latency and memory may get worse by at most a tolerance.

The gate checks every rule and lists every failure with its reason. Every decision, pass or fail, goes into an audit log. A gate with one aggregate metric is the classic mistake: the aggregate can improve while a slice that matters gets worse (see Part A of section 4).

Continuous training (CT) means that the training pipeline itself is deployed and runs on triggers. Google Cloud's MLOps guide lists five triggers: on demand, on a schedule, when new training data arrives, when model performance degrades, and when the data distribution changes significantly (lesson 07). Each run ends with registration and the gate.

The same guide describes three maturity levels:

Level	What is automated	What is deployed
0, manual process	nothing: notebooks and scripts, hand-off to engineers, rare releases, no CI/CD, little monitoring	a model
1, ML pipeline automation	the training pipeline: continuous training with automatic data and model validation	the training pipeline
2, CI/CD pipeline automation	also the build, test and deployment of the pipeline code itself	new pipeline versions, quickly and safely

3.3 Deployment strategies #

Strategy	How	What it tests	User risk	Cost	Rollback
shadow	a copy of live traffic goes to the new version; outputs are logged, not used	latency, errors, load, agreement with production	none	extra compute (up to 2×)	stop mirroring
canary	a small share (1 → 5 → 25 → 100%) is served by the new version and compared with a concurrent control	errors, latency and guard metrics on real use	the canary share	small	route back to stable
A/B test	a randomised split for a planned duration and sample size	the causal effect on a product or science metric	up to half the users get the worse variant	time	end the test
blue-green	two full environments; all traffic switches at once	the whole new stack, smoke-tested while idle	everyone, at the switch	2× capacity	switch back, instantly
rolling	instances replaced a few at a time	that new instances start and pass their probes	mixed versions during the rollout	a few extra instances	another rolling update

Three points to remember:

• A shadow measures no effect. Nobody acts on its outputs, so it cannot show how users or other systems react. Its side effects (database writes, alerts, messages) must be switched off.
• A canary is built for safety, an A/B test for a decision. A canary must detect breakage fast on a small sample. An A/B test must have the power to see small effects on the target metric.
• A plain rolling update is not a canary. In Kubernetes the default rolling update replaces Pods with at most 25% extra and 25% unavailable Pods. It cannot hold a fixed traffic share, query metrics or roll back by itself. Tools such as Argo Rollouts add weighted canary steps, metric analysis and automatic rollback.

The blast radius of a release is the largest share of users that a bad version can reach before the rollback. A canary keeps it small. Blue-green has the largest blast radius but the fastest rollback.

3.4 Deterministic routing #

Here $H$ is SHA-256 read as an integer (its first 8 bytes, as in lesson 01), $p$ is the canary share in percent, and the salt is a name for the experiment, such as nowcast-v5.

Claim (four properties). Model $H$ as a random function: each new input gets an independent, uniformly random output. Then:

1. Sticky: the same id always gets the same version.
2. Monotone: if $p_1 < p_2$, every user in the canary at $p_1$ is also in it at $p_2$.
3. Uniform: each user is in the canary with probability $p/100$, so the share in a group of $N$ users is $p/100$ up to binomial noise with standard deviation $\sqrt{(p/100)(1-p/100)/N}$.
4. Independent across salts: with two salts, a user is in both canaries with probability $(p_1/100)(p_2/100)$.

Proof.

1. The bucket is a function of the salt and the id only. No clock and no random draw enter it.
2. $\text{bucket} < 100\,p_1$ and $100\,p_1 < 100\,p_2$ give $\text{bucket} < 100\,p_2$.
3. A uniform 64-bit integer modulo $10\,000$ is uniform on $\{0, \dots, 9999\}$ up to a bias below $10^4/2^{64} \approx 5 \cdot 10^{-16}$. So $P(\text{bucket} < 100p) = 100p/10\,000 = p/100$. The users are independent, so the count is binomial.
4. The strings salt1:id and salt2:id differ, so the two outputs are independent. Multiply the two probabilities. $\blacksquare$

"Random function" is a model of SHA-256, not a theorem about it. It is the standard assumption, and the notebook checks properties 3 and 4 on 20,000 ids.

Monotone ramps matter for metrics: if a user saw both versions, their clicks, complaints or session length mix the two models, and the per-user comparison is no longer clean.

Two ways to break this rule:

• A random choice per request breaks properties 1 and 2.
• Python's built-in hash() of a string is salted per process. Two replicas of the server would route the same user differently. Use hashlib.

3.5 How many requests does a canary need? #

The picture first. Say an "error" is a failed request. Stable has error rate $p_1$, the canary has $p_2$. Each group gets $n$ requests. The observed difference $D = \hat p_2 - \hat p_1$ is random. By the CLT it is close to a normal distribution. If nothing changed (the null hypothesis $H_0$), $D$ is centred at 0. If the canary is worse by $\Delta = p_2 - p_1$ (the alternative $H_1$), $D$ is centred at $\Delta$. We roll back when $D$ is above a cut-off $c$.

• The area of the $H_0$ curve above $c$ is the false-alarm rate $\alpha$.
• The area of the $H_1$ curve above $c$ is the power $1 - \beta$: the chance to detect a real increase.

Both curves get narrower like $1/\sqrt n$, but their centres stay $\Delta$ apart. So a larger $n$ separates them.

Interactive figure: loading…

Theorem (canary sample size). Each request is an error with probability $p_1$ on stable and $p_2$ on the canary, independently. Let $q_i = 1 - p_i$, $\Delta = p_2 - p_1 > 0$, and $n$ requests per group. The test rejects $H_0: p_2 = p_1$ in favour of $H_1: p_2 > p_1$ when the pooled z-statistic exceeds $z_{1-\alpha}$, the $(1-\alpha)$-quantile of $N(0, 1)$. For significance $\alpha$ and power $1 - \beta$ it needs

Proof.

1. The error rate $\hat p_i$ of group $i$ is a mean of $n$ Bernoulli variables, so its variance is $p_i q_i / n$. By the CLT, $D = \hat p_2 - \hat p_1$ is approximately normal.
2. The test divides $D$ by the pooled standard error $\sqrt{2\hat p \hat q / n}$, where $\hat p$ is the error rate of both groups together. For large $n$, $\hat p \approx \bar p$ (under $H_1$ too). So the test rejects when $D > c$ with $c = z_{1-\alpha}\sqrt{2\bar p \bar q / n}$. Under $H_0$ both groups have the rate $\bar p$, so $\operatorname{Var}_0(D) = 2\bar p \bar q / n$ and the false-alarm rate is $\alpha$.
3. Under $H_1$: $E_1(D) = \Delta$ and $\sigma_1^2 = \operatorname{Var}_1(D) = (p_1 q_1 + p_2 q_2)/n$.
4. Power $1 - \beta$ means $P_1(D > c) = 1 - \beta$. Standardise: $P_1\big((D - \Delta)/\sigma_1 > (c - \Delta)/\sigma_1\big) = 1 - \beta$, so $(c - \Delta)/\sigma_1 = -z_{1-\beta}$, that is $\Delta = c + z_{1-\beta}\,\sigma_1$.
5. Insert $c$ and $\sigma_1$ and multiply by $\sqrt n$: $\Delta\sqrt n = z_{1-\alpha}\sqrt{2\bar p \bar q} + z_{1-\beta}\sqrt{p_1 q_1 + p_2 q_2}$. Square both sides and divide by $\Delta^2$. $\blacksquare$

What the formula says.

• $n$ grows like $1/\Delta^2$: half the effect needs about four times the data (exactly four when the variances barely change).
• Rare errors need many requests: the variances $p_i q_i$ are small, but $\Delta$ is small too.
• The run time of a canary stage is $n$ divided by the canary's traffic.
• For the test itself, $z^2$ equals the chi-square statistic of the $2 \times 2$ table (without continuity correction). The notebook uses this to check the z-test with SciPy.

Unequal groups. In practice the control (stable) gets much more traffic than the canary. Its variance term $p_1 q_1 / n_{\text{ctrl}}$ then almost vanishes, and the canary needs fewer requests than the equal-groups $n$. Solving the same power equation with $n_{\text{ctrl}} = 19\,n_c$ (a 5% canary) gives $n_c = 1711$ in the example of section 4; a simulation with 100,000 runs gives power 0.80.

3.6 The peeking problem #

Every look at a running test is a test with false-alarm rate $\alpha$. If you look many times and stop at the first "significant" result, the false alarms add up.

Why. Split the requests into $K$ equal batches. Under $H_0$ the z-statistic after $k$ batches is about $Z_k = S_k / \sqrt k$, where $S_k$ is a sum of $k$ independent $N(0, 1)$ steps: a Gaussian random walk. One look at the end crosses $z_{0.95} = 1.645$ with probability 5%. But a random walk that is checked 10 times has 10 chances to cross the line.

Interactive figure: loading…

Simulation with one-sided $\alpha = 0.05$ (200,000 random-walk paths per column):

Looks	1	2	5	10	20	100
false-alarm rate	0.050	0.081	0.130	0.171	0.213	0.310

With unlimited looks the rate goes to 1 (a consequence of the law of the iterated logarithm). So "stop at the first $p < 0.05$" is not a 5% test.

Valid ways to look.

• Fixed horizon: compute $n$ with the theorem, then look once.
• Group-sequential design: plan a few looks with stricter thresholds. The Pocock design uses the same threshold at every look; for 10 looks at overall one-sided $\alpha = 0.05$ we found 2.27 by simulation. The O'Brien–Fleming design is very strict at early looks and close to the fixed-horizon threshold at the last look.
• Always-valid p-values: methods built for continuous monitoring (see the deeper block).

In practice the two decisions are not symmetric. A false rollback is cheap: the old model keeps serving. So a rollback rule may look often, with a strict threshold or a sequential method. A promotion needs the planned sample size: "no significant difference" after a few hundred requests means low power, not evidence of safety.

The Google SRE Workbook adds two rules. Compare the canary with a concurrent control, not with "before the release": traffic, weather and users change over time. And select a few metrics, "perhaps no more than a dozen", that show problems users see.

Always-valid inference: SPRT and mixture SPRT

Wald's sequential probability ratio test (SPRT) compares two simple hypotheses, for example $p_2 = p_1$ against $p_2 = p_1 + \Delta$. After each request it updates the likelihood ratio $\Lambda_n = \prod_{i \le n} f_1(x_i)/f_0(x_i)$ and stops when $\Lambda_n \ge 1/\alpha$. Under $H_0$, $\Lambda_n$ is a nonnegative martingale with mean 1, so by Ville's inequality $P_0(\sup_n \Lambda_n \ge 1/\alpha) \le \alpha$: the false-alarm rate stays below $\alpha$ however often you look.

The SPRT needs one fixed $\Delta$. The mixture SPRT (mSPRT) averages the likelihood ratio over a prior on the effect size, so it has power against many effect sizes. Johari, Pekelis and Walsh turn it into always-valid p-values: $p_n = \min(1, 1/\max_{m \le n} \Lambda_m)$. You may stop at any time with $p_n < \alpha$ and keep the false-alarm rate below $\alpha$. The price: at the same number of requests, it has lower power than the fixed-horizon z-test, which is the most powerful one-sided test for one planned look.

Answers to section 1. (1) Yes. The user's bucket is fixed, and bucket $< 500$ implies bucket $< 2500$. The ramp is sticky and monotone, so each user sees one version and the per-user metrics are clean. (2) About 17%, more than three times the nominal 5% (see the widget and the table). (3) About 4 times: $n$ grows like $1/\Delta^2$. For example, at a 1% baseline, 1% → 2% needs 1826 requests per group and 1% → 1.5% needs 6105, a factor 3.3. The factor is a bit below 4 because $p_2$ is smaller, so the variance terms shrink too.

4. Worked example #

A precipitation-nowcasting model for weather stations predicts the chance of rain in the next hour. The critical success index is $\text{CSI} = \text{hits} / (\text{hits} + \text{misses} + \text{false alarms})$ for rain above a threshold; higher is better.

Part A: the gate. The primary metric is CSI at 1 mm/h with margin $\delta = 0.005$. The absolute bound is CSI at 1 mm/h $\ge 0.35$. The guards are p99 latency (tolerance 5 ms) and CSI at 10 mm/h, the heavy-rain slice (tolerance 0.005).

Metric	production v3	candidate v4	Check
CSI 1 mm/h	0.412	0.420	$0.420 \ge 0.35$ ✓; gain $0.420 - 0.412 = 0.008 \ge 0.005$ ✓
p99 latency	38 ms	41 ms	worse by $41 - 38 = 3 \le 5$ ✓
CSI 10 mm/h	0.215	0.198	worse by $0.215 - 0.198 = 0.017 > 0.005$ ✗

The gate rejects v4 with the reason "CSI 10 mm/h worse by 0.017". The aggregate improved, but the slice that matters most for warnings got worse. The rejection goes into the audit log. The pointer stays on v3.

Part B: sizing the canary. A later version v5 passes the gate. An "error" is a failed request: a timeout, a 5xx status, or an output that breaks the output schema. Stable has $p_1 = 0.5\%$. We want to detect $p_2 = 1.0\%$ with $\alpha = 0.05$ and power 0.8, so $z_{0.95} = 1.6449$ and $z_{0.8} = 0.8416$.

1. $\bar p = (0.005 + 0.010)/2 = 0.0075$ and $\bar q = 0.9925$.
2. $2\bar p\bar q = 2 \cdot 0.0075 \cdot 0.9925 = 0.0148875$, so $\sqrt{2\bar p\bar q} = 0.12201$.
3. $p_1 q_1 + p_2 q_2 = 0.004975 + 0.0099 = 0.014875$, so $\sqrt{p_1 q_1 + p_2 q_2} = 0.12196$.
4. Numerator: $1.6449 \cdot 0.12201 + 0.8416 \cdot 0.12196 = 0.20070 + 0.10264 = 0.30334$.
5. $n = 0.30334^2 / 0.005^2 = 0.092016 / 0.000025 = 3680.7$. Round up: 3681 requests per group.
6. Run time: the service gets 30,000 requests per hour, so a 5% canary gets 1,500 per hour, and $3681 / 1500 = 2.45$ hours.
7. With the real split (control 19 times larger), the canary needs only $n_c = 1711$ requests: $1711 / 1500 = 1.14$ hours.
8. The price of the check: if v5 really has 1% errors, the canary serves about $(0.010 - 0.005) \cdot 1711 \approx 9$ extra failed requests.

Part C: the verdict. After 4,000 canary requests there are 38 errors (0.95%). Stable has 380 errors in 76,000 requests (0.50%). The rollback rule uses a strict one-sided $\alpha = 0.01$.

1. Pooled rate: $\hat p = (380 + 38)/(76\,000 + 4\,000) = 418/80\,000 = 0.005225$.
2. Standard error: $\text{se} = \sqrt{0.005225 \cdot 0.994775 \cdot (1/76\,000 + 1/4\,000)} = 0.0011695$.
3. $z = (0.0095 - 0.0050)/0.0011695 = 3.85$.
4. One-sided p-value $= P(Z > 3.85) = 6.0 \cdot 10^{-5} < 0.01$ (equivalently, $3.85 > z_{0.99} = 2.326$).

Roll back. All traffic goes to stable, the champion alias stays on v3, and the registry logs why v5 failed.

5. In code #

SciPy's chi-square test checks Part C (through $z^2 = \chi^2$). A simulation checks the power of $n = 3681$ and the peeking rates, and hashlib checks the routing example.

import hashlib
import numpy as np
from scipy.stats import chi2_contingency, norm

def signed_z(x_ctrl, n_ctrl, x_can, n_can):
    """Pooled two-proportion z from SciPy's 2x2 chi-square (z^2 = chi^2, no correction)."""
    chi2 = chi2_contingency([[x_ctrl, n_ctrl - x_ctrl], [x_can, n_can - x_can]],
                            correction=False)[0]
    return np.sign(x_can / n_can - x_ctrl / n_ctrl) * np.sqrt(chi2)

z = signed_z(380, 76_000, 38, 4_000)
print(f"Part C: z = {z:.2f}, one-sided p = {norm.sf(z):.1e}")   # z = 3.85, p = 6.0e-05

rng, n, z_crit = np.random.default_rng(0), 3681, norm.ppf(0.95)
for p_can in (0.005, 0.010):                       # H0 true, then H1 true
    zs = [signed_z(rng.binomial(n, 0.005), n, rng.binomial(n, p_can), n) for _ in range(4000)]
    print(f"canary error rate {p_can:.3f}: rejects in {np.mean(np.array(zs) > z_crit):.3f}")
# about 0.05 (false alarms) and about 0.80 (power)

# Peeking: under H0 the z after k equal batches is about S_k / sqrt(k).
for looks in (1, 10, 100):
    S = np.cumsum(rng.standard_normal((20_000, looks)), axis=1)
    Z = S / np.sqrt(np.arange(1, looks + 1))
    print(f"{looks:3d} looks: false rollbacks {np.mean(Z.max(axis=1) > z_crit):.2f}")
# 1 look: 0.05, 10 looks: 0.17, 100 looks: 0.31

def bucket(user_id, salt):
    digest = hashlib.sha256(f"{salt}:{user_id}".encode()).digest()
    return int.from_bytes(digest[:8], "big") % 10_000

print(bucket("user-3", "nowcast-v5"), bucket("user-42", "nowcast-v5"))   # 264 1904
# user-3 is in the canary from 5% on (264 < 500); user-42 joins at 25% (1904 < 2500)

6. Where it shows up #

• ML products. Ranking, ads and fraud models usually go shadow → canary → A/B test. The canary protects users; the A/B test decides if the new model is worth it. Prompt and LLM changes are gated by eval suites in the same way (lesson 09).
• ML for science. Weather centres run a new model version in parallel and publish test data before they switch. ECMWF implemented its machine-learned model AIFS Single v1.1 on 31 July 2025 and reverted to v1 on 1 August, because of a technical issue with the stepRange GRIB2 metadata of six accumulated variables. After a fix and a parallel test production, it re-implemented v1.1 on 27 August 2025. The cause was the output format, not the forecast skill. Output-contract tests and a fast rollback path matter as much as the metrics.
• HPC to cloud. Hosted CI runners cannot run a multi-GPU training job. A scheduled or manual pipeline submits the training to a cluster (through a scheduler such as Slurm, or a Kubernetes Job), and the job registers the model with its lineage. The model is then served on Kubernetes behind a canary (lesson 04).

7. Common mistakes #

• Deploying a mutable tag such as latest. Deploy the commit SHA or the image digest, so you know exactly what runs and can go back to it.
• One aggregate metric in the gate. A slice regression stays hidden (Part A). Add guard metrics for the slices that matter.
• No way back. The old artifact was deleted, or the features it needs are no longer produced. Keep old artifacts and their inputs until the new version is safe.
• Random per-request routing, or Python's hash(). Users see both versions, and replicas disagree. Hash the user id with a stable hash such as SHA-256.
• Before/after comparison instead of a concurrent control. Traffic and weather change over time. Compare canary and stable over the same period.
• Peeking, then promoting early on "no significant difference". Repeated looks inflate false alarms, and a small sample has low power. Use the planned $n$ or a sequential design.
• A shadow with side effects. A shadow must not write to databases, send messages or raise alerts.
• A canary step that is too short. Each step must last at least $n$ divided by the canary's traffic; otherwise the ramp reaches 100% before the test can see anything.

8. Check yourself #

1. Name the three kinds of rules in a promotion gate, with one example each.
   Answer

   Absolute bounds (CSI at 1 mm/h $\ge 0.35$), the primary metric beats production by $\delta$ (CSI gain $\ge 0.005$), and guard metrics within a tolerance (p99 latency worse by at most 5 ms; heavy-rain CSI worse by at most 0.005).
2. Baseline error rate 1%. How many requests per group to detect 2% at one-sided $\alpha = 0.05$ and power 0.8? And for 1.5%?
   Answer

   $\bar p = 0.015$: $(1.6449\sqrt{0.02955} + 0.8416\sqrt{0.0295})^2/0.01^2 = 1825.9$, so 1826. For 1.5%: 6105. Halving $\Delta$ multiplies $n$ by 3.3, close to the factor 4 of the $1/\Delta^2$ rule.
3. A user is in the canary at 10% with salt nowcast-v5. Is the user in the canary at 30% with the same salt? At 10% with salt nowcast-v6?
   Answer

   Yes at 30%: bucket $< 1000$ implies bucket $< 3000$. With a new salt the bucket is unrelated to the old one, so the probability is 10%, whatever happened before.
4. Why can a rollback fail even though the registry still lists the old version?
   Answer

   The listing is only metadata. The rollback also needs the old artifact (the file may have been deleted by a retention rule) and its inputs: if a feature was renamed or removed, the old model cannot run on today's requests.
5. A team retrains every month in a notebook and copies a pickle file to the server. Which MLOps level is this, and what is the first step up?
   Answer

   Level 0. Level 1: an automated, versioned training pipeline with data and model validation, triggered by a schedule or new data, that registers each model with lineage behind a gate.
6. After 400 canary requests the test says "no significant difference" ($p = 0.4$). The planned $n$ was 3681. Can you promote?
   Answer

   No. With 400 requests the power to detect the planned increase is low (about 0.20 for 0.5% → 1%), so a real problem would usually give "no significant difference" too. A non-significant result is not evidence of safety. Continue to the planned $n$.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. What is a model alias in a registry?

   Show answer

2. What does a rollback need, besides the registry entry?

   Show answer

3. Name the three kinds of rules in a promotion gate.

   Show answer

4. What can a canary test that a shadow cannot?

   Show answer

5. Canary vs A/B test: what is each one for?

   Show answer

6. Write the hash-routing rule for a canary share of $p$ percent.

   Show answer

7. Why must a canary ramp be monotone?

   Show answer

8. Write the canary sample size per group (one-sided two-proportion z-test).

   Show answer

9. Baseline 0.5% errors, detect 1.0%, $\alpha = 0.05$, power 0.8: requests per group?

   Show answer

10. State the peeking problem in one sentence.

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "How do you deploy a new model safely?" — Register it with lineage; run the gate (bounds, primary metric, slice and latency guards); shadow it on live traffic; then a canary with hash routing and automatic rollback on guard metrics; an A/B test if the business needs a decision. The registry alias gives an instant rollback.
• "How long must the canary run?" — Compute $n$ from the baseline rate, the smallest increase that matters, $\alpha$ and the power; the time is $n$ divided by the canary's traffic. A large control group reduces the canary's $n$.
• "Why hash the user id instead of random routing?" — Sticky and monotone: each user sees one version during the whole ramp, so per-user metrics are clean, and replicas agree.
• "Why not stop at the first $p < 0.05$?" — Peeking inflates false alarms (about 17% for 10 looks). Use a fixed horizon, a group-sequential design or always-valid p-values.
• "What is MLOps level 1?" — The training pipeline is automated and runs on triggers (continuous training), with automatic data and model validation, ending in the registry.

10. Practice #

The notebook for this lesson runs in Colab. It has four exercises, each with a check cell and a folded solution:

1. Write route(user_id, percent, salt) with SHA-256 buckets, and test stickiness, monotonicity, the canary share and the independence of two salts.
2. Write the pooled one-sided two_proportion_ztest (checked against SciPy's chi-square test) and canary_sample_size (checked by Monte Carlo); then size a canary with a 19 times larger control.
3. Write evaluate_gate (collect every failing rule with a reason) and a tiny in-memory Registry with register, promote, rollback and an audit log.
4. Simulate the false-alarm rate against the number of looks, and find by simulation a constant threshold that keeps 10 looks at 5% overall.

11. Watch and read #

Statistical Power, Clearly Explained!!!StatQuest with Josh Starmer · 8 min

Power as the overlap of two distributions, the picture of section 3.5.

Power Analysis, Clearly Explained!!!StatQuest with Josh Starmer · 17 min

How effect size, variance and power fix the sample size.

p-hacking: What it is and how to avoid it!StatQuest with Josh Starmer · 14 min

Why repeated testing until "significant" inflates false positives, the root of the peeking problem.

Peeking at A/B Tests: Why it matters, and what to do about itJohari, Pekelis, Walsh · KDD 2017 · 3 min

The authors of always-valid p-values explain the peeking problem in industry A/B tests.

Top 5 Most-Used Deployment StrategiesByteByteGo · 10 min

A short visual overview of common deployment strategies; compare it with the table in section 3.3.

• Google SRE book, Introduction: "roughly 70% of outages are due to changes in a live system", and progressive rollouts as the answer.
• Google SRE Workbook, ch. 16 "Canarying Releases": choosing the canary population and duration, metrics, and why before/after evaluation is risky.
• Google Cloud, MLOps: Continuous delivery and automation pipelines in machine learning: levels 0, 1 and 2 and the pipeline triggers.
• MLflow Model Registry: versions, aliases and tags.
• Kubernetes, Deployments: the rolling-update strategy (maxSurge, maxUnavailable) and rollback.
• Argo Rollouts: "Why Argo Rollouts?" (the limits of the rolling update) and weighted canary steps with analysis.
• Mitchell et al., Model Cards for Model Reporting (FAT* 2019).
• Johari, Pekelis, Walsh, Always Valid Inference: Bringing Sequential Analysis to A/B Testing (the deeper block of section 3.6).
• ECMWF, Implementation of AIFS Single v1: the v1.1 rollback and re-implementation of 2025.