MLOps: Quick recap

01 Reproducible runs and data contracts

A training run is a function: model = train(code, environment, data, config, seed, hardware). Every input you do not pin or record is a way for two runs to differ.
Bitwise reproducibility (same bits) needs the same hardware, software and determinism settings; statistical reproducibility (same metrics within the seed-to-seed spread) is what comparisons need.
Seeds are not enough: floating-point addition is not associative, and GPU kernels (autotuning, atomic adds, all-reduce) change the order of additions.
Version four things in one record: code (commit SHA), environment (lockfile or image digest, never a tag), data (content hash plus split rule and salt), model (artifact hash linked to the run).
A hash split puts key $k$ into the first split $i$ with $u(k) < c_i$ ; $u(k)$ depends only on the salt and the key, so old keys never move and groups never leak between splits.
A new salt gives an independent split that agrees with the old one on a fraction $\sum_i f_i^2$ of the keys (0.66 for 0.8/0.1/0.1).
A data contract states presence, dtype, nullability, range, domain and uniqueness per column; hard violations block or quarantine, statistical signals alert, and bad rows are never dropped silently.
A schema check sees only the current batch, so it misses every change of distribution that stays inside the allowed values (for example a unit change inside the range).

02 Testing ML systems

Code is tested with exact asserts; learned behaviour is tested with properties (with a tolerance $\tau$ ) and rates over many cases.
Test types of an ML system: code unit, data, model before training (overfit one batch), behavioural, slice and regression, infrastructure. Cheap exact tests run on every commit; statistical tests run before promotion.
CheckList: MFT = known label for a simple case; INV = a label-preserving change must not move the score by more than $\tau$ ; DIR = a change with known direction $d$ fails if $d\,(f(x') - f(x)) < -\tau$ . INV and DIR need no labels.
A behavioural test samples the input space: it can find a counterexample but cannot prove a property. Only a simple model (such as logistic regression) can show a property globally, from its weights.
Rule of three: $n$ independent passing cases give a 95% upper bound $p_u = 1 - 0.05^{1/n} \approx 3/n$ on the failure rate, for the distribution that generated the cases.
A slice regression test flags slices with $n \ge n_{\min}$ whose metric drops by more than $\delta$ . Compare the two models with McNemar's test on the discordant rows $b, c$ ; a slice accuracy has standard error $\sqrt{a(1-a)/n}$ .
The ML Test Score has 28 tests in 4 sections (0.5 manual, 1 automated); the final score is the minimum of the section sums, so the weakest section is the bottleneck.

03 Containers and CI

A container is an isolated process on the host kernel (namespaces for its view, cgroups for its resources), not a virtual machine; its file system is an image of read-only layers.
Cache rule: a build step is reused only if its instruction, its inputs and every earlier step are unchanged; after the first miss, every later step runs again.
Order a Dockerfile by change frequency: copy the lockfile, install, then copy the code; a code change then reruns only the last copy.
Use a .dockerignore, clean up in the same RUN, and a multi-stage build so that compilers and caches stay out of the final image.
The digest (SHA-256 of the manifest) names the exact image; a tag is a movable name. Build once and deploy the same digest.
The kernel-mode and user-mode GPU drivers come from the host; the CUDA runtime, libraries and framework come from the image. The host driver must be new enough for the image's CUDA.
Minor-version compatibility: any 12.x on a driver ≥ 525, any 13.x on ≥ 580, but no PTX JIT; a newer major version needs the forward-compatibility package (data-center GPUs only).
CI: on every pull request run fast checks (lint, unit, data, smoke training, behavioural tests, image build) in parallel jobs; build and push the image after a merge; run full training on a schedule on GPUs.

04 Kubernetes and the cloud

A scheduled task must be idempotent: parameterise it by the data interval, replace (never append) its output, write the parts under a unique attempt prefix and a manifest last; readers trust only the manifest.
Kubernetes keeps the desired state: a Deployment keeps $N$ Pods, a Service sends traffic only to ready Pods, Jobs and CronJobs run batch work.
Requests decide scheduling; limits are enforced (CPU throttled, memory OOM-killed). HPA utilisation is relative to the request.
Liveness failure restarts the container, readiness failure removes the Pod from the Service, and a startup probe holds both back during a slow model load.
HPA rule: $N' = \lceil N m / t \rceil$ , skipped within the tolerance (0.1 by default); scale-down uses the highest recommendation of a window (300 s by default).
Object storage has immutable objects, atomic single-key writes and no atomic rename in general-purpose buckets, so commit with a manifest written last.
Reserved capacity pays off only when the utilisation $u > p_{\text{res}}/p_{\text{od}}$ ; a job on $k$ nodes is interrupted with mean gap $M = M_{\text{node}}/k$ .
Young's checkpoint interval: $\tau^* = \sqrt{2CM}$ with waste about $\sqrt{2C/M}$ (plus $R/M$ ); the minimum is flat.

05 Serving and queueing

A model API validates input with a schema (422), limits batch size (413), makes one model call per batch, loads the model once at startup, and separates liveness (/healthz, never the model) from readiness (/readyz, 503 until loaded).
Latency is a distribution: SLOs use high percentiles such as the p99; percentiles of different replicas cannot be averaged.
Fan-out to $n$ independent backends, each slow with probability $q$ , makes a request slow with probability $1 - (1-q)^n$ (18% for $n = 20$ , $q = 1\%$ ).
Little's law $L = \lambda W$ holds for any stable system.
M/M/1: $W = 1/(\mu - \lambda)$ , $\mu W = 1/(1-\rho)$ , and the latency is exponential, so $p_{99} = \ln(100)\,W \approx 4.6\,W$ .
Every latency percentile carries the factor $1/(1-\rho)$ : from $\rho = 0.9$ to $0.95$ all latencies double. Constant service times halve the wait but keep this wall.
Batching with $t(b) = a + cb$ raises the throughput towards $1/c$ ; under load the batch size settles at $b^* = \lambda a/(1 - \lambda c)$ , and the capacity of one replica is $B/t(B)$ .
A max queue delay helps only under load; at low traffic it adds exactly the delay. Choose $B$ , the delay and the number of replicas from the p99 SLO, and confirm with a simulation or a load test.

06 Registry and canary releases

A model registry stores immutable versions with lineage (code commit, image digest, data version, config hash) and metrics; an alias such as champion is a movable pointer to the serving version.
Promotion moves the pointer after a gate passes: absolute bounds, the primary metric beats production by $\delta$ , and guard metrics (slices, latency, memory) stay within a tolerance. Rollback moves it back and needs the old artifact and its inputs.
Shadow tests load and errors with no user risk but measures no effect; a canary limits the blast radius and compares with a concurrent control; an A/B test is for a decision; blue-green switches everyone at once; a rolling update only checks that instances start.
Hash routing $\text{bucket} = H(\text{salt}:id) \bmod 10\,000$ , canary iff $\text{bucket} < 100p$ , is sticky, monotone, uniform and independent across salts. Random per-request routing and Python's hash() break it.
Canary sample size per group: $n = \big(z_{1-\alpha}\sqrt{2\bar p\bar q} + z_{1-\beta}\sqrt{p_1q_1 + p_2q_2}\big)^2/\Delta^2$ , so $n \propto 1/\Delta^2$ ; the run time is $n$ divided by the canary's traffic.
Peeking at a fixed $\alpha$ inflates false alarms: about 17% for 10 looks and 31% for 100 looks at $\alpha = 0.05$ . Use a fixed horizon, a group-sequential design or always-valid p-values.
"No significant difference" after a small sample is low power, not evidence that the new version is safe.

07 Drift statistics

Shift kinds come from $p(x, y) = p(x)\,p(y \mid x) = p(y)\,p(x \mid y)$ : covariate shift changes $p(x)$ , label shift changes $p(y)$ , concept drift changes $p(y \mid x)$ , prediction drift is a change of $p(\hat y)$ .
For a fixed model, $p(\hat y)$ depends only on $p(x)$ , so pure concept drift causes no prediction drift; only labels reveal it.
Under label shift a calibrated model is corrected by $q(y \mid x) \propto \frac{q(y)}{p(y)}\,p(y \mid x)$ .
$\mathrm{PSI} = \sum_i (a_i - e_i)\ln(a_i/e_i) = \mathrm{KL}(a\|e) + \mathrm{KL}(e\|a) \ge 0$ ; the levels 0.1 and 0.25 are a rule of thumb, not a test.
PSI depends on the binning (merging bins cannot increase it; empty bins depend on $\varepsilon$ ) and has a noise floor of about $(k-1)(1/n + 1/m)$ .
The two-sample KS statistic is $D = \sup_t |F_x(t) - F_y(t)|$ ; its p-value uses $\lambda = \sqrt{nm/(n+m)}\,D$ .
For categories use the chi-square test on the $2 \times k$ table, with $k - 1$ degrees of freedom.
PSI and $D$ are effect sizes; KS and chi-square p-values go to 0 as the sample grows, so alert on effect sizes.

08 Monitoring and skew

Monitor in layers: service health, data quality, input drift, prediction drift (no labels), and performance per slice (delayed labels); give every alert an owner and a runbook.
A drift monitor scores the last $w$ values every $s$ values against a reference (the same season for seasonal inputs), with thresholds set to a high quantile of scores on stable history.
Hysteresis: the state moves up at once and comes down only below a lower clear level, so one incident gives one page.
With $k$ independent features and false-alarm rate $\alpha$ each, the chance of at least one false alarm per check is $1 - (1-\alpha)^k$ (87% for $k = 200$ , $\alpha = 0.01$ ).
With label delay $d$ and window $w > d$ , the accuracy monitor uses predictions from $(t - w, t - d]$ : a drop at $t_0$ starts to show at $t_0 + d$ and shows fully at $t_0 + w$ .
A feature store has an offline store (full history, for training), an online store (latest value per entity, for serving), materialisation between them, and a TTL per feature.
A point-in-time join gives a label at time $t$ the latest feature row with $s < t$ (or $s \le t$ ) and $s \ge t - \text{TTL}$ ; a plain join on the current table leaks the future.
Training-serving skew means the same example gets different inputs offline and online; find it by logging serving features and comparing them with recomputed ones; drift means the world changed.

09 LLMOps and evals

An LLM app is $\text{App}(\text{input}; c)$ . Pin the whole configuration $c$ (prompt, model snapshot, decoding, tools, retrieval, judge) as one version and log its hash with every trace.
A new embedding model needs a full re-index: vectors from two models live in different spaces. Evaluate retrieval (recall@ $k$ ) on its own.
An eval gate runs the baseline and the candidate on the same golden set: assertion checks, an LLM judge with a binary rubric, several epochs when sampling is random, and a gate rule fixed in advance.
Paired comparison: only the discordant cases $b, c$ matter; the drop is $(b - c)/N$ , McNemar's test uses $b \sim \text{Bin}(b + c, 1/2)$ , and the paired standard error is smaller than the unpaired one.
"Not significant" is not "safe": a non-inferiority gate blocks when the one-sided upper bound of the drop exceeds a margin fixed in advance.
LLM judges have position, verbosity and self-preference biases and grade maths poorly; judge both orders and use a reference for maths.
Calibrate a judge with TPR, TNR and Cohen's $\kappa = (p_o - p_e)/(1 - p_e)$ ; correct its pass rate with $\theta = (p_{\text{obs}} + \mathrm{TNR} - 1)/(\mathrm{TPR} + \mathrm{TNR} - 1)$ , valid when $\mathrm{TPR} + \mathrm{TNR} > 1$ .
Guardrails reduce risk but cost latency, tokens and false positives; when attacks are rare, most blocks hit valid users. Prompt injection cannot be fully prevented, so limit tool permissions too.