MLOps · 126

Glossary

The technical terms of this track, with short definitions (and the Persian term where available). Each term links to the lesson where it is introduced.

A/B test: A randomized comparison of two versions on live users, run long enough to make a business decision with planned power. Introduced in: Lesson 6
alert fatigue: The effect of alerts that nobody acts on: people learn to ignore all alerts. Introduced in: Lesson 8
alias: A movable name in a registry, such as champion, that points to the version that serves. Promotion moves it forward; rollback moves it back. Introduced in: Lesson 6
app version: The hash of the one configuration file of an LLM application: prompt, model snapshot, decoding, tools, retrieval and judge. It is logged with every trace. Introduced in: Lesson 9
Apptainer: A container runtime for HPC clusters (formerly Singularity). It converts an image to one file (SIF) and runs it without root; --nv mounts the host's NVIDIA driver. Introduced in: Lesson 3
arrival rate and service rate: The arrival rate $\lambda$ is the number of requests that arrive per second; the service rate $\mu$ is the number one replica can finish per second when busy. Introduced in: Lesson 5
assertion check: An eval check written in code, such as "the output is valid JSON" or "every citation is among the retrieved chunks". It is exact, cheap and runs on every change. Introduced in: Lesson 9
backfill: Creating runs of a pipeline for a past range on purpose, for example after a bug fix. Introduced in: Lesson 4
behavioural test: A test of what a model has learned: change the input in a controlled way and check how the output moves. It can find a failure but never prove that there is none. Introduced in: Lesson 2
bitwise reproducibility: The same bits in every weight when a run is repeated. It needs the same hardware, software and settings; it is useful for debugging and regression tests. Introduced in: Lesson 1
blast radius: The share of users that a bad change can hurt. A staged rollout limits it. Introduced in: Lesson 6
build context: The folder sent to the image builder. A .dockerignore file keeps data, checkpoints and .git out of it. Introduced in: Lesson 3
canary release: A release in which a small share of users really gets the new version, compared at the same time with the users on the stable version. Introduced in: Lesson 6
canary sample size: Requests per group needed to detect an error-rate increase $\Delta = p_2 - p_1$ : $n = \big(z_{1-\alpha}\sqrt{2\bar p\bar q} + z_{1-\beta}\sqrt{p_1q_1 + p_2q_2}\big)^2/\Delta^2$ . Introduced in: Lesson 6
CI pipeline: The jobs that CI runs for one trigger, such as a pull request: each job is a list of steps on one runner, and jobs share no files except through artifacts or a cache. Introduced in: Lesson 3
Cohen's kappa: Agreement corrected for chance, $\kappa = (p_o - p_e)/(1 - p_e)$ , with observed agreement $p_o$ and chance agreement $p_e$ . $\kappa = 0$ is chance level; $\kappa = 1$ is perfect. Introduced in: Lesson 9
concept drift: A change of $p(y \mid x)$ : the same inputs now have other labels. It is visible only with labels. Introduced in: Lesson 7
container: A normal process on the host's Linux kernel, isolated with namespaces (its own view of files, processes, network) and cgroups (resource limits). Introduced in: Lesson 3
content hash: A hash (such as SHA-256) of the exact bytes of a file or a set of rows. It is a true version of data; a path is not, because the files behind it can change. Introduced in: Lesson 1
continuous batching: Batching for LLM servers in which new requests join the running batch at each decoding step, instead of waiting for a batch to finish. Introduced in: Lesson 5
continuous integration (CI): Automatic checks (tests, lint, an image build) that run on every change before it is merged. Introduced in: Lesson 3
continuous training (CT): Deploying the training pipeline itself, so that it retrains on triggers: on demand, on a schedule, on new data, on a performance drop or on drift. Introduced in: Lesson 6
covariate shift: A change of the input distribution $p(x)$ while $p(y \mid x)$ stays the same. Introduced in: Lesson 7
CUDA compatibility rule: The GPU driver comes from the host, not the image, and must be new enough for the CUDA version inside the image (with minor-version and forward compatibility as exceptions). Introduced in: Lesson 3
DAG: A directed acyclic graph of tasks with dependencies, such as ingest, check, features, train, evaluate. A DAG run is one execution of the whole graph. Introduced in: Lesson 4
data contract: A list of rules that every batch of data must pass: presence, dtype, nulls, range, domain and uniqueness of each column. Introduced in: Lesson 1
data interval: The time range $[\text{start}, \text{end})$ that one scheduled run processes. The logical date of the run is the start of its data interval. Introduced in: Lesson 4
data test: A test of the data, not of the code: it runs the checks of a data contract and looks for leakage. Introduced in: Lesson 2
Deployment: A Kubernetes object that keeps $N$ identical Pods running, replaces them one by one on an update and can roll back. Introduced in: Lesson 4
digest: The SHA-256 hash of an image's manifest, written image@sha256:.... If any layer changes, the digest changes, so it names one exact image. Introduced in: Lesson 3
directional expectation test (DIR): A behavioural test with a change $T$ of known effect direction $d \in \{+1, -1\}$ . A case fails if $d\,(f(x') - f(x)) < -\tau$ for some $x' \in T(x)$ . Introduced in: Lesson 2
discordant pairs: In a paired comparison of two models, the rows where they disagree: $b$ rows where only the old model is right and $c$ rows where only the new model is right. Introduced in: Lesson 2
Dockerfile: The recipe that builds an image, one instruction per line. Introduced in: Lesson 3
domain classifier: A classifier trained to separate reference rows from production rows. A held-out AUC near 0.5 means no detectable shift; it also catches changes in the joint distribution. Introduced in: Lesson 7
dynamic batching: Collecting requests in a queue and sending them to the model as one batch when the queue holds $B$ requests or the oldest has waited the max queue delay. Introduced in: Lesson 5
effect size: A number that measures how much a distribution moved and does not grow with the sample size, such as PSI or $D$ . Alerts use effect sizes, not p-values. Introduced in: Lesson 7
egress: Data that leaves a cloud region or the cloud. Providers charge for it per GB. Introduced in: Lesson 4
entity: The object a feature describes, such as a user, a station or a grid cell. A feature value belongs to an entity at an event time. Introduced in: Lesson 8
eval gate: The rule, fixed before the run, that decides if a new app version may ship: no critical assertion fails, the pass-rate drop is below a margin, cost and latency are within budget. Introduced in: Lesson 9
experiment tracker: A tool (such as MLflow or Weights & Biases) that keeps one record per training run: versions of code, environment and data, config, seeds, hardware and metrics. Introduced in: Lesson 1
feature store: A system that holds each feature definition once and provides its values both for training (offline store) and for serving (online store). Introduced in: Lesson 8
future leakage: Training rows that get feature values from after their label time. Offline metrics become too good, and online the model meets other inputs. Introduced in: Lesson 8
gang scheduling: Starting all nodes of a multi-node job together, or none of them. HPC schedulers such as Slurm do this; Kubernetes needs add-ons. Introduced in: Lesson 4
golden set: A fixed set of test cases on which every new app version and the current one are run and compared case by case. Introduced in: Lesson 9
group-sequential design: A plan of a few looks at the data with stricter thresholds, so that looking early does not raise the false-alarm rate. Pocock uses one threshold; O'Brien–Fleming is strict early. Introduced in: Lesson 6
guardrails: Run-time checks around the model: input checks (injection, personal data, scope) and output checks (schema, policy, grounding), with a rule for what to do on failure. Introduced in: Lesson 9
hash routing: Assigning each user to a version by a salted hash of the user id, so the assignment is sticky, uniform and independent across experiments. Introduced in: Lesson 6
hash split: A train/validation/test split that puts each row into a split from a hash value $u(k) \in [0, 1)$ of its key $k$ . New rows never move old rows to another split. Introduced in: Lesson 1
Horizontal Pod Autoscaler (HPA): A controller that changes the number of replicas: $N' = \lceil N\,m/t \rceil$ for $N$ Pods, current mean metric $m$ and target $t$ , with a tolerance and a scale-down window. Introduced in: Lesson 4
hysteresis: An alert rule with two levels: it fires above the alert level and clears only below a lower clear level, so one incident gives one page. Introduced in: Lesson 8
idempotency: Running a task twice with the same inputs leaves the same state as running it once. It makes retries and backfills safe. Introduced in: Lesson 4
image: The file system and configuration a container starts from: a stack of read-only layers plus the command to run and environment variables. Introduced in: Lesson 3
invariance test (INV): A behavioural test with a label-preserving change $T$ of the input. A case fails if $\max_{x' \in T(x)} |f(x') - f(x)| > \tau$ . Introduced in: Lesson 2
Jeffreys divergence: The symmetrised KL divergence $\mathrm{KL}(a\|e) + \mathrm{KL}(e\|a)$ . On binned data it equals the PSI. Introduced in: Lesson 7
Job and CronJob: A Job runs Pods until they finish successfully and retries failures; a CronJob creates a Job on a cron schedule. Introduced in: Lesson 4
Kubernetes: A system that runs containers on a cluster of machines (nodes). You describe objects in YAML, and controllers make the cluster match them. Introduced in: Lesson 4
label shift: A change of the label distribution $p(y)$ while $p(x \mid y)$ stays the same. Introduced in: Lesson 7
latency: The time from the arrival of a request to its answer: waiting time in the queue plus service time. Introduced in: Lesson 5
layer: A set of file changes in an image, added by one RUN, COPY or ADD instruction. The builder reuses a cached layer until one of its inputs changes. Introduced in: Lesson 3
lineage: The record of what a model version was built from: code commit, image digest, data version and config hash, so the version can be rebuilt. Introduced in: Lesson 6
Little's law: In any stable system, the mean number of requests in the system equals arrival rate times mean time in the system: $L = \lambda W$ . Introduced in: Lesson 5
liveness probe: A check that the process is alive. When it fails, Kubernetes restarts the container, so it must never depend on the model or a database. Introduced in: Lesson 4
LLM judge: An LLM that grades outputs, by pairwise comparison, single-answer grading or against a reference answer. It has position, verbosity and self-preference biases. Introduced in: Lesson 9
M/G/1 queue: One server, Poisson arrivals and any service-time distribution. Its mean waiting time (Pollaczek–Khinchine) still grows like $1/(1-\rho)$ ; constant service (M/D/1) halves the M/M/1 wait. Introduced in: Lesson 5
M/M/1 queue: One server, Poisson arrivals with rate $\lambda$ and exponential service times with rate $\mu$ . Its mean latency is $W = 1/(\mu - \lambda)$ . Introduced in: Lesson 5
match rate: The share of logged serving rows whose feature value equals the value recomputed by the training pipeline (exactly or within a tolerance). It measures training-serving skew. Introduced in: Lesson 8
materialisation: The scheduled job that loads the latest feature values from the offline store (or a batch source) into the online store. Introduced in: Lesson 8
McNemar's test: A paired test of two models on the same rows that uses only the discordant pairs. If both are equally accurate, $c \sim \text{Binomial}(b + c, 1/2)$ . Introduced in: Lesson 2
metamorphic testing: Testing without the right output: you know how the output must relate to the output of a changed input. INV and DIR tests come from it, so they need no labels. Introduced in: Lesson 2
minimum functionality test (MFT): A behavioural test with simple inputs and known labels, aimed at one capability. A case fails if the thresholded score differs from the label. Introduced in: Lesson 2
ML lifecycle: The loop of work around a model: data, validate and split, train, test, package, orchestrate, serve, release, monitor, and back to data (retrain). Introduced in: Lesson 1
ML Test Score: A checklist of 28 tests in four sections (data, model, infrastructure, monitoring) by Breck et al. (2017). The score is the minimum of the four section sums. Introduced in: Lesson 2
model card: A short document stored with a model version: intended use, training and evaluation data, metrics per group or slice, and known limits. Introduced in: Lesson 6
model registry: A store of every model version with its artifact, lineage, metrics and tags, plus a movable alias that says which version serves. Introduced in: Lesson 6
monitor: A rule that turns a stream of scores (one per window) into alerts that a person must act on. Introduced in: Lesson 8
multi-stage build: A build that installs compilers and tools in a first stage and copies only the result into a small final stage. Introduced in: Lesson 3
non-inferiority test: A test that passes the candidate only if the one-sided 95% upper bound of its drop is below a margin $\Delta$ fixed in advance: $(b - c)/N + 1.645\,\mathrm{SE}_{\text{paired}} < \Delta$ . Introduced in: Lesson 9
object storage: A map from keys to objects (blobs of bytes) used over HTTP, such as Amazon S3. It is not a filesystem: you put, get, list and delete whole objects. Introduced in: Lesson 4
offline and online store: The offline store keeps the full history of feature values with timestamps for training; the online store keeps the latest value per entity for low-latency serving. Introduced in: Lesson 8
percentile (p50, p99): The $q$ -th percentile of $n$ latencies is $x_{(\lceil qn/100 \rceil)}$ (nearest rank). The p50 is the median; the p99 is the latency that 99% of requests stay below. Introduced in: Lesson 5
Pod: The smallest unit Kubernetes runs: one or more containers that share a network address and are scheduled together on one node. Introduced in: Lesson 4
point-in-time correct join: A join that gives a label row $(u, t)$ the latest feature row of entity $u$ with time $s < t$ (strict) or $s \le t$ (inclusive), and $s \ge t - \text{TTL}$ . Introduced in: Lesson 8
population stability index (PSI): For $k$ bins with reference proportions $e_i$ and current proportions $a_i$ , $\mathrm{PSI} = \sum_i (a_i - e_i)\ln(a_i/e_i) = \mathrm{KL}(a\|e) + \mathrm{KL}(e\|a)$ . Introduced in: Lesson 7
power: The probability $1 - \beta$ that a test detects a real effect of a given size. Introduced in: Lesson 6
pre-training test: A test that the training code can learn at all, for example that the model can overfit one batch of 8 examples. Introduced in: Lesson 2
prediction drift: A change of the distribution $p(\hat y)$ of the model's outputs. It depends only on the inputs, so it is seen without labels. Introduced in: Lesson 7
promotion gate: A set of rules in code that a candidate must pass before promotion: absolute bounds, a primary metric that beats production by a margin, and guard metrics that may not get worse. Introduced in: Lesson 6
prompt injection: An input that makes the model follow the attacker's instructions instead of the app's, directly or hidden in a retrieved document. Guardrails reduce this risk but do not remove it. Introduced in: Lesson 9
quarantine: Storing a batch that fails its checks aside for inspection, while the system keeps using the last good data. Introduced in: Lesson 1
readiness probe: A check that a Pod may receive traffic now. When it fails, Kubernetes removes the Pod from the Service without a restart. Introduced in: Lesson 4
recall@k: The share of labelled relevant documents that appear in the top $k$ retrieval results. Introduced in: Lesson 9
reconciliation: You declare the desired state; a control loop compares it with the observed state and acts on the difference, forever. Kubernetes works this way. Introduced in: Lesson 4
reference and current window: The two samples a drift statistic compares: the reference (usually the training data) and the current window (for example last week's data). Introduced in: Lesson 7
registry (container): A server (Docker Hub, GitHub Container Registry, a cloud registry) that stores images by digest and keeps their tags. Introduced in: Lesson 3
regression test: A test that scores the candidate and the production model on the same frozen test set, overall and per slice, and flags a slice whose metric drops by more than a threshold. Introduced in: Lesson 2
reproducibility: The property that the same inputs give the same output. For a training run the inputs are code, environment, data, seeds and hardware. Introduced in: Lesson 1
requests and limits: Resource requests are used for scheduling a Pod onto a node; limits are enforced while it runs (CPU above the limit is throttled, memory above it gets the container OOM-killed). Introduced in: Lesson 4
resolved config: The full configuration of a run after defaults and command-line overrides are applied. Its config hash is the SHA-256 of the config written as canonical JSON with sorted keys. Introduced in: Lesson 1
Rogan–Gladen correction: The true pass rate from a judge's observed pass rate: $\theta = (p_{\text{obs}} + \mathrm{TNR} - 1)/(\mathrm{TPR} + \mathrm{TNR} - 1)$ , valid if $\mathrm{TPR} + \mathrm{TNR} > 1$ . Introduced in: Lesson 9
rule of three: If $n$ independent test cases all pass, the one-sided 95% upper confidence bound on the failure rate is $1 - 0.05^{1/n} \approx 3/n$ . Introduced in: Lesson 2
runbook: A short list, attached to an alert, of what to check and what to do. Introduced in: Lesson 8
salt: A fixed string that is hashed together with each key, such as split-v1. A new salt gives a new, independent assignment of keys to splits or to release groups. Introduced in: Lesson 1
schema check: A check of one batch of data against the rules of a data contract. Introduced in: Lesson 1
sequential probability ratio test (SPRT): Wald's test that updates the likelihood ratio $\Lambda_n$ after each observation and stops when $\Lambda_n \ge 1/\alpha$ . The mixture SPRT gives always-valid p-values. Introduced in: Lesson 6
Service (Kubernetes): A stable name and virtual IP address that spreads requests over the ready Pods with a given label. Introduced in: Lesson 4
shadow release: A release in which the new version gets live inputs but nobody uses its outputs; they are only logged and compared. Introduced in: Lesson 6
shard: One immutable file of about 100 MB to 1 GB that packs many samples, listed with its hash in a manifest. Shards avoid millions of slow small objects. Introduced in: Lesson 4
shortcut: A feature that predicts the label in the training data for a wrong reason. A behavioural test finds it when the score moves after an irrelevant change, such as a name. Introduced in: Lesson 2
slice: A subset of rows defined by metadata, such as region, season, sensor model or a rare class. A better average can hide a much worse slice. Introduced in: Lesson 2
SLO (service level objective): A target for a service, such as "p99 latency below 100 ms". Introduced in: Lesson 5
spot instance: Spare cloud capacity at a large discount that the provider can take back after a short notice. Introduced in: Lesson 4
staged rollout: Raising the share of traffic on a new version in steps, such as 1%, 5%, 25%, 100%, with a check at each step. Introduced in: Lesson 6
startup probe: A check that a container has finished starting. It holds back the other probes and restarts the container if it does not succeed within its time budget. Introduced in: Lesson 4
statistical reproducibility: The same metrics within the run-to-run spread, measured over several seeds. Papers and model comparisons need it. Introduced in: Lesson 1
status code: The number in an HTTP answer: 200–299 success, 400–499 a client error (such as 422 for invalid input), 500–599 a server error (such as 503 when not ready). Introduced in: Lesson 5
tag: A movable name for an image, such as nowcast:latest. It can point to a different digest tomorrow, so it is not a version unless it is never reused. Introduced in: Lesson 3
throughput: The number of requests finished per second. Introduced in: Lesson 5
TPR and TNR of a judge: TPR $= P(\text{judge pass} \mid \text{expert pass})$ and TNR $= P(\text{judge fail} \mid \text{expert fail})$ , measured on traces labelled by an expert. Introduced in: Lesson 9
trace: The log of one request to an LLM application: the input, the retrieved text, the output, latency and cost. Introduced in: Lesson 9
training-serving skew: A pipeline bug in which the same example gets different input values offline (training) and online (serving). Introduced in: Lesson 7
TTL (time to live): The maximum age of a valid feature value. Older values count as missing, offline and online. Introduced in: Lesson 8
two-sample KS statistic: The largest vertical gap between two empirical CDFs, $D = \sup_t |F_x(t) - F_y(t)|$ . It needs no binning. Introduced in: Lesson 7
utilisation: The share of time a replica is busy, $\rho = \lambda/\mu$ . A queue is stable only if $\rho < 1$ . Introduced in: Lesson 5
virtual machine (VM): A whole guest operating system with its own kernel on emulated hardware. It is heavier than a container, which shares the host kernel. Introduced in: Lesson 3
window and stride: A monitor computes its score on the last $w$ values (the window) every $s$ new values (the stride). A short window reacts fast but is noisy. Introduced in: Lesson 8
Young's checkpoint interval: With checkpoint cost $C$ and mean time $M$ between interruptions, the wasted fraction is $w(\tau) \approx C/\tau + \tau/(2M)$ , smallest at $\tau^* = \sqrt{2CM}$ . Introduced in: Lesson 4