LLMOps: versioning, eval gates, LLM judges and guardrails

Before this lesson: lesson 02 (behavioural and regression tests, McNemar's test: an eval suite is the same idea); lesson 06 (release gates, canaries, rollback); how LLMs are evaluated and retrieval, tools and agents; hypothesis testing and Bayes' rule (base rates). Time: about 55 minutes, plus the notebook.

Why this matters #

A one-sentence prompt edit can break the output format for 3% of requests. A provider can change the model behind an alias such as "latest". A new embedding model can make retrieval worse while every answer still looks fluent. Evals are the tests of an LLM application, and the gate in CI is where they protect users. "How do you evaluate and ship a change to an LLM feature?" is a standard question for ML engineer roles as of 2026. The same holds in science: an assistant that answers questions about a climate model's configuration, or a pipeline that turns field reports into database rows, needs the same gate.

1. Predict first #

2. The picture #

The application is a function of its configuration. Write

The configuration $c$ holds the prompt template, the model snapshot, the decoding parameters, the tool definitions and the retrieval setup. A simple example: an app reads a weather station's maintenance report and returns JSON with the fields station_id, rain_mm and sensor_ok. Any change to $c$ can change the output for any input.

Three consequences give the whole lesson:

1. Version $c$ as one unit, so that every output can be traced back to the configuration that made it.
2. Test each new $c$ on a fixed set of inputs before it ships (the eval gate), and compare it with the current $c$ on the same inputs.
3. Check inputs and outputs at run time (guardrails), for the cases that no test set covers.

Three levels of evals (Hamel Husain). Level 1: assertion checks in code; cheap, run on every change. Level 2: human and LLM-judge review of logged traces (the input, the retrieved text, the output, latency and cost of one request), on a regular schedule. Level 3: A/B tests with users, after large product changes. The cost grows from level 1 to level 3.

Why compare case by case. Run the baseline and the candidate on the same $N$ golden cases. Most cases give the same result for both: easy cases pass for both, hard cases fail for both. They cancel in the difference. Only the cases where the two versions disagree say which one is better.

3. Definitions and main results #

One app version #

Pin and record, in one configuration file:

• prompt: the template in the repository (or a prompt registry), with an id or a content hash;
• model: a dated snapshot id, not a moving alias such as "latest";
• decoding: temperature, top-p, maximum tokens, stop sequences, and a seed if the provider supports one;
• tools: names and JSON schemas;
• retrieval: the embedding model and its version, the chunking (size, overlap), the corpus snapshot, the index build id, top-$k$, the reranker;
• the judge (its model and prompt), because a new judge changes the scores.

The hash of this file is the app version. Log it with every trace. Then a regression points to a version, and rollback works as in lesson 06.

A new embedding model needs a new index. An embedding model maps text to vectors. Vectors from two different embedding models live in different spaces, so a query vector from the new model cannot be compared with document vectors from the old one. Re-embed the whole corpus, build a new index, switch to it in one step (keep the old one for rollback), and evaluate retrieval itself: recall@$k$ (the share of labelled relevant documents that appear in the top $k$ results), not only the final answers.

The eval suite as a regression gate #

• Golden set. Curated inputs with an expected output or a grading rubric. Build it from real traces, above all from failures, with slices (topics, languages, hard and adversarial cases). Keep a held-out part that is never used to tune prompts. Every production failure becomes a new case, so the set does not go stale.
• Assertion checks. Deterministic code: the JSON matches the schema, required fields exist, cited ids are among the retrieved chunks, numbers are in range, out-of-scope questions are refused, no personal data appears. Exact, cheap, and run on every change.
• LLM judge. For qualities that code cannot check: faithfulness to the retrieved text, helpfulness, tone. A binary pass/fail with a written rubric is easier to calibrate than a 1–5 score.
• Noise. With temperature above 0 (and often even at 0, because serving is not always deterministic), the same case can pass on one run and fail on another. Run each case several times (the eval framework Inspect calls these epochs) and use the mean.
• The gate rule, fixed before the run: block if any critical assertion fails; block if the drop in pass rate may exceed a margin, overall or on an important slice (the non-inferiority test below); block if cost or latency exceeds its budget.

Result 1: the paired comparison and non-inferiority #

Run the baseline and the candidate on the same $N$ cases. Let $b$ be the number of cases that only the baseline passes, and $c$ the number that only the candidate passes. (These are the discordant pairs of lesson 02.)

1. The drop. The drop in pass rate is $(b - c)/N$: cases that both pass or both fail cancel.
2. McNemar's test. Under $H_0$ (no difference), each discordant case goes either way with probability $1/2$. So, given $b + c$, $b \sim \text{Binomial}(b + c, 1/2)$.
3. The p-value for a regression is one-sided: $P(B \ge b)$ with $B \sim \text{Binomial}(b + c, 1/2)$. For large $b + c$, $z = (b - c)/\sqrt{b + c}$ is approximately standard normal, and $z^2$ is McNemar's $\chi^2_1$ statistic.
4. The paired standard error. Write $d_i \in \{+1, -1, 0\}$ for the per-case drop (baseline minus candidate). Its sample standard deviation divided by $\sqrt N$ is the standard error of the drop:

5. Non-inferiority. Fix a margin $\Delta$ in advance: the largest drop you accept. The candidate passes the gate only if the one-sided 95% upper bound of the drop is below the margin:

Why the two tests differ. McNemar's test asks "is there any evidence of a drop?". A large p can just mean that there were too few discordant cases to see anything. The non-inferiority gate puts the burden of proof on the candidate: it must show that the drop is small.

Why pairing helps. An unpaired test treats the two pass rates as independent and adds their variances. But both rates move together with the difficulty of the cases. The paired test uses only the $b + c$ discordant cases, so it has a smaller standard error (worked example A: 2.34 points instead of 3.62).

LLM-as-judge and its biases #

A judge model can compare two answers (pairwise comparison), grade one answer (single-answer grading), or grade against a reference answer (reference-guided grading). Zheng et al. (2023) found that GPT-4 as a judge agreed with human experts on 85% of pairwise votes (without ties), while two humans agreed on 81%. They also measured these biases:

Judges are also weak at grading maths: they are misled by the answer they grade. On 20 judgements of maths answers, a default GPT-4 judge failed 14 times; a reference-guided judge (it first solves the problem itself, then uses its own answer as the reference) failed 3 times.

Calibrating the judge against an expert #

A domain expert labels 100–200 traces as pass or fail, each with a one-line critique. Split them into a training part (examples for the judge prompt), a development part (to iterate on the prompt), and a test part (measured once, at the end). With the expert as the truth and "pass" as the positive class:

• TPR (true positive rate) $= P(\text{judge says pass} \mid \text{expert pass})$;
• TNR (true negative rate) $= P(\text{judge says fail} \mid \text{expert fail})$;
• Cohen's kappa $\kappa = (p_o - p_e)/(1 - p_e)$. Here $p_o$ is the observed agreement and $p_e = \pi_J \pi_E + (1 - \pi_J)(1 - \pi_E)$ is the agreement expected by chance, from the judge's pass rate $\pi_J$ and the expert's pass rate $\pi_E$. $\kappa = 0$ means chance-level agreement; $\kappa = 1$ means perfect agreement.

Raw agreement misleads when failures are rare. A judge that always says "pass" agrees with the expert on every good trace and catches no failure. On traces where 75% pass, it has 75% agreement and $\kappa = 0$.

Result 2: correcting the judge's pass rate (Rogan–Gladen) #

Let $\theta$ be the true pass rate on new traces, and $p_{\text{obs}}$ the pass rate that the judge reports.

1. The judge says "pass" for a good trace with probability TPR, and for a bad trace with probability $1 - \text{TNR}$.
2. By the law of total probability, $p_{\text{obs}} = \theta\,\mathrm{TPR} + (1 - \theta)(1 - \mathrm{TNR}) = \theta\,(\mathrm{TPR} + \mathrm{TNR} - 1) + 1 - \mathrm{TNR}$.
3. Solve for $\theta$:

valid if $\mathrm{TPR} + \mathrm{TNR} > 1$. A judge at chance has $\mathrm{TPR} + \mathrm{TNR} = 1$: its verdict is independent of the truth and carries no information. $\blacksquare$

This assumes that the judge's TPR and TNR on new traces equal those on the labelled set. So re-check them when the app, the traffic or the judge changes.

Guardrails #

Guardrails are run-time checks around the model:

• input checks: prompt-injection and jailbreak detection, detection or removal of personal data, a topic or scope filter, size limits;
• output checks: schema validation (a JSON schema, or a structured-output mode that constrains decoding), policy and toxicity classifiers, personal-data leaks, grounding checks (claims and citations supported by the retrieved text), an allow-list for tool calls;
• on failure: block with a safe message, repair or retry (one more model call), fall back to a simpler path, or send to a human.

Prompt injection is an input that makes the model follow the attacker's instructions instead of the app's, either directly or hidden in a retrieved document or web page. Guardrails reduce this risk; they do not remove it. As of 2026, the OWASP Top 10 for LLM applications lists prompt injection first (LLM01:2025) and says that it is unclear whether fool-proof prevention exists. So also limit what the model can do: tools with narrowly scoped credentials, and human approval for high-risk actions.

Costs.

• Latency. Each check is a classifier or an LLM call. Input checks can run in parallel with generation. Output checks need the output, which conflicts with streaming (buffer it, or check chunks).
• Tokens and money. An LLM-based check can cost as much as the main call; a retry after a schema failure is a second call.
• False positives. Valid requests are blocked. When attacks are rare, most blocks hit valid users (base-rate effect, worked example C).
• False negatives and maintenance. No check is complete. The thresholds and prompts of the checks are versioned and evaluated like the app itself.

Answers to section 1. (1) No, not on this evidence. With $b = 14$ and $c = 8$, McNemar's one-sided p-value is 0.14, so a significance test does not block. But "not significant" is not "safe": 22 discordant cases give little power. The paired standard error of the 3-point drop is 2.34 points, so the one-sided 95% upper bound of the drop is $3 + 1.645 \times 2.34 = 6.85$ points. A gate with a 5-point margin blocks it. Read the 14 failures, check the slices, and add cases or epochs. (2) Not necessarily. In worked example B, a judge with 88% agreement misses 30% of the real failures (TNR = 0.70), and an always-pass judge reaches 75% agreement on the same traces. Look at TNR and $\kappa$, not at raw agreement.

4. Worked example #

A. The paired gate. $N = 200$, the baseline passes 172, $b = 14$, $c = 8$, so the candidate passes $172 - 14 + 8 = 166$.

1. Drop: $(14 - 8)/200 = 0.03$, 3 points.
2. Exact one-sided p-value: $P(B \ge 14)$ for $B \sim \text{Binomial}(22, 1/2)$, which is $0.143$. The normal approximation: $z = 6/\sqrt{22} = 1.28$.
3. Paired standard error: $\frac{1}{\sqrt{200}}\sqrt{(22 - 36/200)/199} = \frac{1}{14.14}\sqrt{0.1096} = 0.0234$.
4. Unpaired standard error: $\sqrt{0.86 \cdot 0.14/200 + 0.83 \cdot 0.17/200} = 0.0362$. Pairing removes about a third of the noise.
5. Gate with a 5-point margin: upper bound $0.03 + 1.645 \times 0.0234 = 0.0685 > 0.05$. Block and investigate.

B. Calibrating a judge. An expert labels 200 traces: 150 pass, 50 fail. The judge says "pass" on 141 of the 150 and "fail" on 35 of the 50.

1. $\mathrm{TPR} = 141/150 = 0.94$, $\mathrm{TNR} = 35/50 = 0.70$.
2. Agreement $p_o = (141 + 35)/200 = 0.88$.
3. The judge says "pass" on $156/200 = 78\%$ of the traces, the expert on $75\%$. So $p_e = 0.78 \cdot 0.75 + 0.22 \cdot 0.25 = 0.585 + 0.055 = 0.64$ and $\kappa = (0.88 - 0.64)/(1 - 0.64) = 0.24/0.36 = 0.667$.
4. An always-pass judge: $p_o = 0.75$, $p_e = 1 \cdot 0.75 + 0 \cdot 0.25 = 0.75$, $\kappa = 0$.
5. On new traces the judge reports $p_{\text{obs}} = 0.80$. Corrected: $\theta = (0.80 + 0.70 - 1)/(0.94 + 0.70 - 1) = 0.50/0.64 = 0.781$. Check: $0.94 \cdot 0.781 + 0.30 \cdot 0.219 = 0.734 + 0.066 = 0.80$ ✓.

The judge is lenient (it passes 30% of the failures), so the true pass rate is lower than reported. Improve its TNR first: add the expert's critiques of failures to its prompt.

C. A guardrail and the base rate. 0.5% of requests are injection attempts. A detector catches 95% of them and flags 2% of valid requests. Per $10^6$ requests:

1. Attacks: $5{,}000$. Blocked: $0.95 \times 5{,}000 = 4{,}750$. Missed: 250.
2. Valid requests: $995{,}000$. Blocked: $0.02 \times 995{,}000 = 19{,}900$.
3. Precision of a block: $4{,}750/(4{,}750 + 19{,}900) = 4{,}750/24{,}650 = 19.3\%$.

About four of five blocks hit a valid user. Use a higher threshold, or a softer action (log, warn, send to human review) instead of a block.

5. In code #

NumPy, SciPy and scikit-learn check the three worked examples; the JSON check uses only the standard library.

import json
import numpy as np
from scipy import stats
from sklearn.metrics import cohen_kappa_score

# A. Paired gate: McNemar's exact test on the discordant cases
b, c, N = 14, 8, 200
r = stats.binomtest(b, b + c, 0.5, alternative="greater")
d = np.r_[np.ones(b), -np.ones(c), np.zeros(N - b - c)]   # per-case drop: +1, -1 or 0
se = d.std(ddof=1) / np.sqrt(N)
print(f"p = {r.pvalue:.3f}, drop = {d.mean():.3f}, SE = {se:.4f}, upper = {d.mean() + 1.645 * se:.4f}")
# p = 0.143, drop = 0.030, SE = 0.0234, upper = 0.0685

# B. Judge vs expert: TPR, TNR, kappa, and the corrected pass rate
expert = np.r_[np.ones(150), np.zeros(50)]
judge = np.r_[np.ones(141), np.zeros(9), np.zeros(35), np.ones(15)]
tpr, tnr = judge[expert == 1].mean(), 1 - judge[expert == 0].mean()
print(f"agreement = {(judge == expert).mean():.2f}, kappa = {cohen_kappa_score(expert, judge):.3f}, "
      f"TPR = {tpr:.2f}, TNR = {tnr:.2f}")
# agreement = 0.88, kappa = 0.667, TPR = 0.94, TNR = 0.70
print(f"always-pass kappa = {cohen_kappa_score(expert, np.ones(N)):.1f}")       # 0.0
print(f"corrected pass rate = {(0.80 + tnr - 1) / (tpr + tnr - 1):.3f}")        # 0.781

# C. An output guardrail: check the model's JSON against a small schema
SCHEMA = {"station_id": str, "rain_mm": float, "sensor_ok": bool}
def valid(raw):
    try:
        obj = json.loads(raw)
    except json.JSONDecodeError:
        return False
    return isinstance(obj, dict) and all(isinstance(obj.get(k), t) for k, t in SCHEMA.items())

print(valid('{"station_id": "S1", "rain_mm": 12.5, "sensor_ok": true}'))      # True
print(valid('{"station_id": "S1", "rain_mm": "a lot", "sensor_ok": true}'))   # False: retry, repair or fall back

In a real service, a schema library (for example Pydantic or jsonschema in Python) does the same check with better error messages, and structured-output modes of many model APIs constrain decoding so that the JSON is valid by construction.

6. Where it shows up #

• LLM products. RAG assistants, coding agents, extraction pipelines, support chat. Providers retire model snapshots, so a forced migration is a change like any other: run the gate.
• Science. An assistant over the documentation of an Earth-system model (namelists, parameters); a literature assistant; an agent that writes job scripts for an HPC scheduler such as Slurm (an assertion check can run sbatch --test-only, which validates a script without submitting it); extraction of field reports into a database (schema validation plus a golden set).
• Tools (as of 2026). Inspect (UK AI Security Institute): a task is a dataset, a solver and a scorer; built-in scorers include includes(), match(), model_graded_qa() and model_graded_fact(); epochs repeats each sample. Trace stores such as Langfuse or LangSmith keep the logged traces that feed level-2 review and the golden set.
• Same ideas, other lessons. The eval gate is the release gate of lesson 06 with an LLM in the loop; the paired test is the slice test of lesson 02; watching judge scores over time is the monitoring of lesson 08.

7. Common mistakes #

• A model alias instead of a pinned snapshot. The app changes without a commit.
• A new embedding model on the old index. Re-embed the corpus and rebuild the index.
• One run per case at temperature above 0, compared unpaired. Use epochs and a paired test.
• "Not significant" read as "safe". Use a non-inferiority margin fixed in advance.
• The judge as ground truth. Measure TPR and TNR against an expert; report kappa, not raw agreement.
• Tuning the judge prompt (or the app prompt) on the test cases. Keep a held-out part.
• A golden set that never changes. Add production failures to it.
• Guardrails without numbers. Measure their false-positive rate and added latency like any other model.

8. Check yourself #

1. List what goes into the app version of a RAG assistant.
   Answer

   The prompt template id or hash; the model snapshot id; the decoding parameters; the tool schemas; the embedding model and version, chunking, corpus snapshot, index build id, top-$k$ and reranker; the judge model and prompt. Its hash is logged with every trace.
2. On a 50-case safety slice, the baseline passes 3 cases that the candidate fails, and the reverse never happens. What is the one-sided exact p-value, and what does it teach?
   Answer

   $P(B \ge 3)$ for $B \sim \text{Binomial}(3, 1/2)$ is $1/8 = 0.125$. A small slice cannot reach significance, so critical behaviour needs hard assertion gates (any failure blocks), not a significance test.
3. Why does a new embedding model need a full re-index, and what do you evaluate before switching?
   Answer

   The new model maps text into a different vector space, so query vectors cannot be compared with old document vectors. Re-embed the corpus, build a new index, and compare retrieval (recall@$k$ on labelled query–document pairs) and end-to-end answers with the old version.
4. A judge has TPR = 0.80 and TNR = 0.20. What is its Youden's $J = \mathrm{TPR} + \mathrm{TNR} - 1$, and can you correct its pass rate?
   Answer

   $J = 0$. The judge says "pass" with probability 0.80 whatever the truth, so its verdict is independent of the truth. The correction divides by 0: the judge is useless.
5. Why should the temperature of the app not be set to 0 just to make evals reproducible?
   Answer

   The eval must test the configuration that users get. If production samples at temperature 0.7, an eval at 0 measures another app. Keep the production settings and run several epochs per case, then use the mean pass rate.
6. A judge-based metric on production traces drops from 0.90 to 0.84 after a provider changes the judge model behind an alias. Did the app get worse?
   Answer

   Not necessarily: the judge is part of the measurement, and it changed. Pin the judge's snapshot, re-measure its TPR and TNR on the expert-labelled set, and compare the corrected pass rates. Otherwise a change of the ruler looks like a change of the app.

9. Interview angle #

• "How do you ship a prompt change safely?" A versioned app configuration; a golden set with assertion checks and a calibrated judge in CI; a paired comparison with the baseline, overall and per slice, with a margin fixed in advance; then a canary or an A/B test.
• "How do you know your LLM judge is right?" Expert labels with critiques; TPR, TNR and kappa on a held-out part; correct the pass rate; re-check after changes of the app, traffic or judge.
• "Which biases do LLM judges have?" Position, verbosity, self-preference; weak grading of maths. Name one mitigation for each.
• "What do guardrails cost?" Latency, tokens, false positives (base rates), maintenance; they do not stop prompt injection completely, so also limit tool permissions.
• "How would you evaluate a RAG system?" Retrieval (recall@$k$, MRR) separately from generation (faithfulness, correctness); a calibrated judge; a paired comparison with the current version.

10. Practice #

The notebook for this lesson runs in Colab. It has four exercises, each with a check cell and a folded solution:

1. Write paired_gate(base_pass, cand_pass, margin): drop, exact McNemar p, paired and unpaired standard errors, upper bound and verdict; reproduce worked example A.
2. Write judge_report(expert, judge) (TPR, TNR, agreement, kappa) and corrected_pass_rate(p_obs, tpr, tnr), then a bootstrap interval for the corrected rate.
3. Write validate_output(text, schema) with the json module and count the failures in a list of fake model outputs.
4. Simulate a pairwise judge with position bias, and show that judging both orders and recording ties removes the bias.

11. Watch and read #

• Hamel Husain, Your AI product needs evals: the three levels (unit tests, human and model eval, A/B tests) and logging traces.
• Hamel Husain, Using LLM-as-a-Judge for evaluation: expert pass/fail labels with critiques, and measuring the judge against the expert.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023): §3.3 (position, verbosity, self-enhancement, maths), §3.4 (mitigations), §4.2 (agreement with humans).
• OWASP, LLM01:2025 Prompt Injection: types of injection and mitigation strategies.
• Inspect documentation (UK AI Security Institute): the pages on tasks (epochs), scorers and model grading.
• Wikipedia, Cohen's kappa.