Testing ML systems: unit, data, behavioural, slice and regression tests

Before this lesson: lesson 01 (data contracts, reproducible runs); evaluation metrics and data leakage; logistic regression; the binomial distribution (common distributions) and hypothesis testing. Time: about 60 minutes, plus the notebook.

Why this matters #

A model can pass every code test and still be wrong. A feature computed with a future timestamp, a model that uses the reviewer's name instead of the review text, or a new version that is better on average but worse in the mountains: none of these raises an error. The tests of this lesson find them. They become the release gate in CI (lesson 03) and in the model registry (lesson 06). LLM eval suites (lesson 09) use the same ideas. "How would you test this model?" is also a standard interview question.

1. Predict first #

2. The picture #

Two kinds of correctness. Code correctness is exact. A function gets an input and must return a known output, so the test is assert f(x) == y. Learned behaviour is statistical. You cannot write down the right output for every input. So you test properties (relations between inputs and outputs) with tolerances, and rates (the share of cases that fail) over many cases.

The simplest model. Take a logistic model of the probability of rain in the next hour,

where $h$ is the relative humidity, $T$ is the temperature, $\sigma(z) = 1/(1 + e^{-z})$ is the sigmoid, and $w_h, w_T, b$ are learned. A directional test says: "more humidity must not lower the probability of rain". The sigmoid is increasing, so $p$ grows with $h$ exactly when $w_h \ge 0$. For this model the property holds for every input if one number has the right sign. One check proves a global property.

The general case. For a neural network $f$, the same test computes $f(x + \delta e_h) - f(x)$ at the inputs you chose. Here $e_h$ is the unit vector of the humidity feature and $\delta > 0$ is a step. The test says nothing about inputs you did not choose. This is the central limit of behavioural testing: a test samples the input space. It can find a violation (a counterexample). It cannot prove that the property holds everywhere.

This picture decides where each test runs. Cheap, exact tests (code and data) run on every commit. Slower, statistical tests (behaviour and slices) run before a model is promoted to production.

3. Definitions and main results #

Test types #

The examples are for the running example of this track: a rain nowcast that gives the probability of rain in the next hour at a weather station.

A data test is a test of the data, not of code: it runs the checks of a data contract and looks for leakage. A pre-training test (for example "the model can overfit one batch of 8 examples") checks that the training code can learn at all. If a small network cannot reach 100% training accuracy on 8 points, the code has a bug: wrong labels, a learning rate of zero, or a gradient that does not reach the weights.

Behavioural tests (CheckList) #

Let $f(x) \in [0, 1]$ be the model's score, $T(x)$ a set of perturbed copies of an input $x$, and $\tau \ge 0$ a tolerance. CheckList (Ribeiro et al., 2020) defines three test types.

• Minimum functionality test (MFT): simple inputs $x$ with known labels $y$, aimed at one capability. A case fails if the thresholded score differs from $y$. It is a unit test for a capability. Example: "The food was not good." must be negative.
• Invariance test (INV): $T$ is label-preserving (the right label does not change). A case fails if $\max_{x' \in T(x)} |f(x') - f(x)| > \tau$. Example: replace the reviewer's name.
• Directional expectation test (DIR): $T$ has a known effect direction $d \in \{+1, -1\}$. A case fails if $d\,(f(x') - f(x)) < -\tau$ for some $x' \in T(x)$. Example: appending "The staff was rude." must not raise the positive score ($d = -1$).

The failure rate of a test is (failing cases) / (tested cases). INV and DIR need no labels: they compare predictions with predictions. So they also run on unlabeled production inputs. MFT comes from software unit tests. INV and DIR come from metamorphic testing: you do not know the right output, but you know how the output must relate to the output of a changed input.

How to design them, in five steps.

1. List the capabilities the model must have. For sentiment: vocabulary, negation, named entities. For a weather model: the response to humidity, to pressure, to the time of day.
2. Write each one as a relation between inputs and outputs: a fixed label (MFT), equality (INV) or an order (DIR).
3. Write a template or a perturbation function that generates many cases.
4. Check with domain knowledge that the perturbation really preserves the label or the direction. Swapping city names preserves the label of a restaurant review. For a weather model it does not: the location carries information.
5. Choose $\tau$, a maximum failure rate, and whether a failure blocks a merge or only reports.

What they can show: a concrete counterexample; a shortcut (a feature that predicts the label in the training data for a wrong reason: the score moves when only a name changes); a missing capability (negation). They treat the model as a black box, so they also compare models trained on different data. What they cannot show: that the property holds for all inputs; accuracy or calibration; failures that nobody thought of. And a test is only as valid as its perturbation.

Result: what a clean run shows (the rule of three) #

Theorem (rule of three). If $n$ independent test cases all pass, the one-sided 95% upper confidence bound on the failure rate $p$ is

Why this is true, first in words. If the model failed often, a clean run of $n$ cases would be very unlikely. The bound $p_u$ is the failure rate at which a clean run becomes too unlikely (less than 5%) to believe.

Proof.

1. If the true failure rate is $p$ and the cases are independent, then $P(\text{0 failures in } n) = (1 - p)^n$.
2. The 95% upper bound $p_u$ is the largest $p$ for which this outcome still has probability at least 0.05: $(1 - p_u)^n = 0.05$.
3. Solve: $1 - p_u = 0.05^{1/n}$, so $p_u = 1 - 0.05^{1/n}$.
4. Take logs: $\ln(1 - p_u) = \ln(0.05)/n$. For small $p_u$, $\ln(1 - p_u) \approx -p_u$, so $p_u \approx -\ln(0.05)/n = 2.996/n \approx 3/n$. $\blacksquare$

So 300 passing cases show $p \lesssim 1\%$ (exactly $0.994\%$), and 3,000 show $p \lesssim 0.1\%$. The bound is about the distribution that generated the test cases. Cases from one template are not a sample of production inputs.

Slice and regression tests #

A slice is a subset of rows defined by metadata: region, season, station type, sensor model, a rare class. A regression test scores the candidate model and the production model on the same frozen test set, overall and per slice. It flags a slice when the slice has at least $n_{\min}$ rows and its metric drops by more than $\delta$. The ML Test Score paper (test Model 6) suggests two kinds of threshold: absolute ("error on slice $x$ below 5%") and incremental ("error on slice $x$ up by less than 1% against the previous model").

Small slices are noisy. An accuracy $a$ measured on $n$ rows has standard error

which is $0.043$ for $a = 0.75$ and $n = 100$. A drop of 3 points on 100 rows is less than one standard error.

Pair the comparison. Both models are scored on the same rows, so most rows give the same result for both: easy rows are right for both, hard rows are wrong for both. These rows carry no information about the difference. McNemar's test uses only the rows where the models disagree, the discordant pairs: $b$ = old right and new wrong, $c$ = old wrong and new right. The drop in accuracy is $(b - c)/n$. If the two models are equally accurate, each discordant row goes either way with probability $1/2$, so $c \sim \text{Binomial}(b + c, 1/2)$. The exact two-sided p-value is $2 \cdot P(X \le \min(b, c))$ with $X \sim \text{Binomial}(b + c, 1/2)$.

The ML Test Score #

Breck et al. (2017) list 28 tests in four sections of seven:

Scoring. Each test gets 0.5 points if it is run manually with documented and shared results, and 1 point if a system runs it automatically and repeatedly. Sum the points per section. The final score is the minimum of the four section sums. The authors chose the minimum because all four areas matter: a team must cover each one to raise the score. Reading the final score (Table V of the paper):

Using it to find missing tests. Score each section. The lowest section is the bottleneck: work there first. Inside it, choose the tests whose failure would cost the most, and turn manual tests into automated ones (+0.5 each).

Answers to section 1. (1) Yes. Section 4 shows a model that gains 1.7 points overall and loses 10 points on alpine stations. The ML Test Score paper gives a stronger example: global accuracy up by 1%, accuracy for one country down by 50%. (2) About 1%. The rule of three gives $1 - 0.05^{1/300} = 0.994\% \approx 3/300$. A failure rate below 1% is still possible after 300 clean cases.

4. Worked example #

(a) A slice regression. A rain classifier is tested on 900 lowland and 100 alpine cases.

Overall: +1.7 points. On alpine stations: −10 points. With $n_{\min} = 30$ and $\delta = 0.02$, the slice test blocks the release.

Is the alpine drop real or noise? Build the paired table on the 100 alpine cases:

Check the margins: old $66 + 14 = 80$ ✓, new $66 + 4 = 70$ ✓, total $66 + 14 + 4 + 16 = 100$ ✓.

1. Discordant rows: $b + c = 18$. The drop is $(b - c)/n = 10/100 = 0.10$ ✓.
2. Under "equally accurate", $c \sim \text{Binomial}(18, 1/2)$.
3. $P(X \le 4) = \big(\binom{18}{0} + \binom{18}{1} + \binom{18}{2} + \binom{18}{3} + \binom{18}{4}\big)/2^{18} = (1 + 18 + 153 + 816 + 3060)/262144 = 4048/262144 = 0.0154$.
4. Two-sided: $p = 2 \times 0.0154 = 0.031$. The drop is unlikely to be noise.

An unpaired comparison of 0.80 and 0.70 gives $\mathrm{SE} = \sqrt{0.8 \cdot 0.2/100 + 0.7 \cdot 0.3/100} = 0.061$ and only $z = 0.10/0.061 = 1.64$, because it ignores that 82 of the 100 rows agree.

(b) An ML Test Score. A nowcasting system has:

• data: schema checks, feature-code unit tests and privacy controls automated ($3 \times 1$); a feature-importance study and a feature-cost review done by hand ($2 \times 0.5$) → 4.0;
• model: spec review and slice metrics automated ($2 \times 1$); tuning and a simple-baseline comparison done by hand ($2 \times 0.5$) → 3.0;
• infrastructure: model specs unit tested (1); a manual canary and a documented rollback ($2 \times 0.5$) → 2.0;
• monitoring: model staleness checked by hand each week (0.5) → 0.5.

Final score: $\min(4.0, 3.0, 2.0, 0.5) = 0.5$, "possible serious holes in reliability". Monitoring is the bottleneck. Automate Monitor 2 (run the lesson 01 contract on serving inputs) and Monitor 3 (recompute features offline for logged requests and compare). Monitoring becomes $0.5 + 1 + 1 = 2.5$, and the final score becomes $\min(4.0, 3.0, 2.0, 2.5) = 2.0$. Now infrastructure is the bottleneck: automate the canary (Infra 6) and the rollback (Infra 7).

(c) INV and DIR tests for a ground-motion model. In seismology, a ground-motion model predicts how strongly the ground shakes at a site during an earthquake. Here it predicts the peak ground acceleration (PGA) from the magnitude $M$, the distance $R$ to the source, and $V_{S30}$, the average shear-wave velocity of the top 30 m of soil (low $V_{S30}$ = soft soil).

• DIR: $R \to 1.1R$ at fixed $M$ and $V_{S30}$ must not increase PGA (shaking weakens with distance).
• DIR: $M \to M + 0.5$ at fixed $R$ and $V_{S30}$ must not decrease PGA.
• INV: the record id, the station code (if it is not a model input) and the order of the records must not change any prediction.
• Not a valid DIR test: "softer soil (lower $V_{S30}$) must raise PGA". It holds for weak shaking, but soft soil responds nonlinearly and can reduce PGA in strong shaking. This test would report correct physics as a failure.

5. In code #

The notebook builds the test harness. Here NumPy, SciPy and scikit-learn check the claims of this lesson.

import numpy as np
from scipy import stats
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.neural_network import MLPClassifier

# 1. A bag of words cannot see word order
texts = ["the food was good, not bad", "the food was bad, not good"]
X = CountVectorizer().fit_transform(texts).toarray()
print((X[0] == X[1]).all())     # True: equal inputs, so equal scores for ANY weights

# 2. Rule of three: 0 failures in n cases -> 95% upper bound on the failure rate
n = 300
print(round(1 - 0.05 ** (1 / n), 5), round(stats.beta.ppf(0.95, 1, n), 5), 3 / n)
# 0.00994 0.00994 0.01

# 3. The slice example of section 4: overall gain, slice loss, paired test
old_overall = (900 * 0.90 + 100 * 0.80) / 1000
new_overall = (900 * 0.93 + 100 * 0.70) / 1000
print(round(old_overall, 3), round(new_overall, 3))     # 0.89 0.907
print(round(stats.binomtest(4, 18, 0.5).pvalue, 3))     # 0.031 (McNemar, exact)

# 4. A pre-training test: a small network must fit one tiny batch
rng = np.random.default_rng(0)
X8, y8 = rng.normal(size=(8, 5)), np.array([0, 1, 0, 1, 1, 0, 1, 0])
mlp = MLPClassifier(hidden_layer_sizes=(32,), solver="lbfgs", max_iter=500, random_state=0)
print(mlp.fit(X8, y8).score(X8, y8))                    # 1.0

stats.beta.ppf(0.95, 1, n) is the exact one-sided bound for 0 failures; it equals $1 - 0.05^{1/n}$. The overfit test is fast, seeded and has a wide margin, so it is not flaky. If it fails, the training code has a bug.

6. Where it shows up #

• ML for science. Physical-consistency checks are behavioural tests: non-negative precipitation (an output range), conservation of mass or energy within a tolerance, invariance or equivariance under rotation or reflection where the physics has that symmetry (INV), and monotone responses such as "more rainfall gives more runoff" (DIR). Scoring a weather model per region, season and lead time is slice testing.
• Industry. A fraud model is sliced by country, payment method and merchant type; a recommender by new and old users. CheckList found many failures in commercial sentiment models that had high test accuracy (Ribeiro et al., 2020).
• Production. Slice and behavioural tests become the promotion gate in CI (lesson 03) and in the registry (lesson 06). The monitoring section of the ML Test Score is lesson 07 and lesson 08. LLM eval suites (lesson 09) are behavioural tests, sometimes with a model as the judge.

7. Common mistakes #

• Asserting exact float outputs of a trained model. They change with seeds, hardware and library versions. Assert properties with tolerances.
• Shipping on the aggregate metric alone. Check slices against production.
• A perturbation that changes the label. City names for a weather model, soil type for strong shaking. Check every perturbation with domain knowledge.
• Reading a passing behavioural suite as proof. It bounds a failure rate on the tested distribution only.
• Blocking on tiny slices. Use $n_{\min}$, a margin $\delta$ and a paired test.
• Comparing two models unpaired on the same rows. The unpaired test ignores that most rows agree, so it is too weak.
• Unseeded or slow model tests on every commit. Seed them, keep them tiny, and move long ones to a nightly or release job.
• Averaging the ML Test Score sections. The score is the minimum.

8. Check yourself #

1. What is the difference between an invariance test and a directional expectation test?
   Answer

   INV applies a label-preserving perturbation and requires the score not to move by more than $\tau$. DIR applies a perturbation with a known effect direction and requires the score not to move the other way (by more than $\tau$). Neither needs labels.
2. A slice of 40 rows drops by 3 accuracy points. Do you block the release?
   Answer

   Not on this alone. At accuracy near 0.75 the standard error on 40 rows is $\sqrt{0.75 \cdot 0.25/40} = 0.068$, so 3 points is about 0.4 standard errors. Report it; for a decision, use a paired test (McNemar) with a margin, or collect more rows. Set $n_{\min}$ so that the slices you gate on can show the drops you care about.
3. Which test type catches each problem? (a) a feature computed with a timestamp after the label time; (b) the serving image has another NumPy version and computes a feature differently; (c) the model ignores negation.
   Answer

   (a) A data (leakage) test. (b) An infrastructure test: training-serving consistency, comparing online and offline features or predictions on logged requests. (c) A behavioural test, an MFT on negation.
4. How many independent passing cases do you need to claim a failure rate below 0.5% with 95% confidence?
   Answer

   Solve $1 - 0.05^{1/n} \le 0.005$: $n \ge \ln 0.05 / \ln 0.995 = 597.6$, so 598 cases. The rule of three gives $3/0.005 = 600$.
5. On a slice, the old model is right and the new one wrong on 3 rows; the reverse happens on 9 rows. Is the new model worse on this slice?
   Answer

   No, it is better: $c = 9 > b = 3$, so the new model gains $(9 - 3)/n$. McNemar: $2 P(X \le 3)$ for $X \sim \text{Bin}(12, 1/2)$ gives $p = 0.146$, so the gain is also not clearly larger than noise.
6. Why is "the score must not change when we replace the station id" a bad INV test for a rain model that takes the station id as an input?
   Answer

   The station id carries real information (altitude, local climate), so changing it can and should change the prediction. The perturbation is not label-preserving. INV tests on ids are valid only for ids that are not model inputs, such as a record id.

9. Interview angle #

• "How would you test an ML model before production?" Walk the table: code unit tests, data tests (contract and leakage), pre-training tests (overfit one batch), behavioural tests for the capabilities that matter, slice regression against production, infrastructure tests (the model loads, online = offline); then a canary.
• "The new model is better on the test set. Why not ship it?" A gain can hide a slice regression or a shortcut. Check slices with a paired test, run behavioural tests, check for leakage and for training-serving consistency.
• "How do you keep ML tests in CI from being flaky?" Seeds, tolerances, tiny data, statistical thresholds with margins, and slow suites outside the commit path.
• "Your behavioural suite passes. What does that tell you?" The failure rate on the tested case distribution is below about $3/n$ with 95% confidence. Nothing more.

10. Practice #

The notebook for this lesson runs in Colab. It has five exercises, each with a check cell and a folded solution:

1. Write check_invariance and check_directional, and catch a planted name shortcut in a bag-of-words review model whose test accuracy looks better than a clean model's.
2. Write an MFT on negation that the bag-of-words model fails, and explain why in one sentence.
3. Write slice_regressions, which flags slices with $n \ge n_{\min}$ and a drop larger than $\delta$, and adds McNemar's p-value.
4. Write rule_of_three_bound(n, conf) and check it with a simulation and the exact binomial.
5. Write ml_test_score(table) with the minimum rule and find the bottleneck section.

11. Watch and read #

• Ribeiro et al., Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (ACL 2020), section 2: capabilities, the three test types, generating test cases at scale. The tool is open source: github.com/marcotcr/checklist.
• Breck et al., The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction (IEEE Big Data 2017): Tables I–IV list the 28 tests, section VI.A and Table V the scoring.
• Made With ML, Testing: tests for data, models (training tests, behavioural tests) and the difference between testing and monitoring.
• Wikipedia, McNemar's test and Rule of three (statistics).