Distribution shift and drift statistics: PSI, KS and chi-square

Before this lesson: lesson 01 (data contracts: checks on one batch; this lesson compares two batches); KL divergence; Bayes' rule and hypothesis testing (p-values, the chi-square test); calibration of a classifier. Time: about 60 minutes, plus the notebook.

---

Why this matters #

A model in production fails silently. The API still answers in 50 ms with status 200, but the data is no longer the data the model was trained on. Monitoring must find this, and it usually has no labels to do it: fraud chargebacks, flood reports and forecast verification arrive days or weeks later (lesson 08 handles delayed labels). So we watch the inputs and the predictions.

The same problem appears in science. A weather station gets a new instrument. A satellite channel is added to the data assimilation. A climate model is used in a warmer climate than its training period. In industry, PSI comes from credit scoring, where banks check every month that the population of applicants still looks like the one the score was built on. "How would you detect drift?" is one of the most common questions in ML system design interviews.

1. Predict first #

2. The picture #

One feature, two histograms. Take one input of a rain-nowcast model, for example the relative humidity at a station. Draw two histograms: the training data (the reference) and the last week of production (the current window). Monitoring compares the two, again and again, as new data arrives.

Things go wrong in four ways, from easy to hard to see:

1. The data is broken. Nulls, a unit change (percent to fraction), a renamed column. The data contract of lesson 01 catches this.
2. The data is valid but distributed differently. A dry summer. Only a comparison of two distributions shows it. This lesson is about this case.
3. The rule of the world changed. The same humidity now gives less rain. The inputs look normal; only the labels show it.
4. The model gets other inputs online than offline for the same example. This is training-serving skew, a bug in the pipeline. Lesson 08 finds it.

Two ways to factor the data. Let $x$ be the input and $y$ the label. The joint distribution can be written in two ways:

Each kind of shift changes one factor and keeps the other. Let $p$ be the training distribution and $q$ the production distribution, and let the model predict $\hat y = f(x)$.

Shift	What changes	What stays	Rain example	Seen without labels?
covariate shift	$p(x)$	$p(y \mid x)$	a heat wave: more hot days than in training	yes: input drift
label shift	$p(y)$	$p(x \mid y)$	autumn: more rainy hours, each looks as before	yes: through $p(x)$ and $p(\hat y)$
concept drift	$p(y \mid x)$	often $p(x)$	the same sky now gives less rain	no: needs labels
prediction drift	$p(\hat y)$	—	the share of "rain" alerts doubles	yes

The figure below is a toy world with one feature $x$. The model was trained on the reference data and predicts $\hat y = 1$ when $x > 0.5$. It is never retrained.

Interactive figure: loading…

3. Definitions and main results #

What each kind of shift does to a model #

• Covariate shift. The rule $p(y \mid x)$ stays, so a model that learned it well is still right where the training data was dense. Errors grow when production inputs fall where the training data was sparse, because there the model extrapolates. Accuracy can also go up: if the inputs move far from the decision boundary, most cases become easy. In the toy world, moving $p(x)$ to $N(-2, 1)$ raises the accuracy from 0.85 to 0.99.
• Label shift. By Bayes' rule the posterior changes, so a calibrated model becomes miscalibrated and the best decision threshold moves (proof below).
• Concept drift. The learned rule is now wrong, even on familiar inputs. Only retraining on recent labelled data fixes it.
• Prediction drift is a symptom, not a cause. It is still a very useful signal: one low-dimensional number that reacts to exactly the input changes the model uses.

Theorem (prediction drift depends only on the inputs). Let the model $f$ be fixed. Then the distribution of $\hat y = f(x)$ is determined by $p(x)$ alone. So pure concept drift (with $p(x)$ unchanged) causes no prediction drift.

Proof.

1. For any set $A$ of outputs, $P(\hat y \in A) = P(f(x) \in A) = \int \mathbf 1[f(x) \in A]\, p(x)\, dx$.
2. The integrand uses only $f$ and $p(x)$. The label $y$ and $p(y \mid x)$ do not appear.
3. Under pure concept drift $q(x) = p(x)$, so step 1 gives the same value under $q$ and $p$. $\blacksquare$

Label shift is different: $q(x) = \sum_y p(x \mid y)\, q(y)$ is a new mixture of the same class densities. So label shift moves $p(x)$, and input and prediction monitors can see it.

Label shift correction. Assume $q(x \mid y) = p(x \mid y)$ and the model is calibrated on the training distribution, so it outputs $p(y \mid x)$. Then

Proof.

1. Bayes' rule under $q$: $q(y \mid x) = q(x \mid y)\, q(y) / q(x)$.
2. Label shift: $q(x \mid y) = p(x \mid y)$. Bayes' rule under $p$: $p(x \mid y) = p(y \mid x)\, p(x) / p(y)$.
3. Put step 2 into step 1: $q(y \mid x) = p(y \mid x)\, \frac{q(y)}{p(y)} \cdot \frac{p(x)}{q(x)}$.
4. The last factor does not depend on $y$. Dividing by the sum over $y$ removes it. $\blacksquare$

So the fix is: multiply each predicted class probability by $w(y)$ and renormalise. Worked example C shows the numbers.

Estimating $q(y)$ without production labels (black-box shift estimation)

The correction needs $q(y)$, but production labels are late. Lipton, Wang and Smola (2018) estimate it from the predictions alone. Under label shift, $p(\hat y \mid y)$ does not change (the inputs of each class look the same, and $f$ is fixed). So

$$q(\hat y = i) = \sum_j p(\hat y = i \mid y = j)\, q(y = j).$$

The left side is measured in production. The conditional confusion matrix $p(\hat y \mid y)$ is measured on labelled validation data. Solve the linear system for $q(y)$ (it needs an invertible confusion matrix). Example with two classes: true positive rate 0.8, false positive rate 0.1, and 45% of production predictions are "rain". Then $0.45 = 0.8\,q_1 + 0.1\,(1 - q_1)$, so $q_1 = 0.35/0.7 = 0.5$: half of the hours are rainy, without a single production label.

A sensor change is not a pure covariate shift. If a new rain gauge reads 5% high, the measured input $\tilde x = 1.05\,x$ has a new distribution, and the same reading now means less real rain, so $p(y \mid \tilde x)$ changed too. Input drift monitors see it; the fix is to recalibrate the input or to retrain on data from the new sensor.

Signals, by the labels they need #

Signal	Labels?	Catches
service health: traffic, errors, p99 latency	no	outages, overload (lesson 05)
data quality: schema, nulls, ranges, volume, freshness	no	broken pipelines, unit changes
input drift: PSI, KS, chi-square per feature; a domain classifier	no	covariate shift and label shift
prediction drift: the distribution of $\hat y$	no	input changes that reach the output
performance per slice: accuracy, calibration	yes, delayed	everything, including concept drift

Per-feature statistics look at one feature at a time. They miss a change in the relation between features (each marginal the same, the joint different). A domain classifier catches it: label reference rows 0 and production rows 1, train a classifier to separate them, and measure its AUC on held-out rows. An AUC near 0.5 means no detectable shift; its feature importances point to the features that moved. Rabanser et al. (2019) compare these methods on images.

PSI: the population stability index #

Bin a feature into $k$ bins. Let $e_i$ be the reference (expected) proportion and $a_i$ the current (actual) proportion in bin $i$. The population stability index is

The usual reading is a rule of thumb from credit scoring, not a statistical test: below 0.1 stable, 0.1 to 0.25 a moderate shift, above 0.25 a large shift. The standard binning is $k = 10$ quantile bins of the reference, so each $e_i = 0.1$.

Theorem (PSI is the symmetrised KL divergence). Let all proportions be positive. Then $\mathrm{PSI} = \mathrm{KL}(a \,\|\, e) + \mathrm{KL}(e \,\|\, a)$, and $\mathrm{PSI} \ge 0$ with equality only if $a = e$.

Proof.

1. Split each term: $(a_i - e_i)\ln\frac{a_i}{e_i} = a_i \ln\frac{a_i}{e_i} - e_i \ln\frac{a_i}{e_i} = a_i \ln\frac{a_i}{e_i} + e_i \ln\frac{e_i}{a_i}$.
2. Sum over $i$. The first parts give $\mathrm{KL}(a\,\|\,e) = \sum_i a_i \ln(a_i/e_i)$; the second parts give $\mathrm{KL}(e\,\|\,a)$. This sum is also called the Jeffreys divergence.
3. In each term, $a_i - e_i$ and $\ln a_i - \ln e_i$ have the same sign, because $\ln$ is increasing. So every term is $\ge 0$, and it is $0$ only if $a_i = e_i$. $\blacksquare$

Step 3 is a direct proof; it does not even need Gibbs' inequality. Three properties explain why a PSI value depends on how you compute it.

• Empty bins. If $e_i = 0 < a_i$, the term is infinite. Implementations floor the proportions at a small $\varepsilon$ (here $10^{-4}$). The term is then about $a_i \ln(a_i/\varepsilon)$, so its value depends on the choice of $\varepsilon$. Quantile bins of the reference have no empty reference bin.
• Merging bins cannot increase PSI. By the log-sum inequality, $(a_1 + a_2)\ln\frac{a_1 + a_2}{e_1 + e_2} \le a_1 \ln\frac{a_1}{e_1} + a_2 \ln\frac{a_2}{e_2}$, and the same holds for the other KL term. (This is the data processing inequality for KL: a coarser view cannot show more difference.) Fewer bins see less of the shift.
• The noise floor. Even with no drift, $a$ and $e$ are estimated from finite samples, so they differ a little and PSI is positive. With $n$ reference values and $m$ current values,

Why the noise floor is $(k-1)(1/n + 1/m)$

1. Let $\pi_i$ be the true probability of bin $i$ (the same for both samples, since there is no drift), and write $a_i = e_i + d_i$ with a small $d_i$.
2. For small $d_i$, $\ln(a_i/e_i) = \ln(1 + d_i/e_i) \approx d_i/e_i$. So each term is about $d_i^2/e_i$, and $\mathrm{PSI} \approx \sum_i d_i^2/e_i$. (With a fixed reference, $m$ times this sum is Pearson's chi-square statistic.)
3. A proportion from $m$ values has variance $\pi_i(1 - \pi_i)/m$. The reference proportion (or, for quantile bins, the true mass of a reference bin) has variance about $\pi_i(1 - \pi_i)/n$. The two samples are independent, so $\mathbb{E}[d_i^2] \approx \pi_i(1 - \pi_i)(1/n + 1/m)$.
4. Divide by $e_i \approx \pi_i$ and sum: $\mathbb{E}[\mathrm{PSI}] \approx \sum_i (1 - \pi_i)(1/n + 1/m) = (k-1)(1/n + 1/m)$, because $\sum_i \pi_i = 1$. $\blacksquare$

A simulation confirms it: with $k = 10$, $n = 10\,000$ and $m = 500$ the formula gives $0.0189$, and the mean PSI over 2,000 windows with no drift is $0.0191$.

More bins and smaller windows raise the floor. A threshold such as 0.1 has a meaning only for a fixed binning and window size. With 50 bins and windows of 200 values the floor is $49 \times (1/10\,000 + 1/200) = 0.25$, already the "large shift" level.

So PSI is an effect size: it measures how much the distribution moved, plus a small bias that shrinks as $n$ and $m$ grow. It does not grow with the sample size.

Interactive figure: loading…

The two-sample Kolmogorov–Smirnov test #

Let $F_x$ and $F_y$ be the empirical CDFs of a reference sample of size $n$ and a current sample of size $m$: $F_x(t)$ is the fraction of reference values $\le t$. The two-sample KS statistic is the largest vertical gap between them:

No binning is needed. Under $H_0$ (both samples from the same continuous distribution), $\lambda = \sqrt{nm/(n+m)}\, D$ has, for large $n$ and $m$, the Kolmogorov distribution:

This tail probability is the asymptotic p-value. For example $\lambda = 1.41$ gives $p = 0.037$.

Interactive figure: loading…

Why the p-value goes to 0. The p-value depends on $D$ and on $\sqrt{nm/(n+m)}$. For a fixed true shift, $D$ stays about the same as the samples grow, while $\lambda$ grows like $\sqrt{n}$. So the p-value goes to 0, even for a harmless shift. Limits of KS: it works on one feature at a time; it is most sensitive near the middle of the distribution and less in the tails (the empirical CDF is noisiest where $F(1-F)$ is largest, at the median); and its asymptotic p-value is conservative (too large) for discrete data with ties.

The chi-square test for categories #

For a categorical feature (station id, weather type), build a $2 \times k$ table of counts $O_{ij}$: row 0 is the reference, row 1 the current window, one column per category from the union of both samples. The expected count under $H_0$ (same distribution) is $E_{ij} = R_i C_j / N$ (row total times column total, divided by the grand total). The statistic is

The degrees of freedom are $(2-1)(k-1) = k - 1$. A category that appears only in production causes no problem: its column total is positive, so its $E_{ij} > 0$. The approximation needs expected counts of about 5 or more, so merge rare categories into "other". For a fixed shift, $X^2$ grows linearly with the sample size (as $\lambda^2$ does for KS), so its p-value also goes to 0.

Significant is not important #

All three statistics are computed on samples. PSI and $D$ are effect sizes: they estimate how far apart the two distributions are. The p-values of KS and chi-square answer another question: is there any difference at all? With production volumes, the answer is almost always yes.

Interactive figure: loading…

Answers to section 1. (1) Yes, the p-value is tiny, but nobody should be paged. For a normal mean shift of $\delta$ standard deviations, $D \approx \varphi(0)\,\delta$, where $\varphi(0) = 1/\sqrt{2\pi} \approx 0.399$. So $D \approx 0.004$, $\lambda = \sqrt{5 \cdot 10^5} \times 0.004 \approx 2.8$ and $p \approx 2 \times 10^{-7}$. One simulated sample (section 5) gives $D = 0.0043$ and $p = 3 \times 10^{-8}$. The PSI with 10 bins is about $10^{-4}$. The test detects that the shift is real, not that it matters. (2) No. This is label shift with $w(\text{rain}) = 0.5/0.2 = 2.5$ and $w(\text{dry}) = 0.5/0.8 = 0.625$. The corrected probability is $\frac{2.5 \times 0.5}{2.5 \times 0.5 + 0.625 \times 0.5} = 0.8$ (worked example C).

4. Worked example #

A. Three statistics on one shift. A rain feature has three ordered classes: dry, light, heavy. The reference proportions are $e = (0.5, 0.3, 0.2)$; this month they are $a = (0.3, 0.3, 0.4)$.

• PSI. Term by term:

  • dry: $(0.3 - 0.5)\ln(0.3/0.5)$ $= (-0.2)(-0.5108) = 0.1022$;
  • light: $(0.3 - 0.3)\ln 1 = 0$;
  • heavy: $(0.4 - 0.2)\ln(0.4/0.2)$ $= 0.2 \times 0.6931 = 0.1386$.

  Sum: $\mathrm{PSI} = 0.2408$, a moderate shift close to "large". Check with the theorem: $\mathrm{KL}(a\|e) = 0.1240$ and $\mathrm{KL}(e\|a) = 0.1168$, and $0.1240 + 0.1168 = 0.2408$.

• KS. Cumulative proportions: reference $(0.5, 0.8, 1)$, current $(0.3, 0.6, 1)$. The gaps are $0.2, 0.2, 0$, so $D = 0.20$.

• Chi-square with $n = m = 100$. Counts: reference $(50, 30, 20)$, current $(30, 30, 40)$. Column totals $(80, 60, 60)$, row totals 100 and 100, grand total 200. So the expected counts are $(40, 30, 30)$ in each row. The six cells give $X^2 = 2 \cdot \tfrac{10^2}{40} + 0 + 2 \cdot \tfrac{10^2}{30}$ $= 5 + 6.667 = 11.67$, with $k - 1 = 2$ degrees of freedom. For 2 degrees of freedom the tail is $e^{-x/2}$, so $p = e^{-11.67/2} = 0.0029$.

• KS p-value with $n = m = 100$. $\lambda = \sqrt{100 \cdot 100/200} \times 0.2$ $= \sqrt{50} \times 0.2 = 1.41$, so $p = 0.037$. (The data has ties, so this p-value is conservative.)

• The same proportions with $n = m = 1000$. PSI and $D$ do not change. All counts are 10 times larger, so $X^2 = 116.7$ and $p \approx 5 \times 10^{-26}$. For KS, $\lambda = \sqrt{500} \times 0.2 = 4.47$ and $p \approx 8.5 \times 10^{-18}$.

B. Binning. Merge light and heavy into "wet": $e = (0.5, 0.5)$, $a = (0.3, 0.7)$. $\mathrm{PSI} = (-0.2)\ln 0.6 + 0.2 \ln 1.4$ $= 0.1022 + 0.0673 = 0.1695 < 0.2408$, as the merging property says. The merged bin hides that heavy rain doubled.

C. Label shift correction. Training data: $p(\text{rain}) = 0.2$. Now: $q(\text{rain}) = 0.5$.

1. Weights: $w(\text{rain}) = 0.5/0.2 = 2.5$, $w(\text{dry}) = 0.5/0.8 = 0.625$.
2. The calibrated model says $p(\text{rain} \mid x) = 0.5$ for some hour, so $p(\text{dry} \mid x) = 0.5$.
3. Unnormalised: rain $2.5 \times 0.5 = 1.25$, dry $0.625 \times 0.5 = 0.3125$.
4. Normalise: $q(\text{rain} \mid x) = 1.25 / (1.25 + 0.3125) = 0.8$.

An hour that looked like a coin flip in training is now 80% likely to be rainy. A threshold of 0.5 on the old output is no longer the best threshold.

5. In code #

NumPy and SciPy check the hand computations. The notebook asks you to write PSI, KS and chi-square yourself.

import numpy as np
from scipy import stats
from scipy.stats import entropy

# A. One shift, three statistics (dry, light, heavy rain).
e = np.array([0.5, 0.3, 0.2])      # reference proportions
a = np.array([0.3, 0.3, 0.4])      # current proportions
psi = np.sum((a - e) * np.log(a / e))
print(f"PSI = {psi:.4f} = KL(a||e) + KL(e||a) = {entropy(a, e):.4f} + {entropy(e, a):.4f}")
# PSI = 0.2408 = KL(a||e) + KL(e||a) = 0.1240 + 0.1168
D = np.abs(np.cumsum(a) - np.cumsum(e)).max()
print(f"D = {D:.2f}")                                   # D = 0.20
for n in (100, 1000):
    x2, p, dof, _ = stats.chi2_contingency(np.vstack([e * n, a * n]), correction=False)
    lam = np.sqrt(n * n / (n + n)) * D
    print(f"n = m = {n}: X2 = {x2:.2f} (dof {dof}), p = {p:.1e}; "
          f"KS lambda = {lam:.2f}, p = {stats.kstwobign.sf(lam):.1e}")
# n = m = 100: X2 = 11.67 (dof 2), p = 2.9e-03; KS lambda = 1.41, p = 3.7e-02
# n = m = 1000: X2 = 116.67 (dof 2), p = 4.6e-26; KS lambda = 4.47, p = 8.5e-18

# B. Merging two bins cannot increase PSI.
e2, a2 = np.array([0.5, 0.5]), np.array([0.3, 0.7])
print(f"PSI after merging = {np.sum((a2 - e2) * np.log(a2 / e2)):.4f}")   # 0.1695

# C. Label shift correction: p(rain) = 0.2 in training, q(rain) = 0.5 now.
p_y, q_y = np.array([0.8, 0.2]), np.array([0.5, 0.5])   # (dry, rain)
w = q_y / p_y
post = np.array([0.5, 0.5])                              # calibrated model output p(y | x)
print("corrected q(rain | x) =", (w * post / np.sum(w * post))[1])   # 0.8

# D. Large samples: a 0.01-sigma shift is "significant" but tiny.
rng = np.random.default_rng(0)
x = rng.standard_normal(1_000_000)
y = rng.standard_normal(1_000_000) + 0.01
r = stats.ks_2samp(x, y)
print(f"D = {r.statistic:.4f}, p = {r.pvalue:.1e}")      # D = 0.0043, p = 2.8e-08

stats.kstwobign is the Kolmogorov distribution of section 3; ks_2samp chooses an exact or an asymptotic p-value by itself. correction=False turns off Yates' continuity correction, which SciPy applies only to $2 \times 2$ tables.

6. Where it shows up #

• Weather and climate. Compare a seasonal input with the same season of earlier years, not with the annual mean, or every summer looks like drift. A statistical downscaling model used in a warmer climate than its training period works under covariate shift, with a risk of extrapolation. A new satellite instrument or an upgrade of the data assimilation system shifts input statistics overnight.
• Sensor networks and seismology. An instrument replacement changes gain or response: a sensor change, which is more than covariate shift (section 3). A station is a natural slice: monitor per station.
• Medicine. A new scanner or a new hospital changes the images: covariate shift. During an epidemic the share of positive cases rises while each case looks the same: label shift, the setting that black-box shift estimation was designed for (Lipton et al., 2018).
• Credit scoring and fraud. PSI comes from credit scoring, where it is computed every month on the score and on each input. Fraud has fast concept drift, because fraudsters adapt to the model.
• Tools. Drift libraries such as Evidently and NannyML compute PSI, KS and chi-square per feature and per window, and some offer domain classifiers. They implement the formulas of this lesson; their default bins, floors and thresholds differ, so read them before you compare numbers.

7. Common mistakes #

• Alerting on p-values with production volumes. With $10^6$ rows every shift is significant. Alert on an effect size (PSI or $D$) with thresholds calibrated on stable history (lesson 08).
• Comparing PSI values computed with different bins or window sizes. The value and its noise floor $(k-1)(1/n + 1/m)$ depend on both.
• Reading 0.1 and 0.25 as a statistical test. They are a rule of thumb for 10 bins and large samples. For small windows, the noise floor alone can reach them.
• Thinking "no prediction drift, so the model is fine". Pure concept drift does not move $p(\hat y)$; only labels show it.
• Using PSI with an $\varepsilon$ floor for a new category. The value depends on $\varepsilon$. Use chi-square, and a data-quality rule for unknown categories.
• Checking only marginals. Each feature can look stable while the joint distribution changed. Add a domain classifier or prediction drift.
• Calling every accuracy drop "drift". First check data quality and skew (lesson 08); they are cheaper to find and to fix.

8. Check yourself #

1. Reference proportions $(0.6, 0.4, 0)$, current $(0.5, 0.4, 0.1)$, floor $\varepsilon = 10^{-4}$. Compute PSI, then again with $\varepsilon = 10^{-6}$. What does this show?
   Answer

   With $10^{-4}$: $0.0182 + 0 + (0.1 - 10^{-4})\ln(0.1/10^{-4}) = 0.0182 + 0.6901 = 0.708$. With $10^{-6}$: $0.0182 + 1.1513 = 1.170$. The empty bin dominates, and its term depends on the arbitrary $\varepsilon$. A new category is better tested with chi-square (here $X^2 = 109$ with $n = m = 1000$), or with a data-quality rule ("unknown category").
2. A calibrated model was trained where $p(y=1) = 0.25$. In production $q(y=1) = 0.5$ (label shift). The model outputs $0.6$. What is the corrected probability?
   Answer

   $w(1) = 0.5/0.25 = 2$ and $w(0) = 0.5/0.75 = 2/3$. Unnormalised: $2 \times 0.6 = 1.2$ and $\frac{2}{3} \times 0.4 = 0.267$. Corrected: $1.2/1.467 = 0.818$.
3. With 10 quantile bins, a reference of 10,000 values and windows of 2,000 values, what is the PSI noise floor? And with 50 bins and windows of 200 values?
   Answer

   $9 \times (1/10\,000 + 1/2000) = 0.0054$. With 50 bins: $49 \times (1/10\,000 + 1/200) = 0.25$, already the "large shift" level. A simulation gives an even higher mean (0.34), because many bins are empty and hit the $\varepsilon$ floor, and about 80% of windows with no drift exceed 0.25.
4. Why does label shift change $p(x)$, while concept drift may not?
   Answer

   Under label shift $q(x) = \sum_y p(x \mid y)\,q(y)$: the same class densities with new weights, so the mixture $p(x)$ changes whenever the class densities differ. Concept drift changes $p(y \mid x)$, which does not enter $p(x)$ at all.
5. You monitor 30 numeric features with KS p-values and alert when $p < 0.01$. Traffic grows from $10^4$ to $10^6$ rows per window. What will you see, and what should you change?
   Answer

   Many more alerts, because $\lambda = \sqrt{nm/(n+m)}\,D$ grows by a factor of 10 for the same small shifts, so tiny harmless shifts become "significant". Alert on $D$ or PSI instead, with thresholds calibrated on stable history (for example the 99.9% quantile of past window scores).
6. Each of two features has the same distribution as in training, but their correlation changed sign. Which monitors can see this?
   Answer

   No per-feature statistic (PSI, KS, chi-square), because each marginal is unchanged. A domain classifier on both features can (its AUC rises above 0.5), and so can prediction drift if the model uses the relation between the two features.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. What does covariate shift change, and what does it keep?

   Show answer

2. What does label shift change, and what does it keep?

   Show answer

3. What is concept drift, and which monitor can see it?

   Show answer

4. Why does pure concept drift cause no prediction drift?

   Show answer

5. Write the PSI formula.

   Show answer

6. PSI equals which divergence?

   Show answer

7. What is the PSI noise floor with no drift, for $k$ bins, $n$ reference and $m$ current values?

   Show answer

8. Define the two-sample KS statistic.

   Show answer

9. Why do KS p-values go to 0 at production volumes?

   Show answer

10. How many degrees of freedom has a chi-square drift test on $k$ categories?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "How would you detect drift without labels?" — Data quality checks first; then per-feature drift (PSI or KS for numeric, chi-square for categorical), prediction drift, and a domain classifier for joint shift. Thresholds on effect sizes, calibrated on stable history.
• "PSI or KS?" — PSI is binned and symmetric, an effect size with a known noise floor; it depends on the binning. KS needs no bins, but its p-value goes to 0 with large samples, so use $D$ itself as the score. Use chi-square for categories.
• "What kinds of distribution shift are there?" — Covariate, label and concept shift from the two factorisations of $p(x, y)$, and prediction drift as a symptom. Say which ones need labels.
• "The class balance changed in production. What do you do?" — It is label shift if each class looks the same. Estimate $q(y)$ (from fresh labels or by black-box shift estimation), reweight the predicted probabilities by $q(y)/p(y)$, and move the decision threshold.

10. Practice #

The notebook for this lesson runs in Colab. Each exercise has a check cell and a folded solution:

1. Write psi(reference, current, n_bins=10, eps=1e-4) with reference-quantile bins, and check that PSI equals KL + KL with scipy.stats.entropy.
2. Write ks_statistic(x, y) with sorting and np.searchsorted, and compare it with scipy.stats.ks_2samp.
3. Write chi2_drift(ref_counts, cur_counts) from the expected counts, and compare it with scipy.stats.chi2_contingency.
4. Simulate 2,000 windows with no drift: compare the mean PSI with $(k-1)(1/n + 1/m)$ and take the 99.9% quantile as a calibrated threshold.
5. Simulate label shift: show that a calibrated logistic regression becomes miscalibrated, and fix it with the weights $w(y) = q(y)/p(y)$.

11. Watch and read #

Lecture 6, Part 2, Covariate ShiftAlex Smola · Stanford CS329P Practical Machine Learning · 27 min

What covariate shift is, why it hurts a trained model, and how to correct for it.

Data Distribution Shifts and Monitoring - "Designing ML Systems" Reading GroupML Collective · 50 min

A discussion of Chip Huyen's chapter: kinds of shift, detection with statistics, and monitoring.

Machine Learning Monitoring: What Is Concept Drift?Evidently AI · 5 min

Gradual, sudden and seasonal concept drift, and how it differs from data drift.

Kolmogorov-Smirnov Test Explained | Data Science FundamentalsNannyML · 3 min

The KS statistic as the largest gap between two empirical CDFs.

• Chip Huyen, Data distribution shifts and monitoring (2022): the sections "Types of Data Distribution Shifts" and "Detecting Data Distribution Shifts".
• Zhang, Lipton, Li and Smola, Dive into Deep Learning, Environment and Distribution Shift: covariate, label and concept shift, with the corrections.
• Lipton, Wang and Smola, Detecting and Correcting for Label Shift with Black Box Predictors (ICML 2018): black-box shift estimation.
• Rabanser, Günnemann and Lipton, Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift (NeurIPS 2019): per-feature tests, dimension reduction and domain classifiers compared.