Causality in ML practice: spurious features, distribution shift and off-policy evaluation

Before this lesson: confounders, colliders and selection (lessons 01 and 03); inverse probability weighting (lesson 05); uplift models (lesson 08). Helpful: drift detection (mlops 07), bandits (reinforcement learning 03) and generalization in deep learning (deep learning 17). Time: about 80 minutes, plus the notebook.

---

Why this matters #

• Models that fail after launch. A pneumonia model trained in one hospital used hospital-specific tokens in the images and lost accuracy in other hospitals (Zech et al. 2018). Image classifiers label cows on beaches wrongly, because cows in the training data stand on grass (Beery, Van Horn and Perona 2018). These are spurious features, and they are a causal problem, not a capacity problem.
• Shift in production. Monitoring tools detect that inputs changed (mlops 07). Causal reasoning tells you whether the change matters: under covariate shift in a causal task the old $P(y \mid x)$ is still right; under label shift it is not, but it can be corrected without new labels.
• Domain generalization and robustness. Group DRO, invariant risk minimization and last-layer retraining are used to make models robust to spurious correlations, for example for minority groups in medical imaging and fairness benchmarks.
• Recommenders, ads, search and credit. Every logged dataset was produced by an old policy. Offline evaluation of a new ranking or bidding policy, and unbiased learning to rank, use the importance weights of this lesson every day.
• Decision making. Targeting by predicted churn or purchase is not the same as targeting by effect (uplift, lesson 08).

1. Predict first #

2. The picture #

2.1 A spurious feature #

Take a tabular version of the Waterbirds benchmark (Sagawa et al. 2020). The label $y$ is the bird type. A core feature $x_1$ is caused by the label: it is $+1$ or $-1$ plus noise, and it predicts the label with accuracy about $\Phi(1) = 0.84$ everywhere. A spurious feature $x_2$ (the background) is linked to the label only through how the data were collected: in training, background and bird type agree with probability $(1 + \rho_{\text{train}})/2$. A spurious feature is a feature that predicts the label in the training distribution, but not because of a stable causal mechanism; its link to the label can change when the data collection changes.

Interactive figure: loading…

The answer to question 1: far below 84%. The training data say, correctly, that the background predicts the label. More data from the same source only estimates this association more precisely. The model is not overfitting; it learns the training distribution well, and the training distribution contains a link that the deployment distribution does not.

2.2 A shift is an intervention on one mechanism #

An SCM factorises the joint distribution into mechanisms, one per variable: $P(x, y) = P(x)\,P(y \mid x)$ if $X \to Y$ (causal learning: we predict an effect from its cause), and $P(x, y) = P(y)\,P(x \mid y)$ if $Y \to X$ (anticausal learning: we predict a cause from its effects, like a diagnosis from symptoms). The principle of independent mechanisms (Schölkopf et al. 2012) says that a change in one mechanism does not change the others. A distribution shift is then an intervention on one mechanism.

Interactive figure: loading…

2.3 Invariance across environments #

If we see data from several environments (hospitals, years, countries), each with a different intervention on some variables but not on $Y$'s own mechanism, then the regression of $Y$ on its causal parents is the same in all of them. A set of features that includes a child of $Y$, or a variable linked to $Y$ only through a changing mechanism, gives regressions that differ between environments.

Interactive figure: loading…

3. Definitions and main results #

3.1 Spurious features and shortcut learning #

A learner that minimizes the expected loss on $P$ converges to the best predictor for $P$, for example $P(y \mid x_1, x_2)$ for the log loss. If $x_2$ carries information about $y$ in $P$, the best predictor uses it. Nothing in the loss says "use only stable mechanisms". So:

Fact. A spurious feature is used because it is predictive in the training distribution. More data from the same distribution, stronger models or better regularization do not remove it. Only a change of the data (other environments, balanced groups, augmentation that breaks the link) or of the objective (robust or invariant objectives) does.

Shortcut learning (Geirhos et al. 2020) is the name in deep learning: networks pick the simplest rule that works on the training set (texture instead of shape, a watermark, a hospital token), and fail on data where the shortcut does not hold.

Measuring it. Split the data into groups $g = (y, s)$ by label and spurious attribute. The worst-group accuracy is $\min_g \mathrm{acc}_g$. Average accuracy can be high while the minority groups (where the spurious attribute disagrees with the label) are badly served. Report both.

Tools that help (they need the group labels, at least on a validation set):

• Reweighting or subsampling so that every group has the same weight in the training loss.
• Group DRO (Sagawa et al. 2020): minimize the largest group loss, $\min_\theta \max_g R_g(\theta)$, with strong regularization.
• Last-layer retraining (Kirichenko, Izmailov and Wilson 2022): keep the learned features, retrain only the last linear layer on a small group-balanced set. Often enough, because the core features are usually already present in the representation.
• Augmentation that changes the spurious attribute and keeps the label (new backgrounds, swapped tokens), which is counterfactual data augmentation (lesson 09).

3.2 Distribution shift as an intervention #

Definitions. With $P_{\text{train}}(x, y) = P(x, y)$ and $P'(x, y)$ at deployment:

• Covariate shift: $P'(x) \ne P(x)$, $P'(y \mid x) = P(y \mid x)$.
• Label shift (prior shift): $P'(y) \ne P(y)$, $P'(x \mid y) = P(x \mid y)$.
• Concept shift: $P'(y \mid x) \ne P(y \mid x)$ (or $P'(x \mid y) \ne P(x \mid y)$): the mechanism itself changes.

Theorem (stability of the causal mechanism). If $X \to Y$ with no hidden common cause, and the shift is an intervention on the mechanism of $X$ only, then $P'(y \mid x) = P(y \mid x)$: covariate shift. If $Y \to X$ and the shift is an intervention on the mechanism of $Y$ only, then $P'(x \mid y) = P(x \mid y)$: label shift.

Proof. In the SCM $X := f_X(U_X)$, $Y := f_Y(X, U_Y)$ with $U_X \perp\!\!\!\perp U_Y$, the conditional $P(y \mid x)$ is the distribution of $f_Y(x, U_Y)$, which depends only on $f_Y$ and $P(U_Y)$. An intervention on the mechanism of $X$ changes $f_X$ or $P(U_X)$, not $f_Y$ or $P(U_Y)$. The anticausal case is the same with the roles exchanged. $\blacksquare$

Now question 2 of section 1: if the new hospital differs only in the distribution of the risk factor (the cause), $P(y \mid x)$ is unchanged, and a well-specified model need not be retrained. Two warnings. (1) If the model is misspecified or the new $x$ lie outside the training range, the fit there is poor; reweighting training points by $w(x) = P'(x)/P(x)$ (Shimodaira 2000) focuses the fit on the new region. (2) If a hidden variable affects both $X$ and $Y$ and its distribution changes, $P(y \mid x)$ changes too; the theorem needs the shift to touch only the mechanism of $X$.

Correcting label shift. Under label shift, Bayes' rule gives the new posterior from the old one without new labels:

Proof. $P'(y \mid x) \propto P'(x \mid y)P'(y) = P(x \mid y)P'(y) = P(x \mid y)P(y)\, w(y) \propto P(y \mid x)\,w(y)$. Normalize over $y$. $\blacksquare$

The new prevalence $P'(y)$ can be estimated from unlabeled deployment data, for example with black-box shift estimation (Lipton, Wang and Smola 2018), which inverts the confusion matrix of the classifier.

3.3 Invariance: ICP and IRM #

Invariance principle. Let $S^*$ be the set of causal parents of $Y$. If the environments come from interventions on variables other than $Y$ (and there are no hidden confounders of $Y$), then

So $P^e(y \mid x_{S^*})$ is the same in every environment.

Invariant causal prediction (ICP; Peters, Bühlmann and Meinshausen 2016) turns this into a test.

1. For every subset $S$ of the features, test the null hypothesis "$P^e(y \mid x_S)$ is the same in all environments" (in the linear case: the same coefficients and the same residual variance; for example a Chow F-test, as in the figure).
2. Output the intersection of all accepted sets, $\hat S = \bigcap_{S \text{ accepted}} S$.

Theorem (coverage). If the invariance assumption holds and each test has level $\alpha$, then $P(\hat S \subseteq S^*) \ge 1 - \alpha$: with high probability ICP outputs only causal parents.

Proof. $S^*$ satisfies the null hypothesis, so it is accepted with probability at least $1 - \alpha$. When it is accepted, the intersection of the accepted sets is a subset of $S^*$. $\blacksquare$

ICP can say "I don't know" (the empty set) when the environments are too similar. It needs many tests ($2^p$ subsets), so it is used with few features or after a preselection.

Invariant risk minimization (IRM; Arjovsky, Bottou, Gulrajani and Lopez-Paz 2019) turns invariance into a training objective for a representation $\Phi$ and a classifier $w$: find $\Phi$ such that the same $w$ is optimal in every environment. The practical version (IRMv1) is

where the penalty is small when the scalar classifier $w = 1$ is already optimal in environment $e$.

What is known about the limits.

• Rosenfeld, Ravikumar and Risteski (2021): in a linear model IRM recovers the invariant predictor only with more environments than spurious dimensions; in nonlinear models it can fail and behave like ERM.
• Gulrajani and Lopez-Paz (2020): on standard domain-generalization benchmarks, careful ERM (plain training with good model selection and augmentation) matches most specialised methods.
• Invariance gives robustness to large or new shifts; under small shifts a non-causal predictor can be more accurate (try a slider value of 0.3 in the figure: the set with $X_2$ is still better in that new environment). Choose by the shifts you expect.

3.4 Decisions need effects, and logs are biased #

Prediction vs effect. A churn model ranks customers by $P(\text{churn} \mid x)$. A retention offer should go to customers with the largest effect $\tau(x)$ of the offer on churn (lesson 08). Customers who will surely leave, or surely stay, may have $\tau(x) \approx 0$.

Selection bias in logged data. A deployed model decides which outcomes we see:

• A credit model sees repayment only for approved loans. A new model trained on approved loans learns $P(\text{repay} \mid x, \text{approved})$, not $P(\text{repay} \mid x)$. If the old approval used information that is not in $x$ (an interview, a branch manager's judgement), these differ: this is the selective labels problem (Lakkaraju et al. 2017).
• A recommender sees clicks only on items it showed, mostly items its old version liked. A model trained on these logs inherits the old policy's choices, and the feedback loop gets stronger with each retraining.

In graph terms, "observed" is a collider or a descendant of the old decision. The fixes are causal: randomize a small share of decisions (exploration), log the probability of every decision, and reweight.

3.5 Off-policy evaluation #

We have a log $\{(x_i, a_i, r_i)\}_{i=1}^n$ produced by a logging policy $\pi_0$: $x_i \sim P(x)$, $a_i \sim \pi_0(\cdot \mid x_i)$, $r_i \sim P(r \mid x_i, a_i)$. We want the value $V(\pi)$ of a target policy $\pi$ without deploying it. This is off-policy evaluation (OPE). It is the causal question "what would the mean reward be under $\mathrm{do}(a \sim \pi(\cdot \mid x))$?".

Assumptions.

1. No unmeasured confounding: $\pi_0$ chose $a$ using only the logged $x$ (true when the system logs everything its policy used).
2. Overlap: $\pi_0(a \mid x) > 0$ wherever $\pi(a \mid x) > 0$.
3. The logging probabilities $\pi_0(a_i \mid x_i)$ are recorded (or can be computed exactly).

Estimators. With the importance weight $w_i = \pi(a_i \mid x_i)/\pi_0(a_i \mid x_i)$ and a reward model $\hat q(x, a)$:

• Direct method (DM): $\hat V_{\mathrm{DM}} = \frac1n \sum_i \sum_a \pi(a \mid x_i)\,\hat q(x_i, a)$. Low variance; biased when $\hat q$ is wrong, especially for actions rarely logged.
• Inverse propensity scoring (IPS): $\hat V_{\mathrm{IPS}} = \frac1n \sum_i w_i r_i$.
• Self-normalized IPS (SNIPS): $\hat V_{\mathrm{SNIPS}} = \sum_i w_i r_i / \sum_i w_i$ (Swaminathan and Joachims 2015). Small bias, usually much smaller variance, and always inside the range of the rewards.
• Doubly robust (DR): $\hat V_{\mathrm{DR}} = \frac1n \sum_i \Big[\sum_a \pi(a \mid x_i)\,\hat q(x_i, a) + w_i\,\big(r_i - \hat q(x_i, a_i)\big)\Big]$ (Dudík, Langford and Li 2011): the DM plus an IPS correction of its errors.

Theorem (IPS is unbiased). Under assumptions 1–3, $\mathbb{E}[\hat V_{\mathrm{IPS}}] = V(\pi)$.

Proof. One term is enough, because the rows are identically distributed.

1. $\mathbb{E}[w\, r] = \mathbb{E}_x \sum_a \pi_0(a \mid x)\, \frac{\pi(a \mid x)}{\pi_0(a \mid x)}\, \mathbb{E}[r \mid x, a]$. Assumption 1 lets us use $\mathbb{E}[r \mid x, a] = q(x, a)$ for the logged actions.
2. Overlap lets us cancel $\pi_0(a \mid x)$ for every $a$ with $\pi(a \mid x) > 0$; the other actions have weight 0 in both sides.
3. So $\mathbb{E}[w\, r] = \mathbb{E}_x \sum_a \pi(a \mid x)\, q(x, a) = V(\pi)$. $\blacksquare$

This is the same computation as for IPW in lesson 05, and as importance sampling in probability 14.

Theorem (double robustness). $\hat V_{\mathrm{DR}}$ is unbiased if the weights are correct (any $\hat q$), or if $\hat q = q$ (and then the weights may be wrong).

Proof. Write the expectation of one DR term as $\mathbb{E}_x \sum_a \pi(a \mid x)\hat q(x, a) + \mathbb{E}[w\,(r - \hat q(x, a))]$. With correct weights the second part equals, as in the IPS proof, $\mathbb{E}_x \sum_a \pi(a \mid x)(q(x, a) - \hat q(x, a))$; the $\hat q$ terms cancel and $V(\pi)$ is left. With $\hat q = q$, $\mathbb{E}[r - q(x, a) \mid x, a] = 0$, so the second part is 0 for any weights, and the first part is $V(\pi)$. $\blacksquare$

Variance and overlap. For a deterministic target policy, $w_i = 1/\pi_0(a_i \mid x_i)$ on the rows where the logged action agrees with $\pi$, and 0 elsewhere. If $\pi_0(a \mid x) = 0.02$, one matching row has weight 50. The estimate then depends on a few rows. The effective sample size $\mathrm{ESS} = (\sum_i w_i)^2 / \sum_i w_i^2$ measures how many rows really count. Now question 3 of section 1: IPS stays unbiased, but its variance is huge; a few rewards multiplied by 50 decide the number. Clip weights (bias), use SNIPS or DR, or log with more exploration.

Interactive figure: loading…

Position bias and learning to rank. In search and recommendation, a click needs two things: the user examined the item and found it relevant. Users examine top positions more often, so raw clicks underestimate the relevance of items shown low. Unbiased learning to rank (Joachims, Swaminathan and Schnabel 2017) weights each click by $1/P(\text{examined} \mid \text{position})$, an IPS weight, with the examination propensities estimated from small randomized swaps of results.

3.6 A checklist for causal questions in ML projects #

1. Write the question as an estimand. "Effect of the new ranking on 7-day retention, averaged over active users" ($\tau_{\mathrm{ATE}}$), or "value of policy $\pi$" ($V(\pi)$). Not "does the feature matter?".
2. Draw the graph. Include how the data were collected: which past decisions selected the rows, and which variables the old system used.
3. Choose a design, experiment first. A randomized A/B test answers most launch questions; use it when you can (mlops 06, probability 12). Otherwise: adjustment with logged propensities, OPE, or a natural experiment (lessons 05–07).
4. Check what can be checked. Overlap and weights (ESS), balance of covariates, pre-trends, worst-group metrics, behavior in held-out environments.
5. Do a sensitivity analysis for what cannot be checked (unmeasured confounding, lesson 05).
6. Report the assumptions next to the number, and log propensities from now on, so the next analysis is easier.

4. Worked example #

(a) Off-policy evaluation by hand. Two contexts $x \in \{0, 1\}$, three actions. The logging policy is $\pi_0 = (0.25, 0.5, 0.25)$ for actions $(0, 1, 2)$ in both contexts. The target policy picks $a = 2$ when $x = 0$ and $a = 0$ when $x = 1$. The true mean rewards are $q(0, 2) = 0.7$ and $q(1, 0) = 0.6$, so (with equally likely contexts) $V(\pi) = (0.7 + 0.6)/2 = 0.65$. An old reward model gives $\hat q(0, \cdot) = (0.3, 0.5, 0.6)$ and $\hat q(1, \cdot) = (0.5, 0.4, 0.3)$. The log has 8 rows:

row	$x$	$a$	$r$	$\pi(x)$	$w$	$\hat q(x, \pi(x))$	$\hat q(x, a)$
1	0	1	1	2	0	0.6	0.5
2	0	2	1	2	4	0.6	0.6
3	0	0	0	2	0	0.6	0.3
4	0	1	0	2	0	0.6	0.5
5	1	0	1	0	4	0.5	0.5
6	1	1	0	0	0	0.5	0.4
7	1	0	0	0	4	0.5	0.5
8	1	2	0	0	0	0.5	0.3

The weight is $w = \pi(a \mid x)/\pi_0(a \mid x) = 1/0.25 = 4$ when the logged action is the one $\pi$ would take (rows 2, 5, 7), and 0 otherwise.

1. DM: the mean of the column $\hat q(x, \pi(x))$: $(4 \cdot 0.6 + 4 \cdot 0.5)/8 = 4.4/8 = 0.55$.
2. IPS: $\frac18 (4 \cdot 1 + 4 \cdot 1 + 4 \cdot 0) = 8/8 = 1.0$. Unbiased in expectation, but with only three matching rows this log gives a value above the largest possible mean reward.
3. SNIPS: $\sum w r / \sum w = 8/12 \approx 0.667$.
4. DR: the corrections $w(r - \hat q(x, a))$ are $4(1 - 0.6) = 1.6$ (row 2), $4(1 - 0.5) = 2$ (row 5) and $4(0 - 0.5) = -2$ (row 7). Their mean is $1.6/8 = 0.2$. $\hat V_{\mathrm{DR}} = 0.55 + 0.2 = 0.75$.
5. ESS: $(\sum w)^2 / \sum w^2 = 12^2/(3 \cdot 16) = 144/48 = 3$. Only three rows carry the estimate.

With 8 rows every estimate is noisy; the figure shows their behavior over many logs.

(b) Label shift. A spam filter was trained with 50% spam. In a new mailbox 20% of mail is spam, and the mail of each class looks the same as before (label shift). For a message with $P(\text{spam} \mid x) = 0.7$: $w(\text{spam}) = 0.2/0.5 = 0.4$, $w(\text{ham}) = 0.8/0.5 = 1.6$, and $P'(\text{spam} \mid x) = 0.28/(0.28 + 0.48) = 0.28/0.76 \approx 0.368$. The decision at threshold 0.5 flips from "spam" to "not spam".

5. In code #

NumPy checks the worked examples and shows a spurious feature with scikit-learn.

import numpy as np
from sklearn.linear_model import LogisticRegression

# (a) off-policy evaluation of the 8-row log
p0 = np.array([0.25, 0.5, 0.25])                       # logging policy, same in both contexts
pi = {0: 2, 1: 0}                                      # target policy (deterministic)
qhat = {0: [0.3, 0.5, 0.6], 1: [0.5, 0.4, 0.3]}        # old reward model
log = [(0, 1, 1), (0, 2, 1), (0, 0, 0), (0, 1, 0), (1, 0, 1), (1, 1, 0), (1, 0, 0), (1, 2, 0)]
x, a, r = (np.array(c) for c in zip(*log))
w = np.array([1 / p0[ai] if ai == pi[xi] else 0.0 for xi, ai in zip(x, a)])
q_pi = np.array([qhat[xi][pi[xi]] for xi in x])
q_a = np.array([qhat[xi][ai] for xi, ai in zip(x, a)])
dm, ips = q_pi.mean(), np.mean(w * r)
snips, dr = (w * r).sum() / w.sum(), np.mean(q_pi + w * (r - q_a))
print(round(dm, 3), round(ips, 3), round(snips, 3), round(dr, 3))   # 0.55 1.0 0.667 0.75
print(w.sum() ** 2 / (w ** 2).sum())                                 # 3.0  (effective sample size)

# (b) label shift correction
p, w1, w0 = 0.7, 0.2 / 0.5, 0.8 / 0.5
print(round(p * w1 / (p * w1 + (1 - p) * w0), 3))                    # 0.368

# (c) a spurious feature: train with agreement rho = 0.9, test with rho = -0.8
rng = np.random.default_rng(0)
def gen(n, rho):
    y = rng.integers(0, 2, n)
    s = np.where(rng.uniform(size=n) < (1 + rho) / 2, y, 1 - y)     # spurious attribute
    X = np.column_stack([(2 * y - 1) + rng.normal(size=n), (2 * s - 1) + 0.5 * rng.normal(size=n)])
    return X, y, s
Xtr, ytr, _ = gen(20000, 0.9)
Xte, yte, ste = gen(20000, -0.8)
both = LogisticRegression().fit(Xtr, ytr)
core = LogisticRegression().fit(Xtr[:, :1], ytr)
print(round(both.score(Xtr, ytr), 2), round(both.score(Xte, yte), 2))          # 0.95 0.46
print(round(core.score(Xtr[:, :1], ytr), 2), round(core.score(Xte[:, :1], yte), 2))  # 0.84 0.84
pred = both.predict(Xte)
print(min(np.mean(pred[(yte == g) & (ste == h)] == g) for g in (0, 1) for h in (0, 1)) < 0.4)  # True

The model with both features is better in training (0.95 vs 0.84) and much worse after the shift (0.46 vs 0.84). Its worst group on the shifted test set is below 0.4.

6. Where it shows up #

• Medical imaging. Models pick up scanner type, hospital tokens, rulers in skin-lesion photos or chest drains in X-rays. Multi-site validation, worst-group reporting and stratified evaluation by acquisition site are now expected in good studies.
• Recommendation, ads and search. Offline evaluation of new policies with IPS, SNIPS and DR on logs that record propensities; small randomized exploration traffic keeps overlap; unbiased learning to rank corrects position bias.
• Credit and hiring. Selective labels (outcomes only for approved applicants) and feedback loops; reject inference, randomized approvals of a small share, and fairness audits with causal models.
• Robustness benchmarks. WILDS, Waterbirds, CelebA and colored-MNIST measure worst-group accuracy and out-of-distribution performance; methods include group DRO, IRM, JTT and last-layer retraining.
• Monitoring. Drift detectors (mlops 07) say that inputs changed; the causal view says whether the old $P(y \mid x)$ is still right (covariate shift in a causal task) or needs a prior correction (label shift).
• ML for science. In genomics and Earth science, data come from many labs, sites or years; invariance across these environments is used to find stable predictors and causal genes (ICP was introduced with gene knockout data). In climate modeling, emulators trained on one scenario are tested on others, which is a shift of the cause (the forcing).

7. Common mistakes #

• "More data will fix the spurious feature." It will not: the association is real in the training distribution. Check with worst-group accuracy and data from another environment.
• Reporting only average accuracy. A model can be 94% accurate on average and below 40% on a minority group. Report worst-group or per-group metrics.
• Treating every shift as covariate shift. In anticausal tasks (diagnosis from images) a prevalence change is label shift; importance weighting on $x$ is then the wrong correction. Ask which mechanism changed.
• Dropping a feature and calling the model robust. Other features can carry the same spurious signal (proxies). Test on shifted environments.
• Evaluating a new policy on logs as if they were random. Training or testing on logs from the old policy without weights measures how well the new policy agrees with the old one. Log propensities and use IPS/DR.
• Not logging propensities, or estimating them badly. IPS needs the probability the old system gave to the action it took. If the system was deterministic, there is no overlap: add exploration.
• Trusting an IPS number with a tiny ESS. Look at the largest weights and the effective sample size; use SNIPS or DR, and report a confidence interval (bootstrap over rows).
• Targeting with a risk model when the decision needs an effect. Use uplift models (lesson 08) or an experiment.

8. Check yourself #

1. In the spurious-feature figure, why does the model with both features have a worst-group accuracy near 0.4 even when the test distribution equals the training distribution?
   Answer

   With $\rho = 0.9$ the background agrees with the label 95% of the time and is less noisy than the core feature. The model gives it a large weight. In the minority groups (background disagrees with the label) the large weight on $x_2$ pushes the decision to the wrong side, so most of these units are misclassified. They are only 5% of the data, so the average accuracy stays high.
2. A model predicts crop yield $Y$ from rainfall $X$ (a cause). Next year is much drier. Which part of the data distribution changes, and what is the risk for a linear model fitted last year?
   Answer

   $P(x)$ changes; $P(y \mid x)$ stays the same if nothing else changed (covariate shift). The risk is extrapolation: the dry region had few training points, so a misspecified model (linear where the true curve bends) can be wrong there. Reweighting by $P'(x)/P(x)$ or a more flexible model helps; $P(y \mid x)$ itself is still the right target.
3. Three environments: the regression of $Y$ on $\{X_1\}$ has coefficients 1.02, 0.98, 1.01; on $\{X_1, X_3\}$ the coefficients of $X_3$ are 0.2, 0.9, −0.4 and the test rejects. On $\{X_1, X_2\}$ the coefficients are stable and accepted. What does ICP output, and what does it mean?
   Answer

   Suppose the empty set and the other sets are rejected. The accepted sets are $\{X_1\}$ and $\{X_1, X_2\}$, and their intersection is $\{X_1\}$. ICP claims, with probability at least $1 - \alpha$, that $X_1$ is a causal parent of $Y$. It says nothing about $X_2$: it may be a parent or not.
4. Show that SNIPS equals IPS when $\sum_i w_i = n$, and explain why SNIPS can never be above the largest reward.
   Answer

   $\hat V_{\mathrm{SNIPS}} = \sum w_i r_i / \sum w_i = \sum w_i r_i / n = \hat V_{\mathrm{IPS}}$. SNIPS is a weighted average of the rewards with non-negative weights $w_i / \sum_j w_j$ that sum to 1, so it lies between the smallest and the largest reward. IPS divides by $n$ instead and can leave this range (1.0 in the worked example, with only three matching rows).
5. A credit model is trained on approved loans only. When can it still estimate $P(\text{repay} \mid x)$ for all applicants without bias?
   Answer

   When approval depended only on the logged features $x$ and every region of $x$ had some approvals (overlap). Then $P(\text{repay} \mid x, \text{approved}) = P(\text{repay} \mid x)$ (selection on observables). If approval used information not in $x$, or some regions were never approved, the two differ and the model is biased for the rejected population.
6. In the off-policy figure, why is DR centered on the truth although it uses the same wrong reward model as DM?
   Answer

   The logging probabilities are known exactly, so the weights are correct. Then the correction term $w(r - \hat q(x, a))$ has mean $\mathbb{E}_x \sum_a \pi(a \mid x)(q - \hat q)$, which removes the bias of $\hat q$ (double robustness). DR keeps part of DM's low variance because the residuals $r - \hat q$ are smaller than $r$.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. What is a spurious feature?

   Show answer

2. Why does more training data from the same source not remove a spurious feature?

   Show answer

3. What is worst-group accuracy?

   Show answer

4. Causal vs anticausal learning: which conditional is stable under a shift of the cause?

   Show answer

5. How do you correct a classifier's probabilities for label shift?

   Show answer

6. State the invariance principle used by ICP.

   Show answer

7. What does IRM look for?

   Show answer

8. Write the IPS estimator of a policy's value.

   Show answer

9. What is the doubly robust OPE estimator, and when is it unbiased?

   Show answer

10. What is selection bias in logged feedback?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Your model works offline but fails after launch. How could causality explain it?" — The offline test set came from the same distribution as training, so spurious features (proxies of the collection process) looked good. After launch the mechanism that created the spurious link changed (new site, new device, new policy), or the model's own decisions changed the data (feedback loop). Check per-group metrics, compare environments, ask which mechanism changed, and test on a held-out environment before launch.
• "How do you evaluate a new recommender policy without an A/B test?" — Off-policy evaluation on logs from the current policy: IPS, SNIPS or DR with the logged propensities. Assumptions: the old policy used only logged features, overlap, recorded propensities. Look at the ESS and the largest weights; if overlap is poor, the answer is an A/B test or more exploration traffic.
• "Covariate shift or label shift: how do you tell, and why does it matter?" — Ask the causal direction. For diagnosis-type tasks (label causes features) a prevalence change is label shift: re-weight the posterior by $P'(y)/P(y)$, estimating $P'(y)$ from unlabeled data (BBSE). For prognosis-type tasks it is covariate shift: $P(y \mid x)$ is fine, watch extrapolation.
• "A churn model or an uplift model for retention offers?" — Uplift: the offer should go to customers whose churn it changes most, which needs the effect, from a randomized campaign.

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution:

1. A Waterbirds-like tabular dataset: train logistic regression with and without the spurious feature, compute worst-group accuracy on a shifted test set, then reweight the groups and compare.
2. Covariate shift in a causal and in an anticausal SCM: measure the drop in accuracy, and correct the anticausal classifier for label shift with Bayes' rule.
3. A minimal invariant causal prediction: fit OLS on every candidate set in every environment, test equality with a Chow F-test, and return the intersection of the accepted sets.
4. Off-policy evaluation: implement DM, IPS, SNIPS and DR on a simulated contextual bandit, and measure bias and standard deviation as overlap gets worse.
5. Selection bias in logged data: train a credit model only on approved loans and compare it with a model trained on everybody.

11. Watch and read #

12.1 - Transfer Learning, Domain Generalization, and Covariate ShiftBrady Neal · Introduction to Causal Inference, lecture 12 · 6 min

Covariate shift and domain generalization described with causal graphs.

12.2 - Causal Insights for Transfer LearningBrady Neal · Introduction to Causal Inference, lecture 12 · 15 min

Which conditional distributions transfer between domains, and why invariance points to causal features.

Statistical and causal approaches to machine learningThe Royal Society · Bernhard Schölkopf · 60 min

Independent mechanisms, causal and anticausal learning, and why causal models generalize under shift.

Neural Networks And Spurious CorrelationsSimons Institute for the Theory of Computing · 40 min

How neural networks pick up spurious correlations, worst-group accuracy, and methods that help.

• Schölkopf et al. (2021), Toward causal representation learning: a survey of causality and machine learning.
• Schölkopf et al. (2012), On causal and anticausal learning.
• Geirhos et al. (2020), Shortcut learning in deep neural networks.
• Beery, Van Horn and Perona (2018), Recognition in terra incognita.
• Sagawa, Koh, Hashimoto and Liang (2020), Distributionally robust neural networks for group shifts.
• Kirichenko, Izmailov and Wilson (2022), Last layer re-training is sufficient for robustness to spurious correlations.
• Peters, Bühlmann and Meinshausen (2016), Causal inference using invariant prediction.
• Arjovsky, Bottou, Gulrajani and Lopez-Paz (2019), Invariant risk minimization; Rosenfeld, Ravikumar and Risteski (2021), The risks of invariant risk minimization; Gulrajani and Lopez-Paz (2020), In search of lost domain generalization.
• Lipton, Wang and Smola (2018), Detecting and correcting for label shift with black box predictors.
• Dudík, Langford and Li (2011), Doubly robust policy evaluation and learning.
• Joachims, Swaminathan and Schnabel (2017), Unbiased learning-to-rank with biased feedback.