Adjustment, matching, propensity scores and inverse probability weighting

Before this lesson: potential outcomes, the ATE and ATT, exchangeability and positivity (lesson 02); the backdoor criterion and the adjustment formula (lesson 04); linear regression (probability 13) and logistic regression. Importance sampling (probability 14) helps but is not required. Time: about 95 minutes, plus the notebook.

---

Why this matters #

Most causal questions in industry, medicine and policy are answered without an experiment: users opt in to a feature, doctors choose a drug, workers join a training program. Lesson 04 told us which variables to adjust for. This lesson is about how to adjust with data. The estimators here are the standard tools of epidemiology (Hernán and Robins), economics (Imbens and Rubin) and tech companies (observational studies next to A/B tests). They are also the base of causal machine learning: inverse propensity scoring (IPS) in recommender systems and learning to rank, off-policy evaluation in reinforcement learning (lesson 11), and double machine learning (lesson 08), which is AIPW with ML models.

1. Predict first #

2. The picture #

2.1 The problem: the groups differ before treatment #

A job-training program is open to everyone. People with low past earnings join more often. Past earnings also predict future earnings. So the people who joined would earn less even without the program, and the naive difference in mean earnings mixes the effect of the program with this difference between the groups. This is confounding by $X$ = past earnings (lesson 01 and lesson 04).

If $X$ blocks every backdoor path from $T$ to $Y$, the fix is to compare like with like: treated and control units with the same $X$. There are three ways to do this, and this lesson treats all three:

Idea	Method	What it models
predict the missing potential outcome	outcome regression (the g-formula)	$\mu_t(x) = \mathbb{E}[Y \mid T = t, X = x]$
find a similar unit of the other group	matching	a distance between units
reweight the units so that the groups look alike	inverse probability weighting	$e(x) = P(T = 1 \mid X = x)$

2.2 Weighting creates a pseudo-population #

Start with a population of 100 people and one binary confounder $X$. Half have $X = 1$. In the group $X = 1$, 80% are treated: $e(1) = 0.8$. In the group $X = 0$, 20% are treated: $e(0) = 0.2$. The outcome is $Y = T + 2X + \varepsilon$, so the true effect is $1$ for everyone.

The treated group is mostly $X = 1$ (40 of 50 people), and $X = 1$ adds 2 to $Y$. So the naive difference is 2.2, not 1:

Now give each treated unit the weight $1/e(x)$ and each control unit the weight $1/(1 - e(x))$:

Group	$1/e$	$1/(1 - e)$	Treated: count → weighted	Control: count → weighted
$X{=}1$	1.25	5	40 → 50	10 → 50
$X{=}0$	5	1.25	10 → 50	40 → 50

After weighting, each group stands for the whole population: 50 people with $X = 1$ and 50 with $X = 0$. A treated person in the group $X = 0$ is rare (only 10 of 50), so this person counts 5 times. Together, the 10 treated people of that group now stand for all 50 people of the group, including the 40 who were not treated. Inside each group of $X$ the treatment was "as good as random", so these stand-ins are fair. In the pseudo-population $X$ no longer predicts $T$, and the weighted difference in means is $(1 + 2 \cdot 0.5) - (2 \cdot 0.5) = 1$: the true effect.

Interactive figure: loading…

2.3 Matching: find a twin in the other group #

Matching answers the causal question for each treated unit directly. For a treated unit $i$ we do not observe $Y_i(0)$. We take the control unit $j(i)$ with the closest covariates and use its outcome $Y_{j(i)}$ as a stand-in for $Y_i(0)$. The average of $Y_i - Y_{j(i)}$ over the treated units estimates the ATT, the effect for the treated. If the match were perfect ($X_{j(i)} = X_i$), exchangeability given $X$ would make $Y_{j(i)}$ a fair stand-in for $Y_i(0)$.

With one or two covariates good matches are easy to find. With many covariates they are not: in high dimension every unit is far from every other unit (the curse of dimensionality). The propensity score solves this problem: section 3.4 shows that it is enough to match on the one number $e(X)$.

Interactive figure: loading…

3. Definitions and main results #

3.1 Assumptions and identification #

We observe $n$ independent copies of $(X, T, Y)$, where $X$ is a vector of pre-treatment covariates: measured before $T$ and not affected by $T$. We write $\mu_t(x) = \mathbb{E}[Y \mid T = t, X = x]$ for the outcome regressions and $e(x) = P(T = 1 \mid X = x)$ for the propensity score. Three assumptions (lesson 02):

1. Consistency. If $T_i = t$, then $Y_i = Y_i(t)$. (This includes "no interference": the outcome of unit $i$ does not depend on the treatment of other units.)
2. Conditional exchangeability (also called no unmeasured confounding, ignorability or unconfoundedness): $(Y(0), Y(1)) \perp\!\!\!\perp T \mid X$. Within each level of $X$, the treatment is as good as random. In graph language: $X$ satisfies the backdoor criterion of lesson 04.
3. Positivity (also called overlap): $0 < e(x) < 1$ for every $x$ with $p(x) > 0$. Every kind of unit has some chance of being treated and some chance of being a control.

Which ones can the data check? Positivity can be checked, at least roughly: look at the estimated $\hat e(x)$ (section 3.6). Conditional exchangeability cannot be checked: it is a statement about $Y(0)$ of the treated units, and we never see it. It comes from knowledge of the problem (a causal graph), not from the data. Section 3.9 asks what happens if it fails.

Theorem (g-formula, or standardization). Under the three assumptions,

Proof (for $t = 1$; the case $t = 0$ is the same).

1. By the law of total expectation, $\mathbb{E}[Y(1)] = \mathbb{E}\big[\mathbb{E}[Y(1) \mid X]\big]$.
2. By conditional exchangeability, $Y(1)$ is independent of $T$ given $X$, so $\mathbb{E}[Y(1) \mid X] = \mathbb{E}[Y(1) \mid T = 1, X]$. Positivity ($e(X) > 0$) makes sure that this conditional expectation is defined for every $X$.
3. By consistency, $Y = Y(1)$ when $T = 1$, so $\mathbb{E}[Y(1) \mid T = 1, X] = \mathbb{E}[Y \mid T = 1, X] = \mu_1(X)$. $\blacksquare$

The right-hand side contains only observable quantities. So the ATE is identified: it is a property of the distribution of $(X, T, Y)$. This is the adjustment formula of lesson 04, written with expectations. For the treated, the same steps give $\tau_{\mathrm{ATT}} = \mathbb{E}[Y \mid T = 1] - \mathbb{E}[\mu_0(X) \mid T = 1]$. The ATT needs less: exchangeability only for $Y(0)$, and positivity only in the form $e(x) < 1$.

Every method below estimates the same estimand $\mathbb{E}[\mu_1(X) - \mu_0(X)]$ (or its ATT version). They differ in what they model and in how they fail.

3.2 Outcome regression (standardization) #

Estimator. Fit $\hat\mu_1$ on the treated units and $\hat\mu_0$ on the controls (any regression or ML model). Then predict both potential outcomes for every unit and average:

For the ATT, average only over the treated units. This is the plug-in version of the g-formula.

The usual regression is a special case. Fit one linear model $Y = \alpha + \tau T + \beta^\top X + \varepsilon$ and report $\hat\tau$. This equals $\hat\tau_{\mathrm{reg}}$ when the model is right: $\mu_1$ and $\mu_0$ are linear and parallel (a constant effect). When the effect differs between units, the coefficient $\hat\tau$ is not the ATE in general. For a discrete $X$ with one dummy per level, it is a weighted average of the effects $\tau(x)$ with weights proportional to $P(X = x)\, e(x)(1 - e(x))$ (Angrist 1998). Example: two equal groups with $e = 0.5, 0.9$ and effects $\tau = 4, 2$. The ATE is $3$ and the ATT is $2.71$, but the weights are $0.25$ and $0.09$, so the coefficient is $(0.25 \cdot 4 + 0.09 \cdot 2)/0.34 = 3.47$: neither the ATE nor the ATT.

Weak point: extrapolation. Where there are almost no controls (say, at large $x$ in the figure of section 2.2), $\hat\mu_0$ is a guess made from other regions. A linear model extrapolates without a warning. Regression hides poor overlap; matching and weighting show it.

3.3 Matching #

• Exact matching. Pair units with exactly the same $X$. Possible only with few discrete covariates. If each treated unit is compared with the mean of all controls with the same $X$, the matching estimate equals the plug-in g-formula for the ATT.
• Nearest-neighbor matching (with replacement). For each treated unit $i$, find the control $j(i)$ that minimizes a distance $d(X_i, X_j)$: Euclidean on standardized covariates, Mahalanobis, or $|\operatorname{logit} \hat e(X_i) - \operatorname{logit} \hat e(X_j)|$. Then

• $k$ matches. Average the outcomes of the $k$ nearest controls: lower variance, larger bias.
• Caliper. Drop a treated unit when its best match is farther than a caliper $c$. A common choice is $c = 0.2$ standard deviations of $\operatorname{logit} \hat e$ (Austin 2011). The dropped units change the estimand: we now estimate the effect for the treated units that have a match.

Matching estimates the ATT naturally. Its strength is transparency: you can look at the pairs. Its weakness is bias from imperfect matches. If $X_{j(i)} \ne X_i$, the stand-in has the wrong expected outcome: $\mathbb{E}[Y_{j(i)}] = \mu_0(X_{j(i)}) \ne \mu_0(X_i)$.

Matching bias and how to remove it (Abadie and Imbens)

With $k$ continuous covariates, the distance to the nearest control shrinks like $n^{-1/k}$, so the bias of nearest-neighbor matching also shrinks like $n^{-1/k}$ (Abadie and Imbens 2006). The standard error shrinks like $n^{-1/2}$. For $k \ge 3$ the bias shrinks more slowly than the noise, so in large samples it dominates and confidence intervals become wrong. The bias-corrected matching estimator (Abadie and Imbens 2011) fixes this. It fits a regression $\hat\mu_0$ on the controls and replaces $Y_{j(i)}$ by $Y_{j(i)} + \hat\mu_0(X_i) - \hat\mu_0(X_{j(i)})$: the regression moves the stand-in from $X_{j(i)}$ to $X_i$. A second warning: the standard bootstrap is not valid for nearest-neighbor matching with a fixed number of matches (Abadie and Imbens 2008); use their analytic variance formula or a weighting estimator instead.

3.4 The propensity score #

Definition. The propensity score is $e(x) = P(T = 1 \mid X = x)$ (Rosenbaum and Rubin 1983). It compresses a whole vector $X$ into one number between 0 and 1.

Theorem (balancing property). (a) $T \perp\!\!\!\perp X \mid e(X)$: units with the same propensity score have the same distribution of $X$ in the treated and in the control group. (b) If $(Y(0), Y(1)) \perp\!\!\!\perp T \mid X$, then $(Y(0), Y(1)) \perp\!\!\!\perp T \mid e(X)$.

So conditioning on the single number $e(X)$ is as good as conditioning on all of $X$. We can match, stratify or weight on $e(X)$ instead of $X$.

Proof of the balancing theorem

For a binary $T$, independence of $T$ and $A$ given $B$ means $P(T = 1 \mid A, B) = P(T = 1 \mid B)$.

(a)

1. $e(X)$ is a function of $X$, so $P(T = 1 \mid X, e(X)) = P(T = 1 \mid X) = e(X)$.
2. By the tower rule, $P(T = 1 \mid e(X)) = \mathbb{E}\big[P(T = 1 \mid X) \mid e(X)\big] = \mathbb{E}[e(X) \mid e(X)] = e(X)$.
3. Steps 1 and 2 give the same value, so $P(T = 1 \mid X, e(X)) = P(T = 1 \mid e(X))$: $T \perp\!\!\!\perp X \mid e(X)$.

(b) Write $Y(\cdot) = (Y(0), Y(1))$.

1. By the tower rule, $P(T = 1 \mid Y(\cdot), e(X)) = \mathbb{E}\big[P(T = 1 \mid Y(\cdot), X) \mid Y(\cdot), e(X)\big]$.
2. By exchangeability given $X$, $P(T = 1 \mid Y(\cdot), X) = P(T = 1 \mid X) = e(X)$.
3. So $P(T = 1 \mid Y(\cdot), e(X)) = \mathbb{E}[e(X) \mid Y(\cdot), e(X)] = e(X) = P(T = 1 \mid e(X))$ (the last step is step 2 of part (a)). $\blacksquare$

Estimating $e(x)$. In practice $e$ is unknown. Fit a probabilistic classifier of $T$ on $X$: logistic regression, or a boosted model with calibrated probabilities. The probabilities must be calibrated, because IPW divides by them.

Judge the propensity model by balance, not by accuracy. The goal is not to predict $T$ well. The goal is that, after matching or weighting on $\hat e$, the covariates are balanced (section 3.7). A very accurate model is bad news: if $X$ predicts $T$ almost perfectly, then the treated and control units live in different regions of $X$, and there is little overlap to compare. Also, do not add variables only because they predict $T$: a variable that affects $T$ but not $Y$ (an instrument, lesson 04) adds variance and can amplify bias.

3.5 Inverse probability weighting #

The Horvitz–Thompson IPW estimator of the ATE is

Theorem. Under the three assumptions, with the true $e$: $\mathbb{E}\big[\tfrac{TY}{e(X)}\big] = \mathbb{E}[Y(1)]$ and $\mathbb{E}\big[\tfrac{(1 - T)Y}{1 - e(X)}\big] = \mathbb{E}[Y(0)]$. So $\hat\tau_{\mathrm{IPW}}$ with the true $e$ is unbiased for $\tau_{\mathrm{ATE}}$.

Proof (treated part).

1. By consistency, $TY = T\,Y(1)$ (when $T = 0$ both sides are 0).
2. Condition on $X$: $\mathbb{E}\big[\tfrac{T\,Y(1)}{e(X)}\big] = \mathbb{E}\Big[\tfrac{\mathbb{E}[T\,Y(1) \mid X]}{e(X)}\Big]$. Positivity makes the division safe.
3. By conditional exchangeability, $T$ and $Y(1)$ are independent given $X$. So $\mathbb{E}[T\,Y(1) \mid X] = \mathbb{E}[T \mid X]\,\mathbb{E}[Y(1) \mid X]$, and this is $e(X)\,\mathbb{E}[Y(1) \mid X]$.
4. The $e(X)$ cancels: $\mathbb{E}\big[\mathbb{E}[Y(1) \mid X]\big] = \mathbb{E}[Y(1)]$. $\blacksquare$

IPW is importance sampling. We want averages over the whole population, but the treated units are a sample from $p(x \mid T = 1)$, not from $p(x)$. By Bayes' rule $p(x)/p(x \mid T = 1) = P(T = 1)/e(x)$: the weight $1/e(x)$ is the importance weight, up to a constant (probability 14).

The normalized (Hájek) estimator. The weights $T_i/\hat e_i$ sum to $n$ only on average. Dividing by their actual sum gives a weighted difference in means:

It has a small bias but usually a much smaller variance, and it does not change when you add a constant to $Y$ (the Horvitz–Thompson form does). Use it by default.

Weights for the ATT. Treated units get weight 1. Control units get $\hat e(x)/(1 - \hat e(x))$, the odds of treatment. This reweights the controls to look like the treated group, so the weighted control mean estimates $\mathbb{E}[Y(0) \mid T = 1]$. (Check: $p(x \mid T = 1)/p(x \mid T = 0) = \frac{e(x)}{1 - e(x)} \cdot \frac{P(T = 0)}{P(T = 1)}$.)

Estimated or true $e$? The proof uses the true $e$. In practice we plug in $\hat e$. A surprising fact: with a good estimate (a correct parametric model, or a flexible model that grows with $n$), IPW with $\hat e$ has a variance at least as small as IPW with the true $e$, and with a flexible model it reaches the smallest possible variance (Hirano, Imbens and Ridder 2003). The estimated weights also correct the chance imbalance of $X$ in the sample.

3.6 Overlap and extreme weights #

If $e(x)$ is close to 0 or 1 for some units, their weights are huge, and a few units decide the estimate. The variance of IPW grows without bound as $e \to 0$ or $1$. Two numbers make this visible:

• the largest weight, and
• the effective sample size (Kish) of a group with weights $w_i$: $\mathrm{ESS} = \big(\sum_i w_i\big)^2 / \sum_i w_i^2$. Equal weights give $\mathrm{ESS} = n$; one dominant weight gives about 1. Example: weights $(1, 1, 1, 1, 6)$ give $10^2/40 = 2.5$, not 5.

Remedies, and their price.

• Trimming: keep only units with $\alpha \le \hat e \le 1 - \alpha$ (often $\alpha = 0.1$, Crump et al. 2009). The variance falls, but the estimand changes: it is now the effect for the units with good overlap, not the ATE. Say so when you report it.
• Truncating (or clipping) weights at a maximum value: keeps all units, adds bias.
• Overlap weights (Li, Morgan and Zaslavsky 2018): treated units get $1 - \hat e$ and controls get $\hat e$. They target the effect in the population with the most overlap, $\mathbb{E}[e(X)(1 - e(X))\tau(X)]/\mathbb{E}[e(X)(1 - e(X))]$, and never explode.

Poor overlap is a property of the data, not of the estimator. Outcome regression hides it by extrapolating, and AIPW (section 3.8) also divides by $\hat e$: it reduces the damage but does not remove it. When overlap is poor, the honest answer is often an effect for a smaller population.

Interactive figure: loading…

3.7 Diagnostics: balance and overlap #

Before you report an effect, check the two things the data can check.

Overlap. Plot the histograms of $\hat e$ for the treated and the control group (as in the figure above). Look for regions where one group has no units. Report the largest weight and the ESS.

Balance. For each covariate, compute the standardized mean difference

where $\bar x_t$ and $s_t^2$ are the mean and variance in group $t$. For a binary covariate with shares $p_1, p_0$, use $s_t^2 = p_t(1 - p_t)$. After weighting or matching, recompute the means with the weights (or in the matched sample) and keep the denominator from the raw data. A common rule is $|\mathrm{SMD}| < 0.1$ for every covariate (Austin 2011). Unlike a $t$-test, the SMD does not depend on the sample size. Papers show the means before and after adjustment in a "Table 1" and the SMDs in a "Love plot".

What diagnostics cannot do. Good balance on the measured $X$ says nothing about an unmeasured confounder $U$. Balance checks the estimation step (did the weights or matches do their job?), not the identification step (is $X$ enough?).

3.8 Doubly robust estimation (AIPW) #

Regression needs a correct outcome model. IPW needs a correct propensity model. The augmented inverse probability weighting (AIPW) estimator (Robins, Rotnitzky and Zhao 1994) uses both:

Read it as "regression estimate + a weighted average of the regression's mistakes". If the outcome model is right, the residuals average to zero and the correction does nothing on average. If the outcome model is wrong but the weights are right, the weighted residuals measure the mistake and remove it.

Theorem (double robustness). Let $\tilde\mu_t$ and $\tilde e$ be the limits of the fitted models. Under the three assumptions, AIPW is consistent for the ATE if either $\tilde\mu_0 = \mu_0$ and $\tilde\mu_1 = \mu_1$, or $\tilde e = e$. It does not need both.

Proof of double robustness

Take the treated part, $\psi_1 = \tilde\mu_1(X) + T\,(Y - \tilde\mu_1(X))/\tilde e(X)$; the control part is the same with $1 - T$ and $1 - \tilde e$.

1. By consistency, $T\,Y = T\,Y(1)$, so $\psi_1 = \tilde\mu_1(X) + T\,(Y(1) - \tilde\mu_1(X))/\tilde e(X)$.
2. Condition on $X$. By conditional exchangeability, $\mathbb{E}[T\,(Y(1) - \tilde\mu_1(X)) \mid X] = e(X)\,(\mu_1(X) - \tilde\mu_1(X))$, using $\mathbb{E}[Y(1) \mid X] = \mu_1(X)$ (section 3.1).
3. So $\mathbb{E}[\psi_1] = \mathbb{E}\Big[\tilde\mu_1(X) + \frac{e(X)}{\tilde e(X)}\big(\mu_1(X) - \tilde\mu_1(X)\big)\Big]$.
4. Subtract $\mathbb{E}[Y(1)] = \mathbb{E}[\mu_1(X)]$:

$$\mathbb{E}[\psi_1] - \mathbb{E}[Y(1)] = \mathbb{E}\left[\big(\mu_1(X) - \tilde\mu_1(X)\big)\,\frac{e(X) - \tilde e(X)}{\tilde e(X)}\right].$$

5. The bias is an average of a product of two errors. It is zero if $\tilde\mu_1 = \mu_1$ (first factor zero) or if $\tilde e = e$ (second factor zero). The sample average converges to this expectation by the law of large numbers. $\blacksquare$

Two more facts. When both models are correct, AIPW has the smallest possible asymptotic variance among regular estimators that use only these assumptions (it reaches the semiparametric efficiency bound). And the product form of the bias in step 4 is the key to lesson 08: with flexible ML models and cross-fitting (fit the models on one part of the data, evaluate the formula on the other part), two slowly converging models still give a fast-converging estimate. This is double machine learning.

Interactive figure: loading…

3.9 Sensitivity analysis: what if a confounder is missing? #

Every method above assumes conditional exchangeability. A sensitivity analysis asks: how strong would an unmeasured confounder $U$ have to be to change the conclusion? A result is credible only together with this statement.

Omitted-variable bias (linear case). Suppose the true model is $Y = \tau T + \beta^\top X + \gamma U + \varepsilon$, but we regress $Y$ on $T$ and $X$ only. Then the estimate shifts by

where $\hat\gamma$ is the coefficient of $U$ in the full regression and $\hat\delta$ is the coefficient of $T$ in the regression of $U$ on $T$ and $X$ (this is an exact identity of least squares). The bias is large only if $U$ is related to both the outcome ($\gamma$) and the treatment ($\delta$).

The same idea with partial $R^2$ (Cinelli and Hazlett 2020). Measure the strength of $U$ by two numbers between 0 and 1: $R^2_{T \sim U \mid X}$, the share of the variance of $T$ (after $X$) that $U$ explains, and $R^2_{Y \sim U \mid T, X}$, the same for $Y$. Then

where $\mathrm{df}$ is the residual degrees of freedom of the regression. The robustness value $\mathrm{RV}$ is the common value of the two partial $R^2$ that would bring the estimate to 0: $\mathrm{RV} = \tfrac12\big(\sqrt{f^4 + 4f^2} - f^2\big)$ with $f = |\hat\tau|/(\mathrm{se}\sqrt{\mathrm{df}})$. To judge whether such a $U$ is plausible, compare it with the observed covariates: "a confounder as strong as past earnings".

The E-value (VanderWeele and Ding 2017) does the same on the risk-ratio scale, for a binary outcome. A confounder with risk ratio $\mathrm{RR}_{TU}$ with the treatment and $\mathrm{RR}_{UY}$ with the outcome can change an observed risk ratio by at most the bias factor (Ding and VanderWeele 2016)

The E-value is the smallest strength, equal on both sides, that could fully explain away an observed $\mathrm{RR} > 1$:

For $\mathrm{RR} = 2$, $E = 2 + \sqrt 2 = 3.41$: only a confounder that raises both the chance of treatment and the risk of the outcome by a factor of 3.41 could explain the result away. For a protective effect ($\mathrm{RR} < 1$), use $1/\mathrm{RR}$. Report the E-value of the confidence limit closest to 1, too.

Rosenbaum's $\Gamma$ (for matched pairs): assume that two units with the same $X$ can differ in their odds of treatment by at most a factor $\Gamma$ because of $U$. Report the largest $\Gamma$ at which the result is still significant. $\Gamma = 1$ is "no hidden bias".

Interactive figure: loading…

Answers to section 1. (1) Smaller. A treated person in that group gets $1/0.8 = 1.25$ and an untreated person gets $1/0.2 = 5$: the untreated person's weight is 4 times larger, because untreated people are rare there. (2) Bad news. Near-perfect separation means poor overlap: extreme weights, poor matches, and an estimate that depends on extrapolation. (3) Yes. AIPW is doubly robust: a correct propensity model is enough for consistency.

4. Worked example #

(a) Twelve trainees, by hand #

Twelve people could join a training program. $X = 1$ means "unemployed last year", $T = 1$ means "joined", and $Y$ is earnings next year in thousand dollars. Unemployed people join more often (4 of 6) than employed people (2 of 6).

$X$	$T$	$Y$		$X$	$T$	$Y$
1	1	6, 7, 8, 7		0	1	14, 16
1	0	4, 6		0	0	10, 12, 11, 11

Step 1: cell means. $\hat\mu_1(1) = 28/4 = 7$, $\hat\mu_0(1) = 10/2 = 5$, $\hat\mu_1(0) = 30/2 = 15$, $\hat\mu_0(0) = 44/4 = 11$. The effect is $7 - 5 = 2$ for the unemployed and $15 - 11 = 4$ for the employed.

Step 2: naive difference. Treated mean $= (28 + 30)/6 = 9.67$; control mean $= (10 + 44)/6 = 9.00$; naive $= 0.67$. It is far too small: the treated are mostly unemployed, who earn less anyway.

Step 3: g-formula. $P(X = 1) = P(X = 0) = 6/12$, so $\hat\tau_{\mathrm{ATE}} = \tfrac12 \cdot 2 + \tfrac12 \cdot 4 = 3$. For the ATT, average over the treated: 4 of them have $X = 1$ and 2 have $X = 0$, so $\hat\tau_{\mathrm{ATT}} = (4 \cdot 2 + 2 \cdot 4)/6 = 2.67$.

Step 4: exact matching (ATT). Match each treated unit with the mean of the controls in its cell (5 or 11). This gives the same 2.67 as step 3:

Step 5: propensity scores and IPW weights. $\hat e(1) = 4/6 = 2/3$ and $\hat e(0) = 2/6 = 1/3$.

Cell	$1/\hat e$ (treated)	$1/(1 - \hat e)$ (control)	Weighted treated	Weighted control
$X{=}1$	1.5	3	$4 \cdot 1.5 = 6$	$2 \cdot 3 = 6$
$X{=}0$	3	1.5	$2 \cdot 3 = 6$	$4 \cdot 1.5 = 6$

Both weighted groups now have 6 people of each kind, like the whole sample. Weighted means: treated $(1.5 \cdot 28 + 3 \cdot 30)/12 = 132/12 = 11$, control $(3 \cdot 10 + 1.5 \cdot 44)/12 = 96/12 = 8$. So $\hat\tau_{\mathrm{IPW}} = 11 - 8 = 3$. (Here the weights of each group sum to exactly $n = 12$, so the Horvitz–Thompson and Hájek forms agree.)

Step 6: ATT weights. Controls get $\hat e/(1 - \hat e)$: $2$ in the cell $X = 1$, $0.5$ in the cell $X = 0$. The weighted control mean is $(2 \cdot 10 + 0.5 \cdot 44)/6 = 42/6 = 7$ (the weights sum to $2 \cdot 2 + 0.5 \cdot 4 = 6$). So $\hat\tau_{\mathrm{ATT}} = 9.67 - 7 = 2.67$.

Step 7: balance. The share of $X = 1$ is $p_1 = 0.667$ among the treated and $p_0 = 0.333$ among the controls. Both variances are $p(1 - p) = 0.222$, so $\mathrm{SMD} = 0.333/\sqrt{0.222} = 0.71$: strong imbalance. After IPW, both weighted shares are $6/12 = 0.5$, so the SMD is $0$.

Step 8: effective sample size of the weighted treated group. Weights $(1.5, 1.5, 1.5, 1.5, 3, 3)$: $\mathrm{ESS} = 12^2/(4 \cdot 2.25 + 2 \cdot 9) = 144/27 = 5.33$ out of 6.

Step 9: AIPW. With one mean per cell, the residuals $Y_i - \hat\mu_{T_i}(X_i)$ add up to 0 in each cell, so the correction term is 0 and AIPW gives $3$ as well. With a saturated discrete $X$, all the estimators agree; they differ when $X$ is continuous or high-dimensional and the models must smooth.

(b) A simulated job-training study #

In 1986 LaLonde compared observational estimates of a job-training program with the result of a randomized experiment, and many of them were far off; Dehejia and Wahba (1999) later showed that propensity-score methods on the right covariates came close. The study below is simulated in that spirit, so we know the truth. There are $n = 2000$ people with four covariates (age, years of education, earnings in 1975, unemployed in 1975). People with low past earnings join more often. The program raises earnings by 1 (thousand dollars) for people with a job in 1975 and by 2.5 for the unemployed, so the true ATE is about 1.45 and the true ATT, about 1.76, is larger (the unemployed join more often). The code is in section 5.

Method	Target	Seed 0	Mean of 200	SD of 200
naive difference	none	$-3.12$	$-2.49$	$0.24$
OLS coefficient of $T$	ATE*	$1.32$	$1.42$	$0.16$
g-formula	ATE	$1.30$	$1.45$	$0.16$
NN matching	ATT	$1.62$	$1.67$	$0.22$
IPW (Hájek)	ATE	$0.99$	$1.43$	$0.32$
IPW, ATT weights	ATT	$1.72$	$1.73$	$0.21$
AIPW	ATE	$1.28$	$1.45$	$0.17$

Truth: ATE $= 1.45$ (in seed 0 and on average); ATT $= 1.76$ in seed 0 and $1.74$ on average. "Seed 0" is one study; "mean" and "SD" are over 200 simulated studies. *The OLS coefficient is the ATE only if the effect were the same for everyone. The g-formula uses linear $\hat\mu_t$ in each arm.

What to read from the table:

• The naive difference has the wrong sign. The participants are the people who would earn less anyway.
• All adjusted methods are close to their targets on average, but one study can be off by one or two standard deviations (IPW gives $0.99$ in this study). Always report a standard error, for example by the bootstrap (probability 12).
• IPW for the ATE has the largest spread: some controls with high past earnings are very unlikely to join ($\hat e$ as low as 0.007), and the few treated units that look like them get huge weights. The ATT weights do not have this problem: they give such controls a small weight $\hat e/(1 - \hat e)$.
• Matching is a little biased (1.67 vs 1.74): with four covariates the matches are not exact.
• AIPW is centered on the truth with about the spread of regression.

5. In code #

The code checks the hand computation of part (a) and runs the simulated study of part (b).

import numpy as np
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.neighbors import NearestNeighbors

# (a) the twelve trainees of section 4: X = unemployed last year, T = joined, Y = earnings
X = np.array([1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
T = np.array([1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0])
Y = np.array([6, 7, 8, 7, 4, 6, 14, 16, 10, 12, 11, 11.])
e = np.array([T[X == x].mean() for x in X])          # ê(x) = share treated in the cell
mu = {(t, x): Y[(T == t) & (X == x)].mean() for t in (0, 1) for x in (0, 1)}
m1, m0 = np.array([mu[1, x] for x in X]), np.array([mu[0, x] for x in X])
print("naive", round(Y[T == 1].mean() - Y[T == 0].mean(), 2))               # 0.67
print("g-formula", np.mean(m1 - m0))                                         # 3.0
print("IPW (HT)", np.mean(T * Y / e - (1 - T) * Y / (1 - e)))                # 3.0
w = (1 - T) * e / (1 - e)                                                    # ATT weights
print("IPW (ATT)", round(Y[T == 1].mean() - (w @ Y) / w.sum(), 2))           # 2.67
print("AIPW", np.mean(m1 - m0 + T * (Y - m1) / e - (1 - T) * (Y - m0) / (1 - e)))  # 3.0

# (b) the simulated job-training study (earnings in 1000 $)
def simulate(n=2000, seed=0):
    rng = np.random.default_rng(seed)
    age, educ = rng.uniform(18, 55, n), rng.integers(8, 17, n).astype(float)
    unemp = (rng.random(n) < 0.3).astype(float)
    re75 = np.where(unemp == 1, 0.0, rng.gamma(2.0, 4.0, n))          # earnings in 1975
    logit = -0.8 - 0.12 * (re75 - 8) + unemp - 0.03 * (age - 35) - 0.15 * (educ - 12)
    T = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    tau = 1.0 + 1.5 * unemp                                            # true effects
    Y0 = (3 + 0.8 * re75 + 0.4 * (educ - 12) + 0.1 * (age - 35)
          - 0.004 * (age - 35) ** 2 + rng.normal(0, 3, n))
    return np.column_stack([age, educ, re75, unemp]), T, Y0 + tau * T, tau

X, T, Y, tau = simulate()
Z = (X - X.mean(0)) / X.std(0)
e = LogisticRegression(C=1e6).fit(Z, T).predict_proba(Z)[:, 1]
m1 = LinearRegression().fit(X[T == 1], Y[T == 1]).predict(X)
m0 = LinearRegression().fit(X[T == 0], Y[T == 0]).predict(X)
nn = NearestNeighbors(n_neighbors=1).fit(Z[T == 0])
match = Y[T == 0][nn.kneighbors(Z[T == 1])[1][:, 0]]
w1, w0, wa = T / e, (1 - T) / (1 - e), (1 - T) * e / (1 - e)
results = {
    "naive": Y[T == 1].mean() - Y[T == 0].mean(),
    "OLS coef": LinearRegression().fit(np.column_stack([T, X]), Y).coef_[0],
    "g-formula": np.mean(m1 - m0),
    "matching ATT": np.mean(Y[T == 1] - match),
    "IPW ATE": w1 @ Y / w1.sum() - w0 @ Y / w0.sum(),
    "IPW ATT": Y[T == 1].mean() - wa @ Y / wa.sum(),
    "AIPW ATE": np.mean(m1 - m0 + T * (Y - m1) / e - (1 - T) * (Y - m0) / (1 - e)),
}
print(f"truth: ATE {tau.mean():.2f}  ATT {tau[T == 1].mean():.2f}")
for k, v in results.items():
    print(f"{k:13s} {v:6.2f}")
# truth: ATE 1.45  ATT 1.76
# naive          -3.12
# OLS coef        1.32
# g-formula       1.30
# matching ATT    1.62
# IPW ATE         0.99
# IPW ATT         1.72
# AIPW ATE        1.28

LogisticRegression(C=1e6) turns off the default L2 penalty in practice: we want the propensity model itself, not a shrunken one. The notebook builds every estimator step by step, adds cross-fitting to AIPW and runs the 200-study experiment.

6. Where it shows up #

• Medicine and epidemiology. Observational drug-safety and effectiveness studies use propensity scores and IPW; target trial emulation (Hernán and Robins) designs the analysis as if it were a randomized trial and then adjusts with these tools. Marginal structural models extend IPW to treatments that change over time.
• Tech products. Users opt in to features, subscriptions and notifications. When an A/B test is not possible, analysts estimate the effect with matching or AIPW on logged covariates, and report a sensitivity analysis.
• Recommender systems and search. Users rate or click only what they were shown. Inverse propensity scoring (IPS) reweights the observed feedback by $1/P(\text{shown})$ to get unbiased estimates of ranking quality and to train debiased models (Schnabel et al. 2016; Joachims et al. 2017).
• Off-policy evaluation. The value of a new policy from logs of an old policy is estimated with importance weights $\pi(a \mid x)/\pi_0(a \mid x)$ and with doubly robust estimators (lesson 11; reinforcement learning 03).
• Machine learning under covariate shift. Training points reweighted by $p_{\text{test}}(x)/p_{\text{train}}(x)$, often estimated with a classifier that predicts "test or train": the same computation as a propensity score.
• Survey statistics. The Horvitz–Thompson estimator was invented (1952) to weight survey respondents by the inverse of their probability of being sampled.
• Science. Effects of policies, environmental exposures (air pollution and health) and land use are estimated from observational data with matching and weighting, together with sensitivity analysis.

7. Common mistakes #

• Adjusting for post-treatment variables. Covariates must be measured before the treatment. Adjusting for a mediator or a collider (lesson 04) creates bias, and no estimator can repair it. Check: draw the graph; for every covariate, ask "could the treatment change it?".
• Judging the propensity model by AUC or accuracy. The target is balance, not prediction. A very high AUC signals poor overlap. Check: SMDs after weighting, propensity histograms.
• Putting every predictor of $T$ into the propensity model. Instruments (causes of $T$ only) add variance and amplify any remaining bias. Include confounders and causes of $Y$.
• Ignoring extreme weights. One unit with weight 500 can decide the estimate. Check: largest weight, ESS, a plot of the weights; try Hájek, trimming or overlap weights, and say which estimand you now estimate.
• Mixing up the estimands. Matching treated units gives the ATT; IPW with $1/e$ gives the ATE; a caliper or trimming gives the effect in a subpopulation. Report the target with the number.
• Reading the OLS coefficient as the ATE. With heterogeneous effects, the coefficient of $T$ is a variance-weighted average of the effects, not the ATE. Use the g-formula with separate models for each arm.
• "Balanced, so causal." Balance on measured $X$ says nothing about unmeasured confounders. Check: a sensitivity analysis (robustness value, E-value).
• "Doubly robust means twice as safe." It means one of two models must be right. With both wrong, AIPW can be worse than either method alone.
• Plain bootstrap for nearest-neighbor matching. It is not valid for matching with a fixed number of matches (Abadie and Imbens 2008). Use the analytic variance or a weighting estimator with the bootstrap.

8. Check yourself #

1. In a cell of 20 people, 5 are treated ($\hat e = 0.25$). Give the IPW weights for the ATE and the size of each weighted group.
   Answer

   Treated: $1/0.25 = 4$; controls: $1/0.75 = 4/3$. Weighted treated: $5 \times 4 = 20$; weighted controls: $15 \times 4/3 = 20$. Each group stands for all 20 people of the cell.
2. Compute the effective sample size of the weights $(2, 2, 2, 2)$ and of $(1, 1, 1, 1, 6)$.
   Answer

   $(8)^2/16 = 4$: equal weights keep the full size. $(10)^2/(4 + 36) = 100/40 = 2.5$: one large weight makes five units worth only 2.5.
3. Why does trimming units with $\hat e \notin [0.1, 0.9]$ change what you estimate? Give an example where it matters.
   Answer

   The estimate now averages the effect over the units that remain, not over the whole population. If the effect differs with $X$ and the trimmed units have different effects (in the figure of section 3.6, the units with large $|x|$ have the largest effects), the trimmed target is different from the ATE (there, below 1).
4. Write the AIPW estimator and show in one line why it is unbiased when $\hat\mu_t = \mu_t$ (and the weights are anything bounded).
   Answer

   $\hat\tau = \frac1n\sum_i[\hat\mu_1 - \hat\mu_0 + T_i(Y_i - \hat\mu_1)/\hat e_i - (1 - T_i)(Y_i - \hat\mu_0)/(1 - \hat e_i)]$. With $\hat\mu_1 = \mu_1$: $\mathbb{E}[T(Y - \mu_1(X))/\hat e(X) \mid X] = e(X)(\mu_1(X) - \mu_1(X))/\hat e(X) = 0$, so the correction terms have mean 0 and the estimator reduces to the g-formula.
5. Compute the E-value for an observed risk ratio of 3, and for a protective risk ratio of 0.5.
   Answer

   $3 + \sqrt{3 \cdot 2} = 3 + 2.45 = 5.45$. For 0.5 use $1/0.5 = 2$: $2 + \sqrt 2 = 3.41$.
6. A regression gives $\hat\tau = 1.32$ with standard error $0.154$ and $\mathrm{df} = 1994$. Compute the robustness value.
   Answer

   $f = 1.32/(0.154\sqrt{1994}) = 1.32/6.88 = 0.192$, so $f^2 = 0.0368$. $\mathrm{RV} = \frac12(\sqrt{0.0368^2 + 4 \cdot 0.0368} - 0.0368) = \frac12(0.386 - 0.037) = 0.174$. A confounder that explains 17.4% of the residual variance of both $T$ and $Y$ would bring the estimate to 0.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. State conditional exchangeability in symbols and in words.

   Show answer

2. What is positivity (overlap), and how much of it does the ATT need?

   Show answer

3. Write the g-formula (standardization) for the ATE.

   Show answer

4. What does the balancing theorem of the propensity score say?

   Show answer

5. Write the Horvitz–Thompson IPW estimator of the ATE.

   Show answer

6. Which weights make the control group look like the treated group (ATT)?

   Show answer

7. Compute the effective sample size of the weights $(1, 1, 1, 1, 6)$.

   Show answer

8. Define the standardized mean difference and the usual balance rule.

   Show answer

9. What does "doubly robust" mean for AIPW?

   Show answer

10. Compute the E-value for an observed risk ratio of 2.

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "How would you estimate the effect of a feature that users opted into?" — Draw the graph: what drives opting in and the outcome (activity before the launch, tenure, platform)? Use only pre-treatment covariates. Fit a propensity model, check overlap and balance, estimate with AIPW (or IPW and matching as checks), report a confidence interval and a sensitivity analysis. Say that the result depends on "no unmeasured confounding" and suggest a holdout or an encouragement experiment if possible.
• "What is a propensity score and how do you check it?" — $P(T = 1 \mid X)$. It balances $X$ (Rosenbaum and Rubin). Check it by balance (SMDs after weighting below 0.1), overlap (histograms, largest weight, ESS) and calibration, not by AUC.
• "What does doubly robust mean?" — An estimator that combines an outcome model and a propensity model and is consistent if either one is right. AIPW: regression estimate plus inverse-weighted residuals. Its bias is a product of the two errors, which is why it works well with ML models and cross-fitting (double ML).
• "Matching or weighting?" — Matching is transparent and targets the ATT; it struggles with many covariates and its standard errors need care. Weighting targets the ATE or the ATT and uses all units, but is sensitive to extreme weights. AIPW is usually the best default.
• "Your IPW estimate jumps when you remove one user. Why?" — That user has an extreme weight (propensity near 0 or 1): poor overlap. Look at the ESS, use Hájek or overlap weights, or trim and report the new estimand.

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution. Every exercise uses a simulation with a known true effect:

1. standardize(df): the g-formula with one linear model per arm; compare it with the naive difference and the OLS coefficient.
2. nn_match_att(df, caliper): nearest-neighbor matching on standardized covariates with NearestNeighbors; a balance table of SMDs before and after.
3. ipw(df, e_hat, kind): the Horvitz–Thompson, Hájek and ATT versions with a LogisticRegression propensity model; weighted SMDs and the effective sample size.
4. aipw(df) with 2-fold cross-fitting; the $2 \times 2$ experiment with right and wrong models.
5. Overlap: increase the confounding, compare the spread of IPW and AIPW, and see how trimming changes the estimand.

11. Watch and read #

6.4 - Propensity Scores and Inverse Probability Weighting (IPW)Brady Neal · Introduction to Causal Inference · 11 min

The propensity score, its balancing property and the IPW estimator, with the derivation from the adjustment formula.

6.5 - Doubly Robust Methods, Matching, Double Machine Learning, and Causal TreesBrady Neal · Introduction to Causal Inference · 8 min

A short overview of AIPW and matching, and a preview of double ML for lesson 08.

Propensity score matching: an introductionBen Lambert · 9 min

Matching on the propensity score step by step, with the balance check.

7.5 - Sensitivity Analysis BasicsBrady Neal · Introduction to Causal Inference · 16 min

How strong an unobserved confounder must be to change a conclusion, in a linear model.

Matching Methods: Causal Inference BootcampMod•U · Causal Inference Bootcamp · 3 min

The idea of matching in three minutes: compare each treated unit with a similar untreated unit.

• Miguel Hernán and James Robins, Causal Inference: What If (free): chapter 12 (IP weighting), chapter 13 (standardization and the g-formula), chapter 15 (outcome regression and propensity scores).
• Brady Neal, Introduction to Causal Inference (free PDF): chapter 7 (estimation: outcome models, propensity scores, IPW, doubly robust methods) and chapter 8 (sensitivity analysis).
• Matheus Facure, Causal Inference for the Brave and True: Matching, Propensity Score and Doubly Robust Estimation, with Python code.
• Peter Austin (2011), An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies, Multivariate Behavioral Research: matching, weighting, balance diagnostics and the SMD.
• Elizabeth Stuart (2010), Matching Methods for Causal Inference: A Review and a Look Forward, Statistical Science.
• Joseph Kang and Joseph Schafer (2007), Demystifying Double Robustness, Statistical Science: when AIPW fails with two wrong models.
• Carlos Cinelli and Chad Hazlett (2020), Making Sense of Sensitivity: Extending Omitted Variable Bias, Journal of the Royal Statistical Society B: partial $R^2$, the robustness value and benchmarks.
• Nick Huntington-Klein, The Effect, chapter 14: Matching (free online): matching and weighting with many examples.