Potential outcomes, the fundamental problem and randomized experiments

Before this lesson: lesson 01 (association versus causal effect, confounding); expectation and variance (probability lesson 02) and confidence intervals (probability lesson 06). Helpful: hypothesis testing and A/B testing. Time: about 85 minutes, plus the notebook.

---

Why this matters #

• A/B tests are randomized experiments. Every claim "the new ranking model raises clicks by 2%" rests on the logic of this lesson. When the logic breaks (users talk to each other, the split is not really random), the numbers are wrong.
• Clinical trials are randomized for the same reason. Regulators accept a drug on a randomized trial, not on observational data.
• Observational methods (lessons 05–08) try to imitate a randomized experiment. Their assumptions (exchangeability given $X$, positivity) are the conditions of this lesson, made conditional on covariates.
• Causal ML (uplift models, heterogeneous effects in lesson 08) estimates the CATE $\tau(x)$, defined here.

1. Predict first #

2. The picture #

Two columns for every unit #

Eight students ($i = 1, \dots, 8$) can take a tutoring program ($T_i = 1$) or not ($T_i = 0$). For each student there are two potential outcomes: $Y_i(1)$, the test score if the student takes tutoring, and $Y_i(0)$, the score if not. The individual treatment effect is $\tau_i = Y_i(1) - Y_i(0)$. A table of both columns for all units is called the science table. These numbers are made up for teaching:

student	1	2	3	4	5	6	7	8
$Y_i(0)$	40	45	50	55	70	75	80	85
$Y_i(1)$	50	55	58	63	74	78	84	88
$\tau_i$	10	10	8	8	4	3	4	3

Tutoring helps every student, and it helps the weak students most. The average treatment effect is $\tau_{\mathrm{ATE}} = (10 + 10 + 8 + 8 + 4 + 3 + 4 + 3)/8 = 6.25$ points.

Now suppose the four weakest students choose tutoring: $T = (1, 1, 1, 1, 0, 0, 0, 0)$. We see $Y(1)$ for students 1–4 and $Y(0)$ for students 5–8, and nothing else.

Interactive figure: loading…

The naive difference compares the observed means: $\bar Y_{\text{tutored}} - \bar Y_{\text{not}} = 56.5 - 77.5 = -21$. The data seem to say "tutoring lowers scores by 21 points". The reason: the tutored students were weaker before tutoring. Without tutoring, their mean would have been $(40 + 45 + 50 + 55)/4 = 47.5$, far below the controls' $77.5$. This gap of $-30$ is the selection bias. The effect on the tutored students themselves is $+9$. And indeed $-21 = 9 + (-30)$.

The fundamental problem #

For each unit we observe only

The other potential outcome is the counterfactual: what would have happened under the other treatment. Holland (1986) called this the fundamental problem of causal inference: we can never observe $\tau_i$ for any unit. So causal inference is a missing-data problem. Every method of this track fills the missing half of the science table with assumptions plus data. The question is always: which assumptions, and can we defend them?

3. Definitions and main results #

Potential outcomes and the assumptions that link them to data #

The potential-outcomes framework goes back to Neyman (1923) for experiments and Rubin (1974) for observational studies; it is also called the Neyman–Rubin model. We write $T$ for the treatment and $Y$ for the outcome. Other books use $A$ (Hernán and Robins), $D$ (economics), $W$ (Imbens and Rubin) or $X$ (Pearl) for the treatment; the ideas are the same.

To connect the potential outcomes to the data we need two assumptions.

• Consistency. If unit $i$ receives treatment $t$, its observed outcome is its potential outcome under $t$: $T_i = t \Rightarrow Y_i = Y_i(t)$. This needs a well-defined treatment: "tutoring" must mean one program, not ten different ones. "Being overweight" is a poor treatment: diet, exercise and surgery all change weight, with different effects on health.
• No interference. The potential outcomes of unit $i$ depend only on $i$'s own treatment, not on the treatments of other units. It fails for vaccines (my infection risk depends on whether my family is vaccinated), for social networks (my posts depend on my friends' feed), and for marketplaces (a discount for some buyers takes items away from others).

Together these two are the stable unit treatment value assumption (SUTVA) (Rubin 1980). Without SUTVA, $Y_i(1)$ is not even one number: it depends on everyone else's treatment.

Estimands: which average do we want? #

The estimand is the quantity we want to know. The estimator is the recipe that turns data into a number. The estimate is the number we get. Four common estimands:

Estimand	Definition	Question it answers
ATE	$\tau_{\mathrm{ATE}} = \mathbb{E}[Y(1) - Y(0)]$	What if we treat everyone versus no one?
ATT	$\tau_{\mathrm{ATT}} = \mathbb{E}[Y(1) - Y(0) \mid T = 1]$	What did the treatment do for those who took it?
ATC	$\tau_{\mathrm{ATC}} = \mathbb{E}[Y(1) - Y(0) \mid T = 0]$	What would it do for those who did not take it?
CATE	$\tau(x) = \mathbb{E}[Y(1) - Y(0) \mid X = x]$	What is the effect for units with covariates $x$?

For the tutoring table with the four weakest treated: ATE $= 6.25$, ATT $= (10 + 10 + 8 + 8)/4 = 9$, ATC $= (4 + 3 + 4 + 3)/4 = 3.5$. They differ because the effect differs between students. A school that wants to make tutoring compulsory needs the ATE; a school that asks "was our voluntary program worth it?" needs the ATT. The covariates $X$ are always pre-treatment: measured before $T$ and not affected by it.

Why the naive difference fails #

Proposition (decomposition of the naive difference). Under consistency,

Proof.

1. By consistency, $Y = Y(1)$ when $T = 1$ and $Y = Y(0)$ when $T = 0$. So the naive difference is $\mathbb{E}[Y(1) \mid T{=}1] - \mathbb{E}[Y(0) \mid T{=}0]$.
2. Subtract and add the same term $\mathbb{E}[Y(0) \mid T{=}1]$ (the treated group's mean without treatment, which we never observe): $\big(\mathbb{E}[Y(1) \mid T{=}1] - \mathbb{E}[Y(0) \mid T{=}1]\big) + \big(\mathbb{E}[Y(0) \mid T{=}1] - \mathbb{E}[Y(0) \mid T{=}0]\big)$.
3. By linearity of expectation, the first bracket is $\mathbb{E}[Y(1) - Y(0) \mid T{=}1] = \tau_{\mathrm{ATT}}$. $\blacksquare$

The selection bias compares the two groups as they would be without treatment. It is zero when the groups are alike in their $Y(0)$. With the weakest students treated it is $47.5 - 77.5 = -30$, and $-21 = 9 + (-30)$. With the strongest treated, the naive difference is $81 - 47.5 = 33.5 = 3.5 + 30$.

Identification: exchangeability #

Identification means writing the estimand as a function of the distribution of the observed data $(T, Y)$ (or $(X, T, Y)$). Only then can data estimate it. We separate two steps: what the data can identify (this section), and how to estimate it well (the next sections).

• Exchangeability (also called ignorability or unconfoundedness): $(Y(0), Y(1)) \perp\!\!\!\perp T$. The treatment is independent of the potential outcomes: the treated and the untreated would have had the same distribution of outcomes under either treatment.
• Positivity: $0 < P(T = 1) < 1$. Both groups exist.

Theorem. Under consistency, exchangeability and positivity, for $t \in \{0, 1\}$: $\mathbb{E}[Y(t)] = \mathbb{E}[Y \mid T = t]$. Hence $\tau_{\mathrm{ATE}} = \mathbb{E}[Y \mid T{=}1] - \mathbb{E}[Y \mid T{=}0]$.

Proof.

1. $\mathbb{E}[Y(t)] = \mathbb{E}[Y(t) \mid T = t]$, by exchangeability: $Y(t)$ is independent of $T$, so conditioning on $T$ does not change its mean. The conditional mean is defined because of positivity ($P(T = t) > 0$).
2. $\mathbb{E}[Y(t) \mid T = t] = \mathbb{E}[Y \mid T = t]$, by consistency: in the group with $T = t$, $Y(t)$ is the observed $Y$.
3. Take the difference of the results for $t = 1$ and $t = 0$. $\blacksquare$

In the decomposition, exchangeability makes the selection bias zero and the ATT equal to the ATE. In observational data we cannot test exchangeability: it is a statement about $Y(0)$ of the treated, which is never observed. Conditional exchangeability $(Y(0), Y(1)) \perp\!\!\!\perp T \mid X$ and positivity given $X$, $0 < e(x) = P(T{=}1 \mid X{=}x) < 1$, are the versions used with observed confounders in lesson 05. Positivity can be checked in data; conditional exchangeability cannot.

Randomization makes exchangeability true by design #

In a randomized experiment the experimenter assigns $T$ with a random device: a coin for each unit (Bernoulli randomization), or a lottery that picks exactly $n_1$ of $n$ units (complete randomization). The random device does not look at the units. So $T$ is independent of everything about the units, including $(Y(0), Y(1))$ and every covariate, measured or not. Exchangeability holds by construction, and the theorem gives $\tau_{\mathrm{ATE}} = \mathbb{E}[Y \mid T{=}1] - \mathbb{E}[Y \mid T{=}0]$.

There is also an exact finite-sample version. Fix the $n$ units and their science table. Only the lottery is random.

Theorem (Neyman 1923). Under complete randomization with $n_1$ treated and $n_0 = n - n_1$ controls, the difference in means $\hat\tau = \frac{1}{n_1}\sum_{i} T_i Y_i - \frac{1}{n_0}\sum_{i} (1 - T_i) Y_i$ is unbiased for the sample ATE $\tau = \frac1n \sum_i \tau_i$, and

where $S_t^2 = \frac{1}{n - 1}\sum_i \big(Y_i(t) - \bar Y(t)\big)^2$ and $S_\tau^2 = \frac{1}{n-1}\sum_i (\tau_i - \tau)^2$.

Proof of unbiasedness (and where the variance comes from)

1. By consistency, $\hat\tau = \frac{1}{n_1}\sum_i T_i Y_i(1) - \frac{1}{n_0}\sum_i (1 - T_i) Y_i(0)$. The potential outcomes are fixed numbers; only the $T_i$ are random.
2. Every unit has the same chance to be treated: $P(T_i = 1) = n_1 / n$, so $\mathbb{E}[T_i] = n_1/n$ and $\mathbb{E}[1 - T_i] = n_0/n$.
3. By linearity: $\mathbb{E}[\hat\tau] = \frac{1}{n_1}\sum_i \frac{n_1}{n} Y_i(1) - \frac{1}{n_0}\sum_i \frac{n_0}{n} Y_i(0) = \bar Y(1) - \bar Y(0) = \tau$. $\blacksquare$

The variance needs the covariances $\operatorname{Cov}(T_i, T_j) = -\frac{n_1 n_0}{n^2(n-1)}$ for $i \ne j$ (drawing one unit for treatment makes the others slightly less likely). A direct calculation with these covariances gives the formula (Imbens and Rubin 2015, chapter 6). The notebook checks it by enumerating all assignments.

"Unbiased" means "right on average over the lotteries". One experiment can still miss. With the tutoring table there are $\binom{8}{4} = 70$ possible lotteries. Their 70 estimates have mean exactly $6.25$ and range from $-21$ to $33.5$; 20 of the 70 (28.6%) are negative.

Interactive figure: loading…

Balance. Randomization balances every covariate on average: the expected difference in means of any pre-treatment variable between the groups is zero, for measured and unmeasured variables alike. In one experiment, chance imbalance happens, and it is the reason for the spread above.

Neyman's variance and why the usual estimate is conservative #

For the tutoring table: $S_1^2 = 200.79$, $S_0^2 = 292.86$, $S_\tau^2 = 9.36$, $n_1 = n_0 = 4$, $n = 8$:

The SD is $\sqrt{122.24} = 11.06$. This is exactly the variance of the 70 estimates in the figure.

In a real experiment we do not know $S_\tau^2$: it needs both potential outcomes of the same unit. The usual estimate replaces $S_1^2$ and $S_0^2$ by the sample variances $s_1^2, s_0^2$ of the observed groups and drops the last term:

Its expectation is $S_1^2/n_1 + S_0^2/n_0 \ge \operatorname{Var}(\hat\tau)$ (here $123.41$ against $122.24$). So it is conservative: confidence intervals are a little too wide, never too narrow on average. The two agree when the effect is the same for every unit ($S_\tau^2 = 0$). With large $n$, $\hat\tau$ is close to normal (central limit theorem), and $\hat\tau \pm 1.96\sqrt{\widehat{\operatorname{Var}}}$ is an approximate 95% confidence interval (probability lesson 06).

Fisher's randomization test #

Fisher (1935) asked a different question. The sharp null hypothesis says the treatment has no effect for any unit: $H_0: \tau_i = 0$ for all $i$, that is, $Y_i(1) = Y_i(0)$. Under $H_0$ the science table is fully known: both potential outcomes equal the observed $Y_i$. So we can compute the test statistic for every possible lottery, and the p-value is exact, with no large-sample approximation.

Fisher randomization test.

1. Choose a statistic, for example the difference in means $D(T, Y)$. Compute it on the real assignment: $d_{\text{obs}}$.
2. Under $H_0$, the outcome vector $Y$ stays the same for every assignment. For each possible assignment $T'$ (all $\binom{n}{n_1}$ of them, or many random ones), compute $D(T', Y)$.
3. The two-sided p-value is the share of assignments with $|D(T', Y)| \ge |d_{\text{obs}}|$.

The p-value is valid in any sample size, because it uses only the randomization that the experimenter really did. It is the permutation test of probability lesson 12, seen as a causal test.

Two different nulls. Neyman's null is "the average effect is zero": $\tau_{\mathrm{ATE}} = 0$. Fisher's sharp null is "the effect is zero for every unit". The sharp null is stronger. Effects of $+5$ for half the units and $-5$ for the other half satisfy Neyman's null but not Fisher's.

Better designs: blocking and regression adjustment #

The spread of $\hat\tau$ comes from chance imbalance in covariates that affect the outcome. Two tools reduce it.

• Blocking (stratified randomization). Group units into blocks with similar covariates (age groups, countries, user segments) and run a separate lottery inside each block. The estimate is a weighted average of the block estimates (weights = block sizes). Inside each block the covariate is nearly constant, so the imbalance in it is removed by design. The estimate stays unbiased. Its variance is much smaller when the blocking variable predicts the outcome. When the blocking variable does not predict the outcome, blocking costs little (in small samples it can be slightly worse). Matched pairs are blocks of size 2.
• Regression adjustment. Regress $Y$ on $T$ and pre-treatment covariates $X$. Lin (2013) showed that the regression with $T$, centered $X$ and their interaction $T \cdot (X - \bar X)$ is consistent for the ATE and asymptotically never less precise than the difference in means, even if the linear model is wrong. CUPED, a common variance-reduction method in A/B testing, is regression adjustment on the pre-experiment value of the metric (see probability lesson 12).

Never adjust for a post-treatment variable. A variable measured after the treatment can be affected by it (a mediator, lesson 01). Adjusting for it removes part of the effect, or creates bias by selecting different kinds of units in each group. The covariates in blocking and in regression adjustment must be pre-treatment.

Interactive figure: loading…

A/B tests are randomized experiments #

Online experiments add a few practical points.

• Unit of randomization versus unit of analysis. If you randomize users but analyze page views, the page views of one user are correlated, and the usual standard error is too small. Analyze at the level of the randomized unit, or use cluster-robust standard errors.
• Sample-ratio mismatch (SRM). If the design splits 50/50 and you see 50,600 against 49,400 users, a chi-square test gives $\chi^2 = 2 \cdot 600^2 / 50{,}000 = 14.4$, $p \approx 0.00015$. Something broke the randomization (a bug, a redirect, bots), and the groups may no longer be exchangeable. Check SRM before you read any result.
• Interference. In marketplaces and social networks, treating one user affects others, and SUTVA fails. Cluster randomization randomizes whole groups (cities, schools, friend clusters). Switchback designs switch a whole market between treatment and control over time windows.
• Sample size and peeking. How many users you need, and why stopping at the first significant result inflates false positives, are covered in probability lesson 12 and interview lesson 09.

Answers to section 1. (1) Yes. With the weakest four treated, the tutored mean is 56.5 and the other mean is 77.5: a naive difference of $-21$, while every $\tau_i > 0$. The selection bias ($-30$) is larger than the effect on the treated ($+9$). (2) No. Unbiasedness is about the average over all lotteries; one estimate has random error with SD about 11 in the tutoring example. (3) The smallest two-sided p-value is $2/70 \approx 0.029$: the observed assignment and its mirror image are the two most extreme of the 70.

4. Worked example #

(a) Estimands and the decomposition. Use the tutoring table with the four weakest students treated, $T = (1, 1, 1, 1, 0, 0, 0, 0)$.

1. Individual effects: $\tau = (10, 10, 8, 8, 4, 3, 4, 3)$.
2. ATE: $(10 + 10 + 8 + 8 + 4 + 3 + 4 + 3)/8 = 50/8 = 6.25$.
3. ATT (students 1–4): $(10 + 10 + 8 + 8)/4 = 9$. ATC (students 5–8): $(4 + 3 + 4 + 3)/4 = 3.5$. Check: $\tfrac48 \cdot 9 + \tfrac48 \cdot 3.5 = 6.25$.
4. Observed means: treated $(50 + 55 + 58 + 63)/4 = 56.5$; control $(70 + 75 + 80 + 85)/4 = 77.5$. Naive difference $= -21$.
5. Selection bias: $\mathbb{E}[Y(0) \mid T{=}1] = (40 + 45 + 50 + 55)/4 = 47.5$, minus $77.5$, gives $-30$.
6. Check: $-21 = 9 + (-30)$. ✓

(b) Neyman's variance. Same table, complete randomization with $n_1 = n_0 = 4$.

1. $\bar Y(1) = 550/8 = 68.75$. $S_1^2 = \frac{1}{7}\sum_i (Y_i(1) - 68.75)^2 = 1405.5/7 = 200.79$.
2. $\bar Y(0) = 500/8 = 62.5$. $S_0^2 = 2050/7 = 292.86$.
3. $\bar\tau = 6.25$. $S_\tau^2 = \frac{1}{7}\sum_i (\tau_i - 6.25)^2 = 65.5/7 = 9.36$.
4. $\operatorname{Var}(\hat\tau) = 200.79/4 + 292.86/4 - 9.36/8 = 122.24$; SD $11.06$.
5. Without the last term: $123.41$ (SD $11.11$), the conservative value.

(c) Fisher's test by hand, with four units. Two of four units are treated by lottery: units 1 and 2. Observed outcomes $Y = (9, 7, 3, 1)$. The statistic is $d_{\text{obs}} = (9 + 7)/2 - (3 + 1)/2 = 8 - 2 = 6$.

1. Under the sharp null, the outcomes $(9, 7, 3, 1)$ stay the same under every lottery.
2. There are $\binom42 = 6$ lotteries. Treated pairs and statistics: $\{1,2\}$: $8 - 2 = 6$; $\{1,3\}$: $6 - 4 = 2$; $\{1,4\}$: $5 - 5 = 0$; $\{2,3\}$: $5 - 5 = 0$; $\{2,4\}$: $4 - 6 = -2$; $\{3,4\}$: $2 - 8 = -6$.
3. Lotteries with $|D| \ge 6$: two of six. p-value $= 2/6 = 0.33$.

Even the most extreme result gives $p = 1/3$: four units can never show a significant effect. The test is exact, but the design decides what it can detect.

(d) One real lottery for the eight students. Suppose the lottery tutors students 1, 3, 4 and 7. Observed: treated $(50, 58, 63, 84)$, mean $63.75$; controls $(45, 70, 75, 85)$, mean $68.75$; $\hat\tau = -5$, although the true ATE is $+6.25$. Fisher's test over the 70 lotteries gives $p = 48/70 = 0.686$. With eight students, chance imbalance dominates: here the control group happens to contain three of the four strongest students (5, 6 and 8).

5. In code #

NumPy checks parts (a), (b) and (d). The notebook has these cells and the exercises.

import numpy as np
from itertools import combinations

Y0 = np.array([40, 45, 50, 55, 70, 75, 80, 85.])
Y1 = np.array([50, 55, 58, 63, 74, 78, 84, 88.])
tau = Y1 - Y0
T = np.array([1, 1, 1, 1, 0, 0, 0, 0])          # the four weakest take tutoring

naive = Y1[T == 1].mean() - Y0[T == 0].mean()
att = tau[T == 1].mean()
bias = Y0[T == 1].mean() - Y0[T == 0].mean()
print(tau.mean(), att, naive, bias)             # 6.25 9.0 -21.0 -30.0

# all 70 lotteries that give tutoring to 4 of the 8 students
est = []
for treated in combinations(range(8), 4):
    t = np.zeros(8, dtype=int); t[list(treated)] = 1
    est.append(Y1[t == 1].mean() - Y0[t == 0].mean())
est = np.array(est)
neyman = Y1.var(ddof=1) / 4 + Y0.var(ddof=1) / 4 - tau.var(ddof=1) / 8
print(len(est), est.mean(), est.var().round(2), neyman.round(2))   # 70 6.25 122.24 122.24
print((est < 0).mean().round(3))                                   # 0.286

# Fisher randomization test for one lottery: students 1, 3, 4, 7 tutored
t_obs = np.array([1, 0, 1, 1, 0, 0, 1, 0])
y_obs = np.where(t_obs == 1, Y1, Y0)
stat = y_obs[t_obs == 1].mean() - y_obs[t_obs == 0].mean()
null = []
for treated in combinations(range(8), 4):       # sharp null: y_obs is the whole table
    t = np.zeros(8, dtype=int); t[list(treated)] = 1
    null.append(y_obs[t == 1].mean() - y_obs[t == 0].mean())
p = np.mean(np.abs(null) >= abs(stat) - 1e-9)
print(stat, p.round(3))                          # -5.0 0.686

est.var() divides by the number of lotteries (70), because the 70 lotteries are the whole population of possible experiments. The small tolerance 1e-9 protects the comparison from rounding.

6. Where it shows up #

• Online experiments. Search, ads, feeds and recommenders run thousands of A/B tests a year. Ranking-model launches are decided on randomized comparisons of user metrics, not on offline accuracy alone (Kohavi, Tang and Xu 2020).
• Clinical trials and vaccine trials. Randomized controlled trials are the standard of evidence. Vaccine trials face interference; cluster-randomized trials randomize villages or schools.
• Uplift modeling and heterogeneous effects. Marketing targets users with a high CATE $\tau(x)$, not users with a high predicted outcome. Randomized data are the best training data for these models (lesson 08).
• Policy learning and bandits. Logged data from randomized policies allow unbiased evaluation of new policies by importance weighting (lesson 11, RL lesson 03).
• ML for science. Randomized designs appear in agricultural field trials (Fisher's original setting), in perturbation screens in biology (CRISPR knockouts as interventions) and in randomized evaluation of AI tools in scientific workflows.

7. Common mistakes #

• Reading the naive difference as an effect. It is $\tau_{\mathrm{ATT}}$ plus selection bias. Check: ask how units got their treatment; if they chose it, expect selection bias.
• "Unbiased" means "correct in my sample". No: correct on average over randomizations. Check: report a confidence interval, not only the point estimate.
• Mixing up estimands. ATE, ATT and ATC differ when effects differ between units. Check: write the estimand before the estimator, and say which population it is about.
• Ignoring interference. If treated users affect control users, the difference in means does not estimate any of the usual estimands. Check: ask whether units interact; consider cluster or switchback randomization.
• Adjusting for post-treatment variables. It breaks the balance that randomization gave. Check: every covariate in the analysis must be measured before assignment.
• Skipping the randomization check. A sample-ratio mismatch or covariate imbalance far beyond chance means the randomization failed. Check: test the split ratio and the balance of pre-treatment covariates first.
• Confusing the sharp null with "no average effect". Fisher's test answers "no effect for anyone"; Neyman's confidence interval answers questions about the ATE.

8. Check yourself #

1. Write the observed outcome $Y_i$ in terms of $T_i$, $Y_i(0)$ and $Y_i(1)$. Which assumption does this use?
   Answer

   $Y_i = T_i Y_i(1) + (1 - T_i) Y_i(0)$. It uses consistency (and no interference, so that $Y_i(t)$ depends only on $T_i$).
2. In the tutoring table, let the four strongest students take tutoring. Compute the naive difference, the ATT and the selection bias.
   Answer

   Treated mean $(74 + 78 + 84 + 88)/4 = 81$; control mean $(40 + 45 + 50 + 55)/4 = 47.5$; naive $= 33.5$. ATT $= (4 + 3 + 4 + 3)/4 = 3.5$. Selection bias $= (70 + 75 + 80 + 85)/4 - 47.5 = 77.5 - 47.5 = 30$. Check: $33.5 = 3.5 + 30$.
3. A vaccine trial randomizes individuals inside the same households. Which assumption is in danger, and what design fixes it?
   Answer

   No interference (part of SUTVA): a vaccinated family member lowers the infection risk of an unvaccinated one, so control outcomes depend on others' treatment. Randomize whole households (cluster randomization), and analyze at the household level.
4. Why is the usual variance estimate $s_1^2/n_1 + s_0^2/n_0$ conservative?
   Answer

   The true variance is $S_1^2/n_1 + S_0^2/n_0 - S_\tau^2/n$. The term $S_\tau^2$ (the spread of individual effects) cannot be estimated, because no unit shows both potential outcomes. Dropping it gives an expectation that is at least the true variance; equality holds when all $\tau_i$ are equal.
5. Give one reason to report the ATT instead of the ATE.
   Answer

   When the policy question is about those who take the treatment, for example "was the voluntary program worth it for its participants?". The ATT is also identified under weaker assumptions in some designs (it needs only $Y(0)$ to be exchangeable).
6. In a test with 10 units and 5 treated, what is the smallest possible two-sided Fisher p-value with the difference in means?
   Answer

   There are $\binom{10}{5} = 252$ assignments. The observed assignment and its mirror (treated and control swapped) give $\pm d$, so at best 2 of 252 are as extreme: $p = 2/252 \approx 0.008$ (if no other assignment gives the same absolute value).

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. What are the potential outcomes $Y_i(1)$ and $Y_i(0)$?

   Show answer

2. State the fundamental problem of causal inference.

   Show answer

3. What are the two parts of SUTVA?

   Show answer

4. Decompose the naive difference in means.

   Show answer

5. Define exchangeability, and say why randomization gives it.

   Show answer

6. Compute the ATT and ATE: $\tau = (10, 10, 8, 8, 4, 3, 4, 3)$, units 1–4 treated.

   Show answer

7. Give Neyman's variance of the difference in means under complete randomization.

   Show answer

8. What is Fisher's sharp null, and why does it give an exact test?

   Show answer

9. Why must you not adjust for a post-treatment variable in an experiment?

   Show answer

10. When do you report the ATE and when the ATT?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Why do we randomize?" — Randomization makes the treatment independent of the potential outcomes (exchangeability), so the difference in means is unbiased for the ATE. It balances unmeasured confounders too, which no adjustment can do.
• "What is SUTVA, and when does it break in a marketplace?" — Consistency plus no interference. In a two-sided marketplace, a price change for treated buyers changes supply for control buyers, so control outcomes are contaminated. Use switchback or cluster (market-level) randomization.
• "Your A/B test shows +2%, but the treatment group has more users than planned. What do you check?" — Sample-ratio mismatch: a chi-square test of the split. If it fails, find the cause (bucketing bug, redirects, bot filtering, a crash that drops users from one arm) before trusting any metric.
• "Can you adjust for covariates in an A/B test?" — Yes, for pre-treatment covariates (CUPED, Lin's interacted regression): unbiasedness stays (asymptotically) and variance falls. Never for post-treatment variables.
• See interview lesson 09 for sample size, power and peeking.

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution:

1. naive_decomposition(Y0, Y1, T) returns the naive difference, the ATT and the selection bias; check $-21 = 9 - 30$.
2. Enumerate all 70 lotteries; check unbiasedness and Neyman's variance exactly.
3. fisher_test(y, t, n_perm, seed): the randomization p-value for the sharp null; compare it with a t-test p-value on a larger simulated A/B test.
4. Blocked versus complete randomization: simulate 2,000 experiments and compare the SD of the estimate; then the Lin regression adjustment on a pre-treatment covariate.
5. Bad adjustment: add a post-treatment covariate (affected by $T$) to the regression and measure the bias against the known truth.

11. Watch and read #

2.1 - What are Potential Outcomes?Brady Neal - Causal Inference · 4 min

The definition of potential outcomes and individual effects, with a short example.

2.2 - The Fundamental Problem of Causal InferenceBrady Neal - Causal Inference · 4 min

Why one of the two potential outcomes is always missing.

2 - Potential Outcomes (Week 2)Brady Neal - Causal Inference · 54 min

The full lecture: ATE, the naive difference, ignorability, SUTVA and the identification proof of section 3.

5.2 - The Magic of Randomized ExperimentsBrady Neal - Causal Inference · 12 min

Why randomization removes confounding, also in the language of causal graphs.

Randomized Experiments: Causal Inference BootcampMod•U: Powerful Concepts in Social Science · 6 min

A short, non-technical summary of why experiments answer causal questions.

• Miguel Hernán and James Robins, Causal Inference: What If, free book, chapters 1–2 (potential outcomes, randomized experiments).
• Brady Neal, Introduction to Causal Inference, free PDF, chapter 2.
• Scott Cunningham, Causal Inference: The Mixtape, ch. 3 "Potential Outcomes and Randomization", free online (potential outcomes, Fisher's exact test).
• Winston Lin, "Agnostic notes on regression adjustments to experimental data" (Annals of Applied Statistics, 2013), free on arXiv.
• Books to know (not free): Guido Imbens and Donald Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences (2015), chapters 1–6; Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments (2020); Paul Holland, "Statistics and causal inference", JASA 1986.