Correlation, causation and Simpson's paradox

Before this lesson: nothing from this track; this is the first lesson. You need conditional probability (probability lesson 01), averages and correlation (foundations lesson 08), and the idea of a regression line. Time: about 70 minutes, plus the notebook.

---

Why this matters #

Machine learning models learn associations. A model that predicts well can still give a wrong answer to a causal question. Some examples:

• Decisions from predictions. A model of pneumonia risk learned that patients with asthma have a lower risk of death (Caruana et al. 2015). This is true in the data, because doctors send asthma patients straight to intensive care. A policy "treat asthma patients less" built on this model would kill people.
• Recommenders. A recommender shows item A and sees more clicks on it. Did showing the item cause the clicks, or did the system show it to users who would have clicked anyway?
• Medicine and policy. A hospital gives a stronger drug to sicker patients. The drug group then looks worse, even if the drug helps.
• Robustness. A model that uses a non-causal feature (a hospital's scanner type, a watermark in an image) breaks when that association changes (lesson 11).

Prediction answers "given that I see $X$, what is $Y$?". Causal inference answers "if I set $X$, what happens to $Y$?". This lesson shows why the two questions have different answers.

1. Predict first #

2. The picture #

Five reasons for one association #

Two variables $X$ and $Y$ are associated (or dependent) when knowing the value of one changes what we expect of the other: $P(Y \mid X = x)$ depends on $x$. A correlation is one number that measures a linear association. There are five different reasons why $X$ and $Y$ can be associated:

Source	Causal structure	Example
(a) causation	$X \to Y$	more fertilizer, larger crop
(b) reverse causation	$X \leftarrow Y$	more crime makes a city hire more police; the data say "more police, more crime"
(c) common cause (confounding)	$X \leftarrow Z \to Y$	hot weather raises ice-cream sales and drownings
(d) selection on a common effect (collider, Berkson)	$X \to C \leftarrow Y$, we only see units with some values of $C$	among cast actors, talent and looks are negatively related
(e) chance	no link	among 1,000 unrelated pairs of short series, some correlate strongly

The arrows form a causal graph: an arrow $A \to B$ means "$A$ is a direct cause of $B$". Lesson 03 makes these graphs precise. Here they are only pictures.

Interactive figure: loading…

Only source (a) means "changing $X$ changes $Y$". Sources (b)–(e) give the same kind of scatter, but changing $X$ would do nothing to $Y$. The data alone cannot tell the five apart. We need knowledge about how the data were produced, or an experiment (lesson 02).

Reichenbach's common cause principle (1956) says: if $X$ and $Y$ are dependent, then $X$ causes $Y$, $Y$ causes $X$, or they have a common cause. The principle assumes that the data are a fair sample of the population (no selection) and that the dependence is real (not chance). Sources (d) and (e) are exactly the cases where these two assumptions fail.

Confounding: a common cause tilts the cloud #

Now take a common cause $Z$ that affects both a "treatment" $T$ and an "outcome" $Y$, and let $T$ also have a real effect on $Y$. A small structural causal model (a list of equations, one per variable, each computed from its causes and some noise) is

with independent noises $\varepsilon_T, \varepsilon_Y \sim \mathcal N(0, 1)$. The sign $:=$ means "is computed from", not just "equals". The causal effect of $T$ on $Y$ is $b$: if we raise $T$ by 1 and keep everything else fixed, $Y$ rises by $b$. A variable such as $Z$, which affects both the treatment and the outcome, is a confounder.

Interactive figure: loading…

With $a = 1$, $b = 1$, $c = 2$ the pooled slope is about $2$, twice the causal effect. Inside a thin slice of $Z$, the variation in $T$ no longer comes from $Z$, and the slope is the causal $1$. In section 3 we prove the formula for the pooled slope.

Simpson's paradox with real numbers #

A famous study compared two treatments for kidney stones (Charig et al. 1986). Treatment A is open surgery; treatment B is percutaneous nephrolithotomy (PCNL), a keyhole operation. "Success" means the stone was removed. Here are the counts:

	small stones	large stones	all patients
A (open surgery)	81 / 87 = 93.1%	192 / 263 = 73.0%	273 / 350 = 78.0%
B (PCNL)	234 / 270 = 86.7%	55 / 80 = 68.8%	289 / 350 = 82.6%

A is better for small stones and better for large stones. Pooled, B looks better. This is Simpson's paradox: an association has one direction in every group and the opposite direction when the groups are pooled.

The reason is in the case mix. Large stones are harder to remove. Of the 350 patients who got A, 263 (75.1%) had large stones. Of the 350 patients who got B, only 80 (22.9%) had large stones. So A got the hard cases.

Interactive figure: loading…

The blue line lies above the orange line everywhere: for any fixed case mix, A is better. The pooled comparison puts A's dot far to the right (many hard cases) and B's dot far to the left. The aqua rings compare the two at the same mix, and A wins: 83.3% against 77.9%.

3. Definitions and main results #

Association and causal effect #

Let $T$ be a treatment (something we could change: a drug, a discount, a feature shown to a user) and $Y$ an outcome.

• $T$ and $Y$ are associated if the distribution of $Y$ differs between groups with different values of $T$. For a binary $T \in \{0, 1\}$, the associational difference is $P(Y = 1 \mid T = 1) - P(Y = 1 \mid T = 0)$ (or the difference in means).
• $T$ has a causal effect on $Y$ if changing $T$, with nothing else changed by us, changes the distribution of $Y$. Pearl writes the distribution of $Y$ when we set $T$ to $t$ as $P(Y \mid \mathrm{do}(T = t))$. The average causal effect of a binary treatment is $P(Y = 1 \mid \mathrm{do}(T = 1)) - P(Y = 1 \mid \mathrm{do}(T = 0))$.

"Seeing $T = 1$" and "doing $T = 1$" are different. Seeing selects a subgroup of units that chose or were given $T = 1$, with all their other differences. Doing gives $T = 1$ to everyone, whatever their other properties. Lesson 02 defines causal effects with potential outcomes; lesson 04 defines the do-operator. In this lesson the intuitive meaning is enough.

A confounder is a variable that causes both the treatment and the outcome (more precisely: it opens a non-causal path between them; lesson 03). Confounding is the difference between the association and the causal effect that such variables create. Confounding is a property of the causal structure. It is not a property of the numbers.

The pooled slope under confounding #

Proposition. In the linear model $T := aZ + \varepsilon_T$, $Y := bT + cZ + \varepsilon_Y$ with $\operatorname{Var}(Z) = \operatorname{Var}(\varepsilon_T) = 1$ and independent $Z, \varepsilon_T, \varepsilon_Y$, the slope of the population regression of $Y$ on $T$ is

Proof.

1. The slope of the least-squares line of $Y$ on $T$ is $\operatorname{Cov}(T, Y) / \operatorname{Var}(T)$ (probability lesson 13).
2. $\operatorname{Var}(T) = a^2 \operatorname{Var}(Z) + \operatorname{Var}(\varepsilon_T) = a^2 + 1$, because $Z$ and $\varepsilon_T$ are independent.
3. $\operatorname{Cov}(T, Y) = \operatorname{Cov}(T, bT + cZ + \varepsilon_Y) = b\operatorname{Var}(T) + c\operatorname{Cov}(T, Z) + 0$.
4. $\operatorname{Cov}(T, Z) = \operatorname{Cov}(aZ + \varepsilon_T, Z) = a$.
5. So $\operatorname{Cov}(T, Y) = b(a^2 + 1) + ca$. Divide by step 2. $\blacksquare$

The second term, $ca/(a^2 + 1)$, is the confounding bias. It is zero when $a = 0$ ($Z$ does not affect $T$) or $c = 0$ ($Z$ does not affect $Y$). With $a = 1$, $b = 1$, $c = 2$ the pooled slope is $1 + 2 \cdot 1 / 2 = 2$. With $b = 0$ it is $1$: a positive slope with no causal effect at all.

The regression of $Y$ on $T$ and $Z$ has coefficient exactly $b$ on $T$, because $\mathbb{E}[Y \mid T, Z] = bT + cZ$ (the noise $\varepsilon_Y$ is independent of $T$ and $Z$). This is "adjusting for $Z$". It works here because $Z$ is a confounder and is measured. Section 7 shows that adjusting for the wrong variable can create bias.

Simpson's paradox: why the pooled rate can flip #

Take a binary treatment ($A$ or $B$), a binary outcome (success or not) and a grouping variable $Z$ (small or large stone). For treatment $A$ write $a_s, a_\ell$ for the success rates in the two groups and $s_A$ for the share of large stones among A's patients. Define $b_s, b_\ell, s_B$ in the same way.

Lemma (pooled rate = weighted average). The pooled success rate of $A$ is

Proof. By the law of total probability, $P(\text{success} \mid A) = P(\text{success} \mid A, \text{small}) P(\text{small} \mid A) + P(\text{success} \mid A, \text{large}) P(\text{large} \mid A)$. The four factors are $a_s$, $1 - s_A$, $a_\ell$, $s_A$. $\blacksquare$

So the pooled rate lies on the straight line from $a_s$ (all small) to $a_\ell$ (all large). This is the line in the figure. For the kidney data: $p_A = 0.249 \cdot 0.931 + 0.751 \cdot 0.730 = 0.780$ and $p_B = 0.771 \cdot 0.867 + 0.229 \cdot 0.688 = 0.826$.

Proposition (no reversal with the same mix). If $a_s > b_s$ and $a_\ell > b_\ell$, and both treatments have the same case mix $s_A = s_B = s$, then $p_A > p_B$.

Proof. $p_A - p_B = (1 - s)(a_s - b_s) + s(a_\ell - b_\ell)$. Both differences are positive and both weights are non-negative and sum to 1, so the sum is positive. $\blacksquare$

So Simpson's reversal needs a different case mix in the two groups, and the grouping variable must affect the outcome ($a_s \ne a_\ell$). In the kidney data both hold: stone size affects the choice of treatment (doctors chose open surgery for large stones) and the success.

Standardization. To compare the treatments on the same mix, use the stone mix of all patients, $P(\text{large}) = 343 / 700 = 0.49$:

If stone size is the only common cause of treatment choice and success, these standardized rates are the success rates we would see if everyone got A, or everyone got B. This formula is the adjustment formula; lesson 04 proves when it is valid (the backdoor criterion).

The same table, two answers #

Simpson's paradox is not a puzzle about arithmetic. The arithmetic is clear. The question is which table answers the causal question: the grouped one or the pooled one? The data cannot decide. Look at two stories with exactly the same numbers:

• Story 1 (kidney stones). Stone size $Z$ is known before the treatment. It affects the choice of treatment and the success: $T \leftarrow Z \to Y$, and $T \to Y$. Stone size is a confounder. The grouped table (or the standardized rates) answers the causal question: A is better.
• Story 2 (blood pressure, Pearl). Replace "stone size" by "blood pressure after the treatment" (low or high). The drug lowers blood pressure, and lower blood pressure helps recovery: $T \to Z \to Y$, and $T \to Y$. Now $Z$ is on the causal path from $T$ to $Y$; it is a mediator. Splitting by $Z$ removes part of the drug's effect: we would compare patients whose blood pressure the drug has already changed. The pooled table answers the causal question: B is better.

A second classic: Berkeley admissions, 1973. In the graduate admissions of UC Berkeley in fall 1973, 44% of the men and 35% of the women who applied were admitted (Bickel, Hammel and O'Connell 1975). Within most departments, the gap was small or went the other way. The reason was that women applied more often to departments with low admission rates for everyone. So the pooled gap mainly reflects the choice of department. Two cautions: the department analysis answers the question "is there a bias at the department level?"; and it can have its own confounders (for example, if ability affects both the choice of department and admission). The causal question must be stated first.

Simpson's paradox with continuous variables, and Lord's paradox

The same reversal happens with slopes. Suppose older people exercise more and also have higher cholesterol. Within each age group, more exercise goes with lower cholesterol (a negative slope). Pooled over all ages, the cloud of points can have a positive slope, because the older groups sit to the right (more exercise) and higher up (more cholesterol). This is the confounding picture of the widget above, with age as $Z$: the pooled slope $b + ca/(a^2 + 1)$ can have a different sign from $b$. In the widget, set $b = -1$, $a = 1$, $c = 3$: the pooled slope is $-1 + 1.5 = 0.5 > 0$.

Lord's paradox (Lord 1967) is a related puzzle. Two dining halls; students are weighed in September and in June. Statistician 1 compares the mean weight change in the two halls and finds no difference. Statistician 2 regresses June weight on hall and September weight and finds a difference. Both analyses are correct arithmetic; they answer different causal questions. Pearl (2016) shows that a causal graph tells which question each analysis answers.

Pearl's ladder of causation #

Pearl (Pearl and Mackenzie 2018) sorts questions into three levels, the ladder of causation:

Rung	Activity	Question	Notation	Example from ML
1. Association	seeing	What does seeing $x$ tell me about $y$?	$P(y \mid x)$	a churn model ranks users by risk
2. Intervention	doing	What happens to $y$ if I set $x$?	$P(y \mid \mathrm{do}(x))$	an A/B test of a discount
3. Counterfactual	imagining	What would $y$ have been for this unit if $x$ had been different?	$P(y_x \mid x', y')$	would this user have stayed without the discount she got?

In $P(y_x \mid x', y')$, the unit really had $X = x'$ and $Y = y'$; $y_x$ is its outcome in the imagined world where $X = x$. A question on a higher rung cannot be answered from data of a lower rung alone: we need extra assumptions or extra data (an experiment, a causal graph, a full structural model).

Interactive figure: loading…

Answers to section 1. (1) Yes. This is Simpson's paradox. In the kidney data A wins in both groups (93.1% vs 86.7%, 73.0% vs 68.8%) but B wins pooled (82.6% vs 78.0%), because 75% of A's patients had large stones. (2) Positive. Hot weather is a common cause, so the pooled slope is $0 + ca/(a^2 + 1) > 0$ when heat raises both. (3) Yes, negatively. Selection on a high sum of two independent variables makes them negatively correlated among the selected (about $-0.47$ when we keep the top half of $X + Y$, about $-0.71$ for the top 10%). This is Berkson's paradox.

4. Worked example #

(a) Kidney stones, step by step. Start from the counts in section 2.

1. Within-group rates. A: $81/87 = 0.931$, $192/263 = 0.730$. B: $234/270 = 0.867$, $55/80 = 0.688$. A is better in both groups: $+6.4$ points for small stones, $+4.2$ points for large stones.
2. Case mix. Share of large stones: A $263/350 = 0.751$, B $80/350 = 0.229$.
3. Pooled rates as weighted averages: $p_A = 0.249 \cdot 0.931 + 0.751 \cdot 0.730 = 0.232 + 0.548 = 0.780$ (check: $273/350 = 0.780$); $p_B = 0.771 \cdot 0.867 + 0.229 \cdot 0.688 = 0.669 + 0.157 = 0.826$ (check: $289/350 = 0.826$). B wins by $4.6$ points: a reversal.
4. Common mix. All 700 patients: $87 + 270 = 357$ small and $263 + 80 = 343$ large, so $P(\text{small}) = 0.51$, $P(\text{large}) = 0.49$.
5. Standardized rates: $\text{std}_A = 0.51 \cdot 0.931 + 0.49 \cdot 0.730 = 0.475 + 0.358 = 0.833$; $\text{std}_B = 0.51 \cdot 0.867 + 0.49 \cdot 0.688 = 0.442 + 0.337 = 0.779$. A wins by $5.4$ points.
6. Decision. If stone size is the main common cause of choice and success (story 1), recommend A. The pooled comparison mixes the effect of the treatment with the effect of stone size.

(b) The pooled slope. Take $a = 1$, $b = 0$, $c = 2$: $T := Z + \varepsilon_T$, $Y := 2Z + \varepsilon_Y$. $T$ has no effect on $Y$.

1. $\operatorname{Var}(T) = 1^2 \cdot 1 + 1 = 2$.
2. $\operatorname{Cov}(T, Y) = \operatorname{Cov}(Z + \varepsilon_T, 2Z + \varepsilon_Y) = 2\operatorname{Var}(Z) = 2$.
3. Pooled slope $= 2 / 2 = 1$. Causal effect $= 0$. The whole slope is confounding bias: $ca/(a^2 + 1) = 2 \cdot 1 / 2 = 1$.
4. Correlation: $\operatorname{Var}(Y) = 4 + 1 = 5$, so $r = 2 / \sqrt{2 \cdot 5} = 0.63$. A strong correlation with zero causal effect.

5. In code #

NumPy checks both parts of the worked example. The notebook for this lesson has these cells and the exercises.

import numpy as np

# Kidney stones (Charig et al. 1986): successes and patients
#                small      large
succ = np.array([[ 81, 192],    # A: open surgery
                 [234,  55]])   # B: PCNL
n    = np.array([[ 87, 263],
                 [270,  80]])
within = succ / n
pooled = succ.sum(1) / n.sum(1)
p_large = n[:, 1].sum() / n.sum()               # stone mix of all 700 patients
standard = within[:, 0] * (1 - p_large) + within[:, 1] * p_large
print(within.round(3))      # [[0.931 0.73 ] [0.867 0.688]]
print(pooled.round(3))      # [0.78  0.826]
print(standard.round(3))    # [0.833 0.779]

# Confounding: Z -> T, Z -> Y, T -> Y with a true effect of 1
rng = np.random.default_rng(0)
N = 100_000
Z = rng.normal(size=N)
T = Z + rng.normal(size=N)
Y = 1.0 * T + 2.0 * Z + rng.normal(size=N)
pooled_slope = np.polyfit(T, Y, 1)[0]
X = np.column_stack([np.ones(N), T, Z])
adjusted = np.linalg.lstsq(X, Y, rcond=None)[0][1]
print(round(pooled_slope, 2), round(adjusted, 2))   # 2.0 1.0

The simulation knows the true effect (1), so we can check each estimate against the truth. Every lesson of this track works this way.

6. Where it shows up #

• Healthcare models. The pneumonia model of Caruana et al. (2015) learned "asthma lowers the risk of death" from a real association created by treatment policy. Interpretable models made the problem visible. Lesson 11 returns to such shortcut features.
• A/B tests and observational data in industry. Product teams see that "users who use feature F stay longer". Active users both use F and stay (a common cause). An experiment that turns F on at random answers the causal question (lesson 02).
• Recommenders and feedback loops. Logged clicks depend on what the system chose to show. A model trained on these logs learns the old policy's choices as well as user taste (lesson 11: off-policy evaluation).
• Selection bias in datasets. Hospital data contain only admitted patients; this can create associations between diseases that do not exist in the population (Berkson 1946). Datasets of "published papers" or "approved loans" have the same problem.
• Science. Epidemiology, economics and climate attribution all ask rung-2 questions from data that were not randomized. ML for science models (for example, emulators of physical systems) meet the same issue when they are used to predict the result of an intervention.

7. Common mistakes #

• "Correlation implies causation." A correlation can come from causation, reverse causation, a common cause, selection or chance. Check: write down the possible causal graphs before you interpret an association.
• "Correlation does not imply causation, so we can learn nothing." Also wrong. With an experiment, or with stated assumptions (a causal graph) and the right adjustment, data do answer causal questions. That is what this track teaches.
• "Always control for every variable." Splitting by a mediator (story 2) or by a common effect (Berkson) creates bias. Check: only adjust for variables measured before the treatment that are not caused by it; lessons 03–04 give the exact rule.
• "The pooled table is always right" or "the grouped table is always right". Neither. The causal story decides. Check: ask whether the grouping variable is a cause of the treatment (use groups) or an effect of it (use the pooled data).
• Forgetting selection. If your data contain only selected units (admitted, hired, published, clicked), an association may come from the selection. Check: ask how a unit got into the data.
• Forgetting multiple comparisons. Among 1,000 pairs of unrelated series of length 20, about 50 will have $|r| > 0.444$ (the two-sided 5% critical value for $n = 20$). Check: count how many pairs were tested, and test the finding again on new data.

8. Check yourself #

1. Give an example of reverse causation from your own work or field.
   Answer

   Many answers are possible. Example: "users who see more ads buy more" when the ad system shows more ads to users who are already about to buy. Purchases (or the intent to buy) cause the ads, not the other way around.
2. In the model $T := 2Z + \varepsilon_T$, $Y := 0.5\,T - Z + \varepsilon_Y$ (unit-variance, independent $Z, \varepsilon_T, \varepsilon_Y$), compute the pooled regression slope of $Y$ on $T$.
   Answer

   $b + ca/(a^2 + 1) = 0.5 + (-1)(2)/(4 + 1) = 0.5 - 0.4 = 0.1$. The confounder hides most of the positive effect.
3. Treatment A: 90 successes in 100 easy cases and 30 in 100 hard cases. Treatment B: 160 successes in 200 easy cases, and B was never used on hard cases. Compute the pooled rates. Which treatment is better for easy cases?
   Answer

   Pooled A $= 120/200 = 60\%$, pooled B $= 160/200 = 80\%$. For easy cases A is better (90% vs 80%). B was never used on hard cases, so the data say nothing about B for hard cases: we cannot standardize B to a mix that contains hard cases. (This lack of overlap is called a positivity problem; lesson 02.)
4. Why can a reversal not happen when both treatments have the same case mix?
   Answer

   Then $p_A - p_B = (1 - s)(a_s - b_s) + s(a_\ell - b_\ell)$, a weighted average of the within-group differences with the same weights. If both differences are positive, the average is positive.
5. A dataset contains only students admitted to a selective university, which admits on grades plus test score. Grades and test score are negatively correlated in the dataset. Does this mean that good grades hurt test scores?
   Answer

   No. Admission is a common effect of grades and test score, and the data are selected on it (Berkson's paradox). A student with low grades was admitted only with a high test score, and the other way around. This creates a negative correlation even if the two are unrelated, or positively related, in all applicants.
6. Which rung of the ladder is "If we raise the price by 5% for all users, how many will cancel?"
   Answer

   Rung 2 (intervention): $P(\text{cancel} \mid \mathrm{do}(\text{price} = 1.05\,p))$. It is about the average result of an action in the population, not about one user's past.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. What is the difference between an association and a causal effect of $T$ on $Y$?

   Show answer

2. Name the five sources of an association between $X$ and $Y$.

   Show answer

3. What is a confounder?

   Show answer

4. In $T := aZ + \varepsilon_T$, $Y := bT + cZ + \varepsilon_Y$ (unit variances), what is the pooled slope of $Y$ on $T$?

   Show answer

5. What is Simpson's paradox?

   Show answer

6. Why does B look better than A in the pooled kidney-stone data although A is better for both stone sizes?

   Show answer

7. When is the pooled table the right one to answer a causal question, even if group tables exist?

   Show answer

8. State Berkson's paradox in one sentence.

   Show answer

9. Give the three rungs of Pearl's ladder with their notation.

   Show answer

10. What does Reichenbach's common cause principle say?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Correlation is not causation. Explain with an example." — Give the five sources, with one example each; say that only an experiment or explicit causal assumptions separate them.
• "Explain Simpson's paradox." — Use the kidney-stone numbers. Pooled rates are weighted averages; the case mix differs; standardization fixes the comparison if the grouping variable is a confounder. Then add the blood-pressure story: the causal graph decides.
• "Users who use feature F have higher retention. Should we push F to everyone?" — Not from this data alone. Engaged users both use F and stay (common cause). Run an experiment (randomize F or an encouragement to use it), or adjust for pre-treatment engagement with care. Link interview lesson 09.
• "Your model says asthma lowers pneumonia mortality. What happened?" — The data reflect treatment policy: asthma patients get intensive care. The model is fine for prediction under the same policy and wrong for decisions.

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution:

1. simpson_table(succ, n): within-group, pooled and standardized rates, and a flag for a reversal; check it on the kidney-stone data.
2. simulate(kind, n, seed) for the five sources of association; check that each gives $|r| \approx 0.5$ and see which one disappears with more data.
3. Confounding: pooled slope, slope within thin bins of $Z$, and the regression on $(T, Z)$, against the true effect.
4. Berkson: simulate talent and looks, keep the top 10% of the sum, and compute the correlation (about $-0.71$).
5. Multiple comparisons: 1,000 pairs of unrelated series of length 20; count how many have $|r| > 0.444$ (about 5%).

11. Watch and read #

Simpson's Paradoxminutephysics · 5 min

The kidney-stone example with pictures; it ends with the point that the causal story decides.

Are University Admissions Biased? | Simpson's Paradox Part 2minutephysics · 4 min

The Berkeley admissions data and why the department level changes the picture.

Correlation vs. Causation: Causal Inference BootcampMod•U: Powerful Concepts in Social Science · 7 min

A short overview of why correlation is not causation, with confounding as the main reason.

1 - A Brief Introduction to Causal Inference (Course Preview)Brady Neal - Causal Inference · 42 min

The first lecture of a full course; it covers Simpson's paradox and the idea of a causal graph.

• Judea Pearl, Madelyn Glymour and Nicholas Jewell, Causal Inference in Statistics: A Primer (Wiley 2016), chapter 1 (Simpson's paradox, the blood-pressure story); companion site: bayes.cs.ucla.edu/PRIMER.
• Miguel Hernán and James Robins, Causal Inference: What If, free book, chapter 1.
• Judea Pearl, "Understanding Simpson's paradox" (The American Statistician, 2014), free PDF.
• Judea Pearl, "The seven tools of causal inference, with reflections on machine learning" (Communications of the ACM, 2019): the ladder of causation, free PDF.
• Brady Neal, Introduction to Causal Inference, course notes and lectures, chapter 1.
• Rich Caruana et al., "Intelligible models for healthcare" (KDD 2015): the asthma example.
• Original papers (not free): C. R. Charig et al., "Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy", BMJ 1986; P. J. Bickel, E. A. Hammel and J. W. O'Connell, "Sex bias in graduate admissions: data from Berkeley", Science 1975.