Hypothesis testing: p-values, power, t-tests and multiple testing

Before this lesson: lesson 01 (Bayes' rule and base rates), lesson 02 (the probability integral transform), lesson 06 (the CLT, the t-distribution and confidence intervals). Lesson 10 helps for the contrast with credible intervals. Time: about 80 minutes, plus the notebook.

---

Why this matters #

• Comparing models. "Model B beats model A on the test set" is a claim about noise. Seeds, test examples and data splits all add randomness. A paired test or an interval tells you whether the gain is larger than this noise.
• A/B tests. Every change to a product with many users (a new ranking model, a new layout, a new LLM prompt) is launched or rejected after a two-proportion z-test or a t-test. Lesson 12 turns this into an engineering process.
• Monitoring. Drift detection tests hundreds of input features every day. Without a multiple-testing correction, most alarms are false.
• Science. Genomics tests thousands of genes, climate science tests a trend in every grid cell, and particle physics uses the "5σ" rule. All of them are multiple-testing problems.
• Reading papers. Many results in ML papers are a mean over 3–5 seeds. To read them well, you need to know what a p-value and a confidence interval say, and what they do not say.

1. Predict first #

2. The picture #

A test in four steps. You flip a coin $n = 100$ times and see $S = 61$ heads. Is the coin fair?

1. Null hypothesis $H_0$: the default model, which you try to reject. Here $H_0: p = 1/2$, where $p$ is the probability of heads. The alternative hypothesis is $H_1: p \ne 1/2$.
2. Test statistic: a number that measures how far the data are from $H_0$. Under $H_0$, $\mathbb{E}[S] = 50$ and $\operatorname{SD}(S) = \sqrt{100 \cdot 0.5 \cdot 0.5} = 5$. So use the standardised count $z = (S - 50)/5 = 11/5 = 2.2$.
3. Null distribution: the distribution of the statistic when $H_0$ is true. Here $S \sim \text{Bin}(100, 1/2)$, and by the CLT (lesson 06) $z$ is approximately $\mathcal N(0, 1)$.
4. p-value: the probability, under $H_0$, of a statistic at least as extreme as the observed one. "Extreme" means far from 50 in either direction (two-sided): $p = P(|S - 50| \ge 11) = 2P(S \ge 61) = 0.0352$ with the exact binomial distribution. The CLT gives $2\Phi(-2.2) = 0.028$, and $0.036$ with the continuity correction ($z = 10.5/5 = 2.1$).

Fix the level $\alpha = 0.05$ before you see the data. Since $p \le \alpha$, you reject $H_0$: the result is "statistically significant at the 5% level".

Interactive figure: loading…

Two kinds of errors. The truth is unknown, and the test decides. There are four cases:

	$H_0$ true	$H_0$ false
reject $H_0$	type I error (false positive), probability $\le \alpha$	correct, probability = power
do not reject	correct	type II error (false negative), probability $\beta_{\text{II}}$

The power is $1 - \beta_{\text{II}}$: the probability that the test finds an effect that is really there. We write $\beta_{\text{II}}$ (not $\beta$) because $\beta$ has other meanings in this track.

Power as two bell curves. Draw the distribution of $z$ under $H_0$: a bell centred at 0. The test rejects when $|z| > 1.96$; the two tails beyond $\pm 1.96$ have area $\alpha = 0.05$. Now suppose the coin really has $p = 0.6$. Then $z$ is centred at $(60 - 50)/5 = 2$, with standard deviation $\sqrt{100 \cdot 0.6 \cdot 0.4}/5 = 0.98$, about 1. Only about half of this second bell lies beyond 1.96. So the power is about one half (0.52 with the normal approximation): half of the time, 100 flips miss a coin that lands heads 60% of the time. With $n = 400$ flips the centre moves to $(240 - 200)/10 = 4$, and the power is about 0.98. The effect is the same; more data make it visible.

Interactive figure: loading…

3. Definitions and main results #

Tests, errors and power #

Data $X$ come from a model $P_\theta$ with an unknown parameter $\theta$. The two hypotheses split the set of parameters: $H_0: \theta \in \Theta_0$ against $H_1: \theta \in \Theta_1$. A test is a rule $\psi(X) \in \{0, 1\}$, where 1 means "reject $H_0$". Usually $\psi = \mathbf 1\{T > c\}$ for a test statistic $T$ and a critical value $c$.

• Type I error rate: $P_\theta(\psi = 1)$ for $\theta \in \Theta_0$. The test has level $\alpha$ if this is at most $\alpha$ for every $\theta \in \Theta_0$. Its size is the largest type I error rate, $\sup_{\theta \in \Theta_0} P_\theta(\psi = 1)$.
• Type II error rate: $\beta_{\text{II}}(\theta) = P_\theta(\psi = 0)$ for $\theta \in \Theta_1$.
• Power function: $\theta \mapsto P_\theta(\psi = 1)$. On $\Theta_1$ it equals $1 - \beta_{\text{II}}(\theta)$. Power is a function of the true $\theta$, not one number.

Power of the z-test. Take $n$ i.i.d. observations with unknown mean $\mu$ and known standard deviation $\sigma$. Test $H_0: \mu = \mu_0$ with $T = \sqrt n(\bar X_n - \mu_0)/\sigma$. Let $z_{1-\alpha}$ be the $(1-\alpha)$-quantile of $\mathcal N(0, 1)$, so $z_{0.95} = 1.645$ and $z_{0.975} = 1.960$.

• One-sided ($H_1: \mu > \mu_0$, reject when $T > z_{1-\alpha}$). If the true mean is $\mu_0 + \delta$, then $T \sim \mathcal N(\lambda, 1)$ with $\lambda = \sqrt n\,\delta/\sigma$, so

• Two-sided (reject when $|T| > z_{1-\alpha/2}$): $\text{power}(\delta) = \Phi(\lambda - z_{1-\alpha/2}) + \Phi(-\lambda - z_{1-\alpha/2})$. The second term (rejecting in the wrong direction) is tiny unless $\delta$ is close to 0.

Power grows with the effect $\delta$, with $n$ and with $\alpha$, and falls with the noise $\sigma$. Only the combination $\lambda = \sqrt n\,\delta/\sigma$ matters. Lesson 12 solves this formula for $n$ to plan an experiment.

The p-value #

Definition. Let large values of $T$ count as evidence against $H_0$, and let $t$ be the observed value. The p-value is

the probability, computed under $H_0$, of a statistic at least as extreme as the observed one. For a two-sided test with a statistic that is symmetric under $H_0$, use $|T|$: $p = P_{H_0}(|T| \ge |t|)$. Equivalently, the p-value is the smallest level $\alpha$ at which the test rejects. So $p \le \alpha$ exactly when the level-$\alpha$ test rejects.

Theorem (p-values are uniform under $H_0$). If $H_0$ fixes the distribution of $T$, and its CDF $F$ is continuous, then $p \sim \text{Uniform}(0, 1)$ under $H_0$. Hence $P_{H_0}(p \le \alpha) = \alpha$ for every $\alpha$.

Proof.

1. For a continuous $F$, $P_{H_0}(T \ge t) = 1 - F(t)$, so the p-value is the random variable $p = 1 - F(T)$.
2. By the probability integral transform (lesson 02), $F(T) \sim \text{Uniform}(0, 1)$ when $F$ is continuous.
3. If $U \sim \text{Uniform}(0, 1)$, then $1 - U \sim \text{Uniform}(0, 1)$. $\blacksquare$

For a discrete statistic (counts, permutations), $P_{H_0}(p \le \alpha) \le \alpha$: the test is conservative. The exact coin test above rejects only if $S \ge 61$ or $S \le 39$. This region has probability $0.035$ under $H_0$, not $0.05$, because no region of exact size $0.05$ exists.

What a p-value is not.

• Not $P(H_0 \mid \text{data})$. That probability needs a prior, by Bayes' rule (lesson 01). Suppose 10% of the ideas that a team tests are real effects, its tests have power 0.8, and $\alpha = 0.05$. Out of 1,000 ideas, 900 are null and give $900 \cdot 0.05 = 45$ false positives; 100 are real and give $100 \cdot 0.8 = 80$ true positives. So $45/(45 + 80) = 36\%$ of the significant results are false. This is the same base-rate effect as the medical test of lesson 01.
• Not an effect size. A test statistic is roughly $\sqrt n \times$ (effect / noise). An effect of $0.01\sigma$ with $n = 10^6$ gives $z = 10$ and $p \approx 10^{-23}$. Report the estimate and a confidence interval, not only $p$.
• Not evidence for $H_0$ when it is large. $p = 0.4$ can mean "no effect" or "too little data". Only the power, or the width of the confidence interval, tells you which.

The t-tests #

Two distributions appear here. Both were introduced in lesson 06.

• The χ² distribution with $k$ degrees of freedom, $\chi^2_k$, is the distribution of $Z_1^2 + \dots + Z_k^2$ for i.i.d. $Z_i \sim \mathcal N(0, 1)$. It is $\text{Gamma}(k/2, 1/2)$ in the rate parametrisation, with mean $k$.
• The Student t distribution with $\nu$ degrees of freedom, $t_\nu$, is the distribution of $Z/\sqrt{V/\nu}$ for independent $Z \sim \mathcal N(0, 1)$ and $V \sim \chi^2_\nu$. It has heavier tails than $\mathcal N(0, 1)$, and it tends to $\mathcal N(0, 1)$ as $\nu \to \infty$. Its quantiles are larger: $t_{9,\,0.975} = 2.262$, $t_{30,\,0.975} = 2.042$, while $z_{0.975} = 1.960$.

Interactive figure: loading…

One-sample t-test. Data $X_1, \dots, X_n$ i.i.d. $\mathcal N(\mu, \sigma^2)$, with $\sigma$ unknown. Let $\bar X_n$ be the sample mean and $S^2 = \frac{1}{n-1}\sum_i (X_i - \bar X_n)^2$ the sample variance. To test $H_0: \mu = \mu_0$ use

Why $t_{n-1}$. For Gaussian data, $\bar X_n$ and $S^2$ are independent, and $(n-1)S^2/\sigma^2 \sim \chi^2_{n-1}$ (Cochran's theorem). So $T$ is the standard normal $\sqrt n(\bar X_n - \mu_0)/\sigma$ divided by $\sqrt{\chi^2_{n-1}/(n-1)}$, which is the definition of $t_{n-1}$. For non-Gaussian data with finite variance, $T \to \mathcal N(0, 1)$ by the CLT and Slutsky's lemma, so the test is still approximately valid for large $n$. For small, skewed samples it is not.

Paired data. If $x_i$ and $y_i$ belong together (the same student before and after a course, the same test example scored by two models), compute the differences $d_i = x_i - y_i$ and run the one-sample t-test on them. Pairing removes the variation between the pairs, which is often most of the variation.

Two independent samples. Samples $x$ (size $n_x$) and $y$ (size $n_y$), with sample means $\bar x, \bar y$ and sample variances $s_x^2, s_y^2$. Test $H_0: \mu_x = \mu_y$.

• Student's t-test (pooled). Assume equal variances. Pool them, $s_p^2 = \frac{(n_x - 1)s_x^2 + (n_y - 1)s_y^2}{n_x + n_y - 2}$, and use $T = \frac{\bar x - \bar y}{s_p\sqrt{1/n_x + 1/n_y}} \sim t_{n_x + n_y - 2}$.
• Welch's t-test. Do not assume equal variances. Use

which is approximately $t_\nu$ with the Welch–Satterthwaite degrees of freedom

Where the Welch–Satterthwaite $\nu$ comes from

Let $W = s_x^2/n_x + s_y^2/n_y$, the estimated variance of $\bar x - \bar y$. The idea: find the $\chi^2$ distribution that has the same mean and variance as $W$.

1. For Gaussian data, $\operatorname{Var}(s^2) = 2\sigma^4/(n - 1)$. So $\mathbb{E}[W] = \sigma_x^2/n_x + \sigma_y^2/n_y$ and $\operatorname{Var}(W) = \frac{2\sigma_x^4}{n_x^2(n_x - 1)} + \frac{2\sigma_y^4}{n_y^2(n_y - 1)}$.
2. In the one-sample case, $s^2$ is exactly $\sigma^2\chi^2_{n-1}/(n-1)$. So approximate $W$ by $\mathbb{E}[W]\,\chi^2_\nu/\nu$. This has mean $\mathbb{E}[W]$ and variance $2\,\mathbb{E}[W]^2/\nu$.
3. Set the two variances equal and solve for $\nu$: $\nu = 2\,\mathbb{E}[W]^2/\operatorname{Var}(W)$. Replace each $\sigma^2$ by its estimate $s^2$. This gives the formula above.

$\nu$ always lies between $\min(n_x, n_y) - 1$ and $n_x + n_y - 2$. It is close to the lower end when one group has most of the uncertainty.

Which one to use. When $n_x = n_y$, the two statistics are identical (only the degrees of freedom differ), so the pooled test is robust. When the sizes differ, the pooled variance weights each $s^2$ by its sample size. If the small group has the large variance, $s_p^2$ is too small, $|T|$ is too large, and the pooled test rejects far too often: in the setting of section 4, 22% of the time instead of 5%. Welch loses almost nothing when the variances are equal. Use Welch by default.

Two proportions and the χ² tests #

Two-proportion z-test. Group 1 has $x_1$ successes in $n_1$ trials, group 2 has $x_2$ in $n_2$. Let $\hat p_j = x_j/n_j$. Under $H_0: p_1 = p_2 = p$, the difference $\hat p_2 - \hat p_1$ has variance $p(1 - p)(1/n_1 + 1/n_2)$. Estimate the common $p$ by the pooled rate $\hat p = (x_1 + x_2)/(n_1 + n_2)$:

The two-sided p-value is $2\Phi(-|z|)$.

Pearson's χ² test. Counts $O_1, \dots, O_K$ in $K$ categories, with expected counts $E_j$ under $H_0$. The statistic is

• Goodness of fit. $H_0$ fixes the cell probabilities $\pi_j$ (for a fair die, $\pi_j = 1/6$), so $E_j = n\pi_j$. Under $H_0$, $X^2 \to \chi^2_{K-1}$ as $n \to \infty$. Why $K - 1$: the counts are approximately Gaussian (CLT), $X^2$ is a squared standardised distance, and the constraint $\sum_j O_j = n$ removes one dimension. Each parameter that you estimate from the same data removes one more.
• Independence in an $r \times c$ table. $E_{ij} = (\text{row } i \text{ total}) \cdot (\text{column } j \text{ total})/n$, and $X^2 \to \chi^2_{(r-1)(c-1)}$.
• Rule of thumb: all $E_j \ge 5$. Otherwise use an exact test (Fisher's exact test for a $2 \times 2$ table) or a simulated null distribution.

For a $2 \times 2$ table, Pearson's $X^2$ equals $z^2$ of the pooled two-proportion test, and $\chi^2_1$ is the distribution of $Z^2$. So the two tests give the same two-sided p-value. Yates' continuity correction subtracts $0.5$ from each $|O - E|$ in a $2 \times 2$ table; it makes the test more conservative, and SciPy's chi2_contingency applies it by default.

Confidence intervals and duality #

Definition. A $(1 - \alpha)$ confidence interval is a random interval $C(X)$ with $P_\theta(\theta \in C(X)) \ge 1 - \alpha$ for every $\theta$. This probability is the coverage (lesson 06 checks it by simulation).

95% coverage is a property of the procedure, before you see the data: if you repeat the study many times, about 95% of the intervals contain the fixed true $\theta$. After you compute one interval, $\theta$ is either in it or not. The sentence "$\theta$ lies in this interval with probability 0.95" needs a prior; it describes a credible interval (lesson 10).

Theorem (duality of tests and intervals). For each value $\theta_0$, let $\psi_{\theta_0}$ be a level-$\alpha$ test of $H_0: \theta = \theta_0$. Then the set of values that the test does not reject, $C(X) = \{\theta_0 : \psi_{\theta_0}(X) = 0\}$, is a $(1 - \alpha)$ confidence set.

Proof.

1. Fix the true $\theta$. By the definition of $C$, $\theta \in C(X)$ exactly when $\psi_\theta(X) = 0$.
2. So $P_\theta(\theta \in C(X)) = 1 - P_\theta(\psi_\theta(X) = 1)$.
3. $\psi_\theta$ tests a true null hypothesis, so $P_\theta(\psi_\theta(X) = 1) \le \alpha$. $\blacksquare$

The converse also holds: "reject $\theta_0$ when $\theta_0 \notin C(X)$" is a level-$\alpha$ test. So a level-0.05 test of "no difference" rejects exactly when the matching 95% interval excludes 0. The matching intervals are:

Test	$(1 - \alpha)$ interval
one-sample or paired t	$\bar x \pm t_{n-1,\,1-\alpha/2}\; s/\sqrt n$
Student (pooled)	$\bar x - \bar y \pm t_{n_x+n_y-2,\,1-\alpha/2}\; s_p\sqrt{1/n_x + 1/n_y}$
Welch	$\bar x - \bar y \pm t_{\nu,\,1-\alpha/2}\sqrt{s_x^2/n_x + s_y^2/n_y}$
two proportions	$\hat p_2 - \hat p_1 \pm z_{1-\alpha/2}\sqrt{\hat p_1(1-\hat p_1)/n_1 + \hat p_2(1-\hat p_2)/n_2}$

The proportion interval uses the unpooled standard error, because no null hypothesis says $p_1 = p_2$ there. So it is not the exact dual of the pooled z-test: when $p$ is close to $\alpha$, the test and the interval can disagree.

Multiple testing #

You test $m$ hypotheses, and $m_0$ of the null hypotheses are true. A procedure makes $R$ rejections ("discoveries"), and $V$ of them are false (true nulls that were rejected).

• Family-wise error rate: $\text{FWER} = P(V \ge 1)$, the probability of at least one false discovery.
• False discovery proportion: $\text{FDP} = V/\max(R, 1)$; the $\max$ sets $\text{FDP} = 0$ when $R = 0$.
• False discovery rate: $\text{FDR} = \mathbb{E}[\text{FDP}]$, the average FDP over repeated studies.

Since $V/\max(R, 1) \le \mathbf 1\{V \ge 1\}$, always $\text{FDR} \le \text{FWER}$. Without any correction, $m$ independent true nulls tested at level $\alpha$ give $\text{FWER} = 1 - (1 - \alpha)^m$: 0.40 for $m = 10$ and 0.64 for $m = 20$.

Theorem (Bonferroni). Reject $H_i$ if $p_i \le \alpha/m$. Then $\text{FWER} \le \alpha$, under any dependence between the tests.

Proof. By the union bound over the true nulls, $P(V \ge 1) \le \sum_{i \text{ null}} P(p_i \le \alpha/m) \le m_0\,\alpha/m \le \alpha$. $\blacksquare$

Holm's method is a step-down version that is never less powerful and also controls the FWER: sort the p-values, compare $p_{(i)}$ with $\alpha/(m - i + 1)$ for $i = 1, 2, \dots$, and stop at the first failure.

The Benjamini–Hochberg (BH) procedure at level $q$.

1. Sort the p-values: $p_{(1)} \le \dots \le p_{(m)}$.
2. Find the largest $k$ with $p_{(k)} \le kq/m$.
3. Reject the $k$ hypotheses with the smallest p-values (none if no such $k$ exists).

BH is a step-up rule: a p-value above its own line $iq/m$ is still rejected if a larger p-value lies below its line. The BH-adjusted p-values $\tilde p_{(i)} = \min_{j \ge i}\min(1, m\,p_{(j)}/j)$ give the same decision: reject $H_i$ when $\tilde p_i \le q$.

Interactive figure: loading…

Theorem (BH controls the FDR). If the null p-values are independent of each other and of the non-null p-values, and each null p-value is $\text{Uniform}(0, 1)$, then $\text{FDR} = q\,m_0/m \le q$.

Proof that BH controls the FDR (independent p-values)

Write $V_i = \mathbf 1\{H_i \text{ rejected}\} = \mathbf 1\{p_i \le Rq/m\}$, so $\text{FDP} = \sum_{i \text{ null}} V_i/\max(R, 1)$. Let $R^{(i)}$ be the number of BH rejections when $p_i$ is replaced by 0. It depends only on the other p-values, and $R^{(i)} \ge 1$.

1. Split by the value of $R$: for a true null $i$, $\mathbb{E}[V_i/\max(R, 1)] = \sum_{k=1}^m \frac1k P(p_i \le kq/m,\ R = k)$.
2. On the event $\{p_i \le kq/m\}$, $R = k$ exactly when $R^{(i)} = k$. If $R = k$, then $p_i$ is among the $k$ smallest; moving it to 0 keeps the same $k$ smallest values and changes no larger order statistic. Conversely, if $R^{(i)} = k$, putting back a $p_i \le kq/m$ keeps $k$ values below $kq/m$ and can only raise the larger order statistics.
3. By independence and uniformity, $P(p_i \le kq/m,\ R^{(i)} = k) = P(p_i \le kq/m)\,P(R^{(i)} = k) = \frac{kq}{m} P(R^{(i)} = k)$.
4. So $\mathbb{E}[V_i/\max(R, 1)] = \frac qm \sum_k P(R^{(i)} = k) = \frac qm$.
5. Summing over the $m_0$ true nulls gives $\text{FDR} = m_0 q/m$. $\blacksquare$

When BH controls the FDR under dependence

• Independent p-values: yes, at exactly $q\,m_0/m$.
• A specific kind of positive dependence (PRDS, positive regression dependence on the subset of nulls; Benjamini and Yekutieli, 2001): yes, $\text{FDR} \le q\,m_0/m$. One-sided tests from jointly Gaussian statistics with nonnegative correlations are PRDS. Two-sided p-values of such statistics are in general not PRDS; BH usually works well there in practice, but the theorem does not guarantee it.
• Arbitrary dependence: not guaranteed. The Benjamini–Yekutieli procedure runs BH at level $q/\sum_{i=1}^m 1/i$ and controls the FDR in every case.
• FDR control is a statement about the average FDP over repeated studies. In one study the FDP can be well above $q$.

FWER or FDR? Control the FWER when one false claim is costly: a confirmatory trial, a safety claim, a few planned comparisons. Control the FDR when you screen many hypotheses and follow up the hits: genes, features, grid cells, many metrics. Then a known small share of false leads is acceptable, and the FDR keeps much more power.

p-hacking and the garden of forking paths. A p-value is valid only for a test that was fixed before you saw the data. If you try several metrics, subgroups, outlier rules or tests and report the one with $p < 0.05$, you have run many tests but corrected for none. This is p-hacking. Even without bad intent, an analysis that would have been different for different data has the same problem (the "garden of forking paths"). The fix: write down the hypothesis, the metric, the test and $\alpha$ before you look, and treat everything else as exploration.

Answers to section 1. (1) Only (b). (a) is wrong because $P(H_0 \mid \text{data})$ needs a prior: with 10% real effects and power 0.8, about 36% of the results with $p \le 0.05$ are true nulls. (c) is wrong because $p$ mixes the effect size with $n$. (2) $1 - 0.95^{20} = 0.64$. Bonferroni at $0.05/20 = 0.0025$ brings this back below 0.05. (3) About half the time: the power is about 0.5 (section 2).

4. Worked example #

A. Welch vs Student: a new course. A school tests a new online course. The score gain (after minus before, in points) of each student:

course	$n$	mean gain	$s$
new	8	6.2	5.0
old	30	3.0	2.0

Do the two courses give the same mean gain?

• Welch. Step 1, the two variance terms: $s_x^2/n_x = 25/8 = 3.125$ and $s_y^2/n_y = 4/30 = 0.1333$. Step 2, the standard error: $\sqrt{3.125 + 0.1333} = \sqrt{3.2583} = 1.805$. Step 3, the statistic: $t = (6.2 - 3.0)/1.805 = 1.773$. Step 4, the degrees of freedom: $\nu = 3.2583^2/(3.125^2/7 + 0.1333^2/29) = 10.617/1.3957 = 7.6$. Step 5, the p-value: two-sided $p = 2P(t_{7.6} > 1.773) = 0.116$. Not significant at 5%. Step 6, the interval: with $t_{7.6,\,0.975} = 2.327$, it is $3.2 \pm 2.327 \cdot 1.805 = 3.2 \pm 4.2 = [-1.0, 7.4]$. It contains 0, as duality requires.
• Student (pooled). $s_p^2 = (7 \cdot 25 + 29 \cdot 4)/36 = 291/36 = 8.08$, and $\operatorname{SE} = \sqrt{8.08 \cdot (1/8 + 1/30)} = 1.131$. So $t = 3.2/1.131 = 2.829$ with 36 degrees of freedom, and $p = 0.0076$.

The two tests disagree: "not significant" against "significant at 1%". The pooled SE is too small. It is dominated by the 30 old-course students with a small spread, but the uncertainty of the difference comes mostly from the 8 noisy new-course students. Trust Welch. A simulation with these sizes and spreads, and equal true means, shows the pooled test rejecting 22% of the time at $\alpha = 0.05$ (section 5).

B. Two proportions: two sign-up pages. Page A is shown to 400 visitors and 18 sign up ($\hat p_1 = 0.045$). Page B is shown to 400 other visitors and 33 sign up ($\hat p_2 = 0.0825$).

• Pooled rate: $\hat p = 51/800 = 0.06375$.
• Standard error under $H_0$: $\operatorname{SE}_0 = \sqrt{0.06375 \cdot 0.93625 \cdot (1/400 + 1/400)} = 0.01728$.
• Statistic: $z = (0.0825 - 0.045)/0.01728 = 0.0375/0.01728 = 2.171$, so $p = 2\Phi(-2.171) = 0.030$. Significant at 5%.
• 95% interval with the unpooled SE $\sqrt{0.045 \cdot 0.955/400 + 0.0825 \cdot 0.9175/400} = 0.01722$: $0.0375 \pm 1.96 \cdot 0.01722 = [0.004, 0.071]$, that is 0.4 to 7.1 percentage points.
• The χ² test of the $2 \times 2$ table (rows: page; columns: signed up or not) gives $X^2 = 4.712 = 2.171^2$ and the same $p = 0.030$. With Yates' correction, $X^2 = 4.105$ and $p = 0.043$.

C. χ² goodness of fit: is a weather ensemble calibrated? An ensemble forecast runs a weather model 9 times with slightly different starting states. For each of 200 forecast cases, rank the observation among the 9 members; there are 10 possible ranks. If the ensemble is calibrated, the observation behaves like one more member, so each rank has probability 1/10 and expected count 20. The observed rank histogram:

rank	1	2	3	4	5	6	7	8	9	10
count	31	22	17	15	14	13	16	18	21	33

$X^2 = \frac{1}{20}(11^2 + 2^2 + 3^2 + 5^2 + 6^2 + 7^2 + 4^2 + 2^2 + 1^2 + 13^2) = 434/20 = 21.7$. With $10 - 1 = 9$ degrees of freedom, $p = 0.0099$; the 5% critical value is 16.9. The U shape (too many observations outside the ensemble's range) says that the ensemble is underdispersed: its spread is too small. One caveat: the test assumes independent cases. Forecasts on consecutive days are correlated, which makes $X^2$ too large.

D. Multiple testing: drift in 8 features. A team checks whether each of 8 input features of a model has drifted between the training data and this month's data. Sorted p-values, and the BH lines $iq/m = 0.00625\,i$ at $q = 0.05$:

$i$	1	2	3	4	5	6	7	8
$p_{(i)}$	0.0004	0.0021	0.0075	0.026	0.031	0.18	0.44	0.87
$0.00625\,i$	0.00625	0.0125	0.01875	0.025	0.03125	0.0375	0.04375	0.05
below the line?	yes	yes	yes	no	yes	no	no	no

• Bonferroni: the threshold is $0.05/8 = 0.00625$. It rejects 2.
• Holm: $0.0004 \le 0.05/8$, $0.0021 \le 0.05/7$, $0.0075 \le 0.05/6 = 0.0083$, then $0.026 > 0.05/5 = 0.01$: stop. It rejects 3.
• BH: the largest $i$ below its line is $i = 5$. It rejects 5, including $p_{(4)} = 0.026$, which lies above its own line. This is the step-up rule. The adjusted p-values are $0.0032, 0.0084, 0.020, 0.0496, 0.0496, 0.24, 0.503, 0.87$. For example, $\tilde p_{(4)} = \min(8 \cdot 0.026/4,\ 8 \cdot 0.031/5,\ \dots) = \min(0.052, 0.0496, \dots) = 0.0496$.

5. In code #

SciPy checks the numbers of sections 2 and 4, and a simulation checks the type I error of the two t-tests.

import numpy as np
from scipy import stats

# coin: 61 heads in 100 flips
print(round(stats.binomtest(61, 100, 0.5).pvalue, 4))   # 0.0352  exact two-sided
print(round(2 * stats.norm.sf((61 - 50) / 5), 4))       # 0.0278  CLT approximation

# A. Welch vs Student from summary statistics (mean, sd, n)
new, old = (6.2, 5.0, 8), (3.0, 2.0, 30)
for equal_var in (False, True):
    r = stats.ttest_ind_from_stats(*new, *old, equal_var=equal_var)
    print(equal_var, round(r.statistic, 3), round(r.pvalue, 4))
# False 1.773 0.1161   (Welch)
# True 2.829 0.0076    (Student)

# B. two proportions = chi-squared test of the 2x2 table, without Yates' correction
table = np.array([[18, 382], [33, 367]])                 # rows: page A, page B
chi2, p, dof, _ = stats.chi2_contingency(table, correction=False)
print(round(chi2, 3), round(np.sqrt(chi2), 3), round(p, 4))   # 4.712 2.171 0.0299

# C. goodness of fit of the rank histogram
print(stats.chisquare([31, 22, 17, 15, 14, 13, 16, 18, 21, 33]))   # statistic 21.7, pvalue 0.0099

# D. Benjamini-Hochberg adjusted p-values
pv = np.array([0.0004, 0.0021, 0.0075, 0.026, 0.031, 0.18, 0.44, 0.87])
print(stats.false_discovery_control(pv, method="bh").round(4))
# [0.0032 0.0084 0.02   0.0496 0.0496 0.24   0.5029 0.87  ]

# type I error when H0 is true (equal means) with the sizes and spreads of example A
rng = np.random.default_rng(0)
x = rng.normal(0, 5.0, size=(20_000, 8))                # 20,000 simulated studies
y = rng.normal(0, 2.0, size=(20_000, 30))
for equal_var in (False, True):
    pval = stats.ttest_ind(x, y, axis=1, equal_var=equal_var).pvalue
    print(equal_var, np.mean(pval < 0.05))
# False 0.05      (Welch keeps the level)
# True 0.21825    (Student rejects 4 times too often)

chi2_contingency applies Yates' continuity correction to $2 \times 2$ tables by default. Pass correction=False when you compare with the z-test.

6. Where it shows up #

• Comparing ML models. Score both models on the same test examples or seeds and use a paired test: a paired t-test on per-seed scores, or McNemar's test for the errors of two classifiers on the same examples. With 3–5 seeds the interval for the gain is often wider than the gain. A leaderboard of many models on one test set is a multiple-testing problem.
• A/B tests. The two-proportion z-test (conversion rates) and Welch's t-test (revenue, time spent) are the standard analyses. A χ² test of the assignment counts detects a broken randomisation (sample-ratio mismatch, lesson 12).
• Drift monitoring. Tests on hundreds of input features every day need BH, or most alarms are false.
• Ensemble forecasts. The rank histogram of section 4C is a standard calibration check for weather and climate ensembles. Correlated cases (consecutive days, nearby stations) must be thinned or handled with an effective sample size.
• Maps of significance. A trend test in each of thousands of grid cells is $m$ tests at once; BH is the usual remedy. Neighbouring cells are positively correlated, so check the PRDS condition or use Benjamini–Yekutieli.
• Physics. The 5σ discovery rule is a one-sided p-value of $2.9 \cdot 10^{-7}$. The "look-elsewhere effect" is a multiple-testing correction over the many places where a signal could have appeared.

7. Common mistakes #

• Reading $p$ as $P(H_0 \mid \text{data})$, or as "the probability that the result is due to chance". The p-value is computed assuming $H_0$; it cannot also give the probability of $H_0$.
• "Not significant, so there is no effect." The interval $[-1.0, 7.4]$ of example 4A allows no effect and also a large one. To show that an effect is small, use an equivalence test (two one-sided tests, TOST) or a narrow interval.
• Statistical vs practical significance. With a huge $n$ every tiny effect is "significant". Report the effect size and its interval.
• The pooled t-test with unequal sizes and unequal variances. Use Welch.
• An unpaired test on paired data (it wastes power), or a test for independent data on dependent data (time steps, pixels, sessions of one user): the SE and $p$ are then too small.
• Choosing the test after seeing the data: one-sided after seeing the sign, or trying tests until $p < 0.05$. Fix the test, the metric and $\alpha$ first.
• χ² with expected counts below 5, and forgetting that chi2_contingency applies Yates' correction by default.
• Ignoring multiplicity, or using Bonferroni to screen thousands of hypotheses (no power left).
• Reading FDR control as "each rejection is false with probability $q$". It bounds the average FDP over repeated studies, not the FDP of this study.

8. Check yourself #

1. $n = 25$ paired differences have mean 0.4 and $s = 1.0$. Test $H_0: \mu = 0$ (two-sided) and give the 95% interval ($t_{24,\,0.975} = 2.064$).
   Answer

   $\operatorname{SE} = 1/\sqrt{25} = 0.2$, so $t = 0.4/0.2 = 2.0$ with 24 degrees of freedom, and $p = 0.057$: not significant at 5%. The interval is $0.4 \pm 2.064 \cdot 0.2 = [-0.013, 0.813]$. It contains 0 (duality), but it also contains effects as large as 0.8: the data cannot tell "no effect" from "a large effect".
2. Under $H_0$, with a continuous test statistic, what is $P(p \le 0.2)$? Why can $P(p \le 0.05)$ be smaller than 0.05 for the exact binomial test?
   Answer

   0.2, because $p \sim \text{Uniform}(0, 1)$ (probability integral transform). A discrete statistic takes only some values, so only some p-values are possible. The rejection region $\{p \le 0.05\}$ then has probability at most 0.05; for 100 coin flips it is 0.035.
3. A one-sided z-test at $\alpha = 0.05$ has effect $\delta/\sigma = 0.2$ and $n = 100$. What is its power? Which $n$ gives power 0.8?
   Answer

   $\lambda = 0.2\sqrt{100} = 2$, so the power is $\Phi(2 - 1.645) = \Phi(0.355) = 0.64$. Power 0.8 needs $0.2\sqrt n - 1.645 = 0.842$, so $\sqrt n = 2.487/0.2 = 12.43$ and $n = 154.6$, that is $n = 155$.
4. When do the pooled and Welch t-statistics coincide? When does the pooled test reject far too often?
   Answer

   They coincide when $n_x = n_y = n$: then $s_p^2 = (s_x^2 + s_y^2)/2$ and $s_p^2 (2/n) = s_x^2/n + s_y^2/n$. Only the degrees of freedom differ. The pooled test rejects too often when the sizes differ and the smaller group has the larger variance, because the pooled variance then underestimates $\operatorname{Var}(\bar x - \bar y)$.
5. Ten independent tests, all nulls true, each at level 0.05. Give the FWER without correction and with Bonferroni.
   Answer

   Without correction: $1 - 0.95^{10} = 0.401$. Bonferroni tests each at $0.005$: $1 - 0.995^{10} = 0.0489 \le 0.05$. The union bound gives $10 \cdot 0.005 = 0.05$ for any dependence.
6. BH at $q = 0.10$ rejects 40 hypotheses. A colleague says: "so exactly 4 of them are false discoveries." What is right and what is wrong?
   Answer

   Wrong: BH bounds the *expected* false discovery proportion over repeated studies, $\mathbb{E}[V/\max(R, 1)] \le q\,m_0/m \le 0.10$ (under independence or PRDS). It says nothing exact about these 40: $V$ can be 0 or well above 4, and you do not know which rejections are false. "About 4 or fewer" is a reasonable rough guess, not a guarantee.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Define the p-value.

   Show answer

2. What are the type I and type II errors of a test?

   Show answer

3. What is the power of a one-sided z-test with effect $\delta$, noise $\sigma$ and $n$ observations?

   Show answer

4. What is the distribution of a p-value under $H_0$ when the test statistic is continuous, and why?

   Show answer

5. Write Welch's t statistic.

   Show answer

6. Write the pooled two-proportion z statistic.

   Show answer

7. What does the duality of tests and confidence intervals say?

   Show answer

8. State the Bonferroni rule and what it controls.

   Show answer

9. State the Benjamini–Hochberg procedure.

   Show answer

10. What is the difference between the FWER and the FDR?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Explain a p-value to a product manager." — If the change did nothing, a difference at least this large would show up in $p \times 100\%$ of repeated experiments. It is not the probability that the change does nothing.
• "Your test gives $p = 0.06$. What do you do?" — Report the estimate and the interval. Check the planned power. Do not extend the test until it becomes significant, and do not call it a "trend".
• "Student or Welch? And when not a t-test at all?" — Welch by default. For heavy tails or small skewed samples, use a permutation test, a bootstrap interval (lesson 12) or a rank test (Mann–Whitney). Mann–Whitney tests whether one group tends to give larger values ($P(X > Y) \ne 1/2$), not whether the means differ.
• "You test 100 features and 7 are significant at 5%. What do you conclude?" — Under the global null you expect 5 false positives. Use BH (screening) or Holm (confirmation), and validate on new data.
• "How are confidence intervals and tests related?" — Duality: the 95% interval is the set of null values that the 5% test does not reject.
• "What is the power of a test, and what increases it?" — The probability of rejecting $H_0$ when a given effect is real. It grows with the effect, with $n$, with $\alpha$, and with less noise (pairing, a better metric).

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution:

1. Write welch_t_test (statistic, Welch–Satterthwaite $\nu$, p-value) and compare it with scipy.stats.ttest_ind(equal_var=False).
2. Write two_proportion_z_test with its 95% interval, and check that $z^2$ equals the χ² statistic of the $2 \times 2$ table.
3. Measure the type I error of Student's and Welch's tests by simulation, with unequal variances and sizes.
4. Write benjamini_hochberg (adjusted p-values) and compare it with scipy.stats.false_discovery_control.
5. Estimate the power of a z-test by simulation and compare it with the formula $\Phi(\lambda - z_{1-\alpha/2}) + \Phi(-\lambda - z_{1-\alpha/2})$.

11. Watch and read #

p-values: What they are and how to interpret themStatQuest with Josh Starmer · 11 min

A slow, clear walk through what a p-value measures, with a drug example.

Statistical Power, Clearly Explained!!!StatQuest with Josh Starmer · 8 min

Power as the overlap of two distributions, as in the power figure of this lesson.

False Discovery Rates, FDR, clearly explainedStatQuest with Josh Starmer · 18 min

Why many tests give many false positives, and how the Benjamini–Hochberg procedure works.

7. Parametric Hypothesis TestingMIT OpenCourseWare · 18.650 Statistics for Applications · 79 min

The formal view: level, power, p-values, and tests built from asymptotically normal estimators.

• Philippe Rigollet, MIT 18.650 Statistics for Applications (2016), lecture slides, "Parametric hypothesis testing": errors, level, power and p-values; the χ² and t distributions; Student's and Welch's tests.
• Larry Wasserman, 36-705 lecture notes on multiple testing: the BH procedure, FDR vs FWER, and the proof that BH controls the FDR under independence.
• Kevin Murphy, Probabilistic Machine Learning: An Introduction, free PDF: §4.7.4–4.7.5 (confidence intervals are not credible intervals) and §5.5 (type I and II errors, p-values, "p-values considered harmful").
• SciPy scipy.stats reference: ttest_ind (equal_var=False is Welch), chi2_contingency (correction), chisquare, false_discovery_control.