Chance and data: probability, mean, variance and the normal curve

Before this lesson: lesson 02 ($\sum$-notation), lesson 03 (seeded random numbers, axis) and lesson 04 ($e^x$). Time: about 60 minutes, plus the notebook. The full story is in the Probability and statistics track.

---

Why this matters #

• Preprocessing. StandardScaler in scikit-learn turns each feature into z-scores: subtract the mean, divide by the standard deviation. Many models train better after this step.
• Initialization. Neural network weights start as random numbers, often from a normal distribution $\mathcal N(0, \sigma^2)$ with $\sigma^2$ chosen from the layer size (Glorot and He initialization).
• Losses. If you assume that the noise in the targets is normal, the best fit is the one with the smallest squared error. This is why the mean squared error is the default loss for regression.
• Evaluation. A test accuracy is a frequency. On 1000 test examples, an accuracy of 0.90 has an uncertainty of about $\pm 0.02$. Two models that differ by 0.005 may not differ at all.
• Training noise. A mini-batch gradient is an average over $B$ examples. Its noise shrinks like $1/\sqrt{B}$, the same law that you see for coin flips below.

1. Predict first #

2. The picture #

2.1 Probability as long-run frequency #

Flip a fair coin many times and keep a running score: the relative frequency of heads, that is, (number of heads so far) / (number of flips so far). After 10 flips it can be 0.3 or 0.7. After 10 000 flips it is very close to 0.5. This is the frequency view of probability: $P(\text{heads}) = 0.5$ means that the relative frequency gets close to 0.5 in a long series of independent flips. (The Probability and statistics track also treats probability as a degree of belief; the rules are the same.)

The important word is slowly. The typical gap between the frequency and $p$ is $\sqrt{p(1-p)/n}$ (section 3.4 proves this). It shrinks like $1/\sqrt n$: to make the gap 10 times smaller, you need 100 times more flips.

Interactive figure: loading…

2.2 Centre and spread #

Put the data $2, 4, 4, 4, 5, 5, 7, 9$ on a number line. The mean $\bar x$ is the point where the line would balance if each data point were a small equal weight. The median is the middle value after sorting. To measure the spread, take each deviation $x_i - \bar x$, square it, and average the squares: this is the variance. Its square root is the standard deviation.

Interactive figure: loading…

2.3 Histograms #

For many data points, a dot plot is too crowded. A histogram cuts the number line into bins of equal width $w$ and draws a bar for each bin. The bar height is either the count (number of points in the bin) or the density $\text{count} / (n w)$. With density heights, the area of each bar is the fraction of the data in that bin, so all bar areas add up to 1. Only density histograms can be compared with a density curve, or with a histogram of a different size $n$.

Interactive figure: loading…

2.4 The normal curve #

As $n$ grows and $w$ shrinks, a density histogram gets close to a smooth curve: the density of the distribution. The most common one is the bell-shaped normal curve $\mathcal N(\mu, \sigma^2)$. Its centre is the mean $\mu$, and its width is the standard deviation $\sigma$. A probability is an area under the curve: $P(a \le X \le b)$ is the area between $a$ and $b$.

Interactive figure: loading…

3. Definitions and main results #

3.1 Experiments, events and the rules of probability #

An experiment has outcomes that we cannot predict, for example one roll of a die. The set of all outcomes is the sample space $\Omega = \{1, 2, 3, 4, 5, 6\}$. An event is a set of outcomes, for example $A = \{2, 4, 6\}$ ("even"). A probability $P$ gives each event a number with three properties:

1. $0 \le P(A) \le 1$ for every event $A$;
2. $P(\Omega) = 1$ (something happens);
3. if $A$ and $B$ are disjoint (they have no outcome in common), then $P(A \text{ or } B) = P(A) + P(B)$.

For a fair die, every outcome has probability $1/6$ and $P(A) = |A|/6$, so $P(\text{even}) = 3/6$.

Two rules that follow.

• Complement: $P(\text{not } A) = 1 - P(A)$. Proof. $A$ and "not $A$" are disjoint, and together they are $\Omega$. Property 3 and property 2 give $P(A) + P(\text{not } A) = P(\Omega) = 1$. $\blacksquare$
• "Or" for any two events: $P(A \text{ or } B) = P(A) + P(B) - P(A \text{ and } B)$. The outcomes in both events are counted twice in $P(A) + P(B)$, so we subtract them once. Example with one die, $A = \{2, 4, 6\}$ and $B = \{5, 6\}$: $\frac36 + \frac26 - \frac16 = \frac46$, and indeed $A \text{ or } B = \{2, 4, 5, 6\}$.

Independence. Two events are independent if $P(A \text{ and } B) = P(A)\,P(B)$. In words: knowing that $B$ happened does not change the chance of $A$. Separate flips of a coin are independent, so $P(\text{two heads}) = \frac12 \cdot \frac12 = \frac14$. The product rule is only for independent events: for one die, $P(\text{even and odd}) = 0$, not $\frac14$.

Conditional probability

The probability of $A$ given that $B$ happened is $P(A \mid B) = P(A \text{ and } B) / P(B)$ (for $P(B) > 0$). Independence is the same as $P(A \mid B) = P(A)$. Bayes' rule, $P(A \mid B) = P(B \mid A) P(A) / P(B)$, follows from writing $P(A \text{ and } B)$ in two ways. The Probability and statistics track builds on this.

3.2 Random variables and distributions #

A random variable $X$ is a number that depends on the outcome: the face of a die, the height of a random person, the error of a model on a random test example.

• A discrete random variable takes separate values. Its distribution is a table of probabilities $P(X = x)$ that add up to 1. For a fair die, $P(X = x) = 1/6$ for $x = 1, \dots, 6$.
• A continuous random variable can take any value in an interval. Its distribution is a density $p(x) \ge 0$ with total area 1 under the curve, and

(The integral sign $\int$ means "area under the curve"; the Calculus track defines it.)

A density is not a probability. For a small width $h$, $P(x \le X \le x + h) \approx p(x)\,h$. So $p(x)$ is "probability per unit length", and it can be larger than 1 when the distribution is narrow. Example: the uniform distribution on $[0, 0.5]$ has density $p(x) = 2$ on that interval; the area is $2 \times 0.5 = 1$. Also, for a continuous $X$, $P(X = a) = 0$ for every single value $a$: a line has no area.

3.3 Mean, median, variance and standard deviation #

For data $x_1, \dots, x_n$ (a sample):

Quantity	Formula	Unit
mean	$\bar x = \frac{1}{n} \sum_{i=1}^n x_i$	same as the data
median	middle value after sorting (the average of the two middle values if $n$ is even)	same as the data
variance	$\sigma^2 = \frac{1}{n} \sum_{i=1}^n (x_i - \bar x)^2$	data unit squared
standard deviation	$\sigma = \sqrt{\sigma^2}$	same as the data

The standard deviation (std) is the one to report: it has the unit of the data (cm, not cm²), and it is a typical distance from the mean.

For a random variable, the same ideas use probabilities as weights. The expectation (or expected value) is $\mathbb E[X] = \sum_x x\, P(X = x)$, and the variance is $\operatorname{Var}(X) = \mathbb E\big[(X - \mu)^2\big]$ with $\mu = \mathbb E[X]$. For a fair die, $\mathbb E[X] = \frac{1 + 2 + \dots + 6}{6} = 3.5$ (not a possible face: an expectation is an average, not a typical outcome), and $\operatorname{Var}(X) = 35/12 \approx 2.92$. The data formulas are the special case where each of the $n$ points has probability $1/n$.

Three facts with short proofs.

1. The deviations add up to 0: $\sum_i (x_i - \bar x) = \sum_i x_i - n\bar x = n\bar x - n\bar x = 0$. This is why the mean is the balance point.
2. Shortcut formula: $\sigma^2 = \frac1n \sum_i x_i^2 - \bar x^2$ (mean of the squares minus the square of the mean). Proof. Expand $(x_i - \bar x)^2 = x_i^2 - 2 x_i \bar x + \bar x^2$ and average over $i$: $\frac1n\sum_i x_i^2 - 2\bar x \cdot \bar x + \bar x^2 = \frac1n\sum_i x_i^2 - \bar x^2$. $\blacksquare$
3. Shift and scale: if $y_i = a x_i + b$, then $\bar y = a\bar x + b$ and $\sigma_y = |a|\,\sigma_x$. Proof. Averaging gives $\bar y = a \bar x + b$. So $y_i - \bar y = a(x_i - \bar x)$: the shift $b$ cancels. Squaring gives $(y_i - \bar y)^2 = a^2 (x_i - \bar x)^2$, so $\sigma_y^2 = a^2 \sigma_x^2$ and $\sigma_y = |a|\,\sigma_x$. $\blacksquare$

Outliers. One extreme value can move the mean and the std a lot, because the mean uses the values and the variance uses their squares. The median uses only the order, so it is robust to outliers.

3.4 Sums and averages of independent variables #

Two rules for random variables $X, Y$ (proofs in the probability track):

• $\mathbb E[X + Y] = \mathbb E[X] + \mathbb E[Y]$ always, and $\mathbb E[aX + b] = a\,\mathbb E[X] + b$;
• $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$ if $X$ and $Y$ are independent, and $\operatorname{Var}(aX) = a^2 \operatorname{Var}(X)$.

Why the variances add for independent variables

Write $\mu_X, \mu_Y$ for the means. Then $(X + Y) - (\mu_X + \mu_Y) = (X - \mu_X) + (Y - \mu_Y)$. Square and take expectations:

$$\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\,\mathbb E\big[(X - \mu_X)(Y - \mu_Y)\big].$$

For independent variables, the expectation of a product is the product of the expectations, so the last term is $2\,\mathbb E[X - \mu_X]\,\mathbb E[Y - \mu_Y] = 2 \cdot 0 \cdot 0 = 0$. $\blacksquare$

The $1/\sqrt n$ law. Let $X_1, \dots, X_n$ be independent, each with mean $\mu$ and variance $\sigma^2$, and let $\bar X = \frac1n \sum_i X_i$. Then

Back to the coin. Let $X_i = 1$ for heads and $0$ for tails. Then $\mathbb E[X_i] = p$ and $\mathbb E[X_i^2] = p$, so $\operatorname{Var}(X_i) = p - p^2 = p(1-p)$ (shortcut formula). The running frequency is $\bar X$, so its typical gap to $p$ is $\sqrt{p(1-p)/n}$. This is the band in the coin figure. The law of large numbers says the same in words: as $n$ grows, $\bar X$ gets close to $\mu$ with probability close to 1.

3.5 Dividing by $n$ or by $n - 1$ #

The sample variance divides by $n - 1$:

Why? We usually want the variance $\sigma^2$ of the whole population, but we measure the deviations from the sample mean $\bar x$, not from the true mean $\mu$. The sample mean sits in the middle of this sample, so the deviations from it are a bit too small on average. Dividing by $n - 1$ instead of $n$ corrects this exactly: $\mathbb E[s^2] = \sigma^2$ (we say $s^2$ is unbiased).

Why $n - 1$ gives an unbiased estimate

Let $x_1, \dots, x_n$ be independent with mean $\mu$ and variance $\sigma^2$.

1. Write $x_i - \mu = (x_i - \bar x) + (\bar x - \mu)$ and square: $(x_i - \mu)^2 = (x_i - \bar x)^2 + 2(x_i - \bar x)(\bar x - \mu) + (\bar x - \mu)^2$.
2. Sum over $i$. The middle term is $2(\bar x - \mu)\sum_i (x_i - \bar x) = 0$ (fact 1 of section 3.3). So $\sum_i (x_i - \mu)^2 = \sum_i (x_i - \bar x)^2 + n(\bar x - \mu)^2$.
3. Take expectations. On the left, each term has expectation $\sigma^2$, so we get $n\sigma^2$. On the right, $\mathbb E[(\bar x - \mu)^2] = \operatorname{Var}(\bar x) = \sigma^2/n$ (section 3.4). So $n\sigma^2 = \mathbb E\big[\sum_i (x_i - \bar x)^2\big] + \sigma^2$.
4. Hence $\mathbb E\big[\sum_i (x_i - \bar x)^2\big] = (n - 1)\sigma^2$, and dividing by $n - 1$ gives $\mathbb E[s^2] = \sigma^2$. $\blacksquare$

The square root $s$ is still slightly too small on average, but for large $n$ the difference is tiny.

Library defaults differ. NumPy's np.var and np.std divide by $n$ (ddof=0, "delta degrees of freedom"); pandas' .var() and .std() divide by $n - 1$ (ddof=1). scikit-learn's StandardScaler uses $n$. For the lesson data, np.std gives 2.0 and pandas gives 2.138. For large $n$ the difference is small; for small $n$, say which one you use.

3.6 The normal distribution #

The normal (or Gaussian) distribution $\mathcal N(\mu, \sigma^2)$ has the density

Read it from the inside out: $(x - \mu)/\sigma$ is the distance from the centre in units of $\sigma$; it is squared, so the curve is symmetric; $e^{-(\cdot)}$ makes the curve fall fast away from the centre (lesson 04); the factor $\frac{1}{\sigma\sqrt{2\pi}}$ makes the total area 1. The peak height is $\frac{1}{\sigma\sqrt{2\pi}}$: $0.399$ for $\sigma = 1$ and $3.99$ for $\sigma = 0.1$.

z-scores. The standard normal is $Z \sim \mathcal N(0, 1)$. If $X \sim \mathcal N(\mu, \sigma^2)$, then the z-score $Z = (X - \mu)/\sigma$ is standard normal (shift and scale, section 3.3). So every normal probability is a standard normal probability:

where $\Phi(z) = P(Z \le z)$ is the standard normal cumulative distribution function (CDF), in scipy.stats.norm.cdf.

The 68–95–99.7 rule. For any normal distribution, $P(|X - \mu| \le \sigma) = 0.6827$, $P(|X - \mu| \le 2\sigma) = 0.9545$, $P(|X - \mu| \le 3\sigma) = 0.9973$. A value more than 3 standard deviations from the mean is rare (about 3 in 1000), if the data are normal.

3.7 The central limit theorem #

Theorem (central limit theorem, CLT). Let $X_1, \dots, X_n$ be independent and have the same distribution, with mean $\mu$ and finite variance $\sigma^2 > 0$. Then, for large $n$, the mean $\bar X$ is approximately $\mathcal N(\mu, \sigma^2/n)$. Precisely: for every $z$,

The same holds for the sum $X_1 + \dots + X_n$, which is approximately $\mathcal N(n\mu, n\sigma^2)$.

Interactive figure: loading…

What the CLT does not say.

• It does not say that data are normal. It is about averages and sums of many independent effects. Incomes, waiting times and word frequencies are strongly skewed.
• It does not say how large $n$ must be. For a symmetric distribution, $n = 5$ can be enough; for a skewed one you need more.
• It needs a finite variance and independence. The mean of samples from the Cauchy distribution (which has no finite variance) never settles: it has the same distribution for every $n$.

This is why the normal curve is everywhere: a measurement error is often the sum of many small independent errors.

Answers to section 1. (1) Not necessarily. A fair coin gives 7 or more heads in 10 flips with probability $176/1024 \approx 0.17$ (section 4c), about 1 time in 6. Ten flips are too few to tell. (2) The standard deviation is 2 (section 4a). After adding 50, the mean jumps from 5 to 10, but the median only moves from 4.5 to 5. (3) Yes. Density is probability per unit length; only areas must be at most 1. The peak of $\mathcal N(0, 0.1^2)$ is 3.99.

4. Worked example #

(a) Mean, variance and standard deviation by hand. Data: $2, 4, 4, 4, 5, 5, 7, 9$, so $n = 8$.

Mean: $\bar x = \frac{2 + 4 + 4 + 4 + 5 + 5 + 7 + 9}{8} = \frac{40}{8} = 5$.

$x_i$	2	4	4	4	5	5	7	9	sum
$x_i - \bar x$	$-3$	$-1$	$-1$	$-1$	0	0	2	4	0
$(x_i - \bar x)^2$	9	1	1	1	0	0	4	16	32

The deviations add up to 0 (fact 1). Variance: $\sigma^2 = 32/8 = 4$, so $\sigma = 2$. Sample variance: $s^2 = 32/7 \approx 4.571$, so $s \approx 2.138$. Median: the sorted data are already in order; the two middle values are 4 and 5, so the median is 4.5.

Check with the shortcut formula: $\sum_i x_i^2 = 4 + 3 \cdot 16 + 2 \cdot 25 + 49 + 81 = 232$, so $\frac{232}{8} - 5^2 = 29 - 25 = 4$ ✓.

z-scores $z_i = (x_i - 5)/2$: $-1.5, -0.5, -0.5, -0.5, 0, 0, 1, 2$. The value 9 is 2 standard deviations above the mean.

Shift and scale: for $y_i = 3x_i + 5$, the mean is $3 \cdot 5 + 5 = 20$ and the std is $3 \cdot 2 = 6$.

Outlier: add 50. Mean $= 90/9 = 10$, median $= 5$, and $\sigma$ grows from 2 to about 14.27.

(b) A normal probability. Adult heights in some population are about $\mathcal N(170, 10^2)$ (in cm). What fraction is between 160 and 180 cm, and what fraction is above 190 cm?

1. z-scores: $z = (160 - 170)/10 = -1$ and $z = (180 - 170)/10 = 1$.
2. $P(160 \le X \le 180) = \Phi(1) - \Phi(-1) = 0.8413 - 0.1587 = 0.6827$: the "68" of the rule.
3. For 190 cm: $z = (190 - 170)/10 = 2$. By symmetry, the two tails outside $\pm 2$ have $1 - 0.9545 = 0.0455$ together, so each tail has $0.0228$. About 2.3 % are taller than 190 cm.

(c) Is 7 heads in 10 flips surprising? Assume the coin is fair. Each of the $2^{10} = 1024$ sequences of heads and tails has probability $1/1024$. The number of sequences with exactly $k$ heads is $\binom{10}{k}$, the number of ways to choose which $k$ flips are heads. So

A fair coin does this about once in 6 tries. It is not good evidence of a bias.

5. In code #

NumPy, pandas and SciPy check the numbers of section 4. The notebook for this lesson has these cells and the exercises.

import numpy as np
import pandas as pd
from scipy import stats

x = np.array([2, 4, 4, 4, 5, 5, 7, 9], dtype=float)
print(x.mean(), np.median(x))               # 5.0 4.5
print(np.var(x), np.std(x))                 # 4.0 2.0           (divide by n, ddof=0)
print(np.var(x, ddof=1), np.std(x, ddof=1)) # 4.571... 2.138... (divide by n - 1)
print(pd.Series(x).std())                   # 2.138...          (pandas: ddof=1 by default)
print((x - x.mean()) / x.std())             # z-scores: [-1.5 -0.5 -0.5 -0.5  0.   0.   1.   2. ]
print(np.std(3 * x + 5))                    # 6.0 = 3 * 2

# (b) normal probabilities with the CDF
print(stats.norm.cdf(180, 170, 10) - stats.norm.cdf(160, 170, 10))  # 0.6826894921370859
print(stats.norm.sf(190, 170, 10))          # 0.022750131948179198  (sf = 1 - cdf)

# (c) 7 or more heads in 10 fair flips
print(stats.binom.sf(6, 10, 0.5))           # 0.171875 = 176/1024  (P(X > 6))

# the running frequency of heads, with a seed
rng = np.random.default_rng(0)
flips = rng.random(10_000) < 0.5            # True = heads
freq = np.cumsum(flips) / np.arange(1, 10_001)
print(freq[[9, 99, 999, 9999]])             # [0.3   0.44  0.473 0.499]  (n = 10, 100, 1000, 10 000)

With this seed, the first 10 flips give only 3 heads. The frequency gets close to 0.5, but slowly. Another seed gives other numbers, with the same slow approach.

6. Where it shows up #

• Feature standardization. StandardScaler stores the mean and std of each column of the training data and maps $x \mapsto (x - \bar x)/\sigma$. Use the training statistics also for the test data.
• Batch normalization and layer normalization. Inside a network, these layers compute a mean and a variance over a batch or over the features, and standardize. To normalize, they divide by $n$.
• Weight initialization. He initialization draws weights from $\mathcal N(0, 2/n_\text{in})$, so that the variance of the activations stays about the same from layer to layer (variances of independent terms add, section 3.4).
• Gaussian noise and squared error. Linear regression assumes $y = w^\top x + b + \varepsilon$ with $\varepsilon \sim \mathcal N(0, \sigma^2)$. Maximizing the likelihood of this model is the same as minimizing the squared error (the ML Fundamentals track shows this).
• Uncertainty of a metric. An accuracy $\hat p$ on $n$ test examples has a standard error of about $\sqrt{\hat p(1 - \hat p)/n}$: $\sqrt{0.9 \cdot 0.1/1000} \approx 0.0095$ for $\hat p = 0.9$, $n = 1000$.
• Generative models. Diffusion models and variational autoencoders start from standard normal noise and transform it into images, molecules or weather fields.
• ML for science. Repeated measurements are averaged and reported as mean ± std (or ± std$/\sqrt n$ for the uncertainty of the mean). Ensembles of climate or weather simulations are summarized by their mean and spread.

7. Common mistakes #

• Reading a short run as the probability. 7 heads in 10 flips does not mean $p = 0.7$. The typical error after $n$ trials is about $\sqrt{p(1-p)/n}$, which is 0.16 for $p = 0.5$ and $n = 10$.
• Treating a density value as a probability. $p(x)$ can be larger than 1. Probabilities are areas.
• Mixing $n$ and $n - 1$. NumPy and pandas use different defaults. Say which one you use, and do not compare a NumPy std with a pandas std.
• Reporting the variance in data units. The variance of heights is in cm². Report the std, which is in cm.
• Using the product rule for events that are not independent. $P(A \text{ and } B) = P(A)P(B)$ only for independent events.
• Trusting the mean with outliers. A single extreme value can move the mean and the std a lot. Look at the median and at a histogram too.
• Believing the CLT makes data normal. It is about averages of many independent terms, not about the data themselves.
• Choosing a histogram's bin width without looking. Too wide hides structure (two peaks become one); too narrow shows noise. Try a few widths, or bins="auto" in NumPy and matplotlib.

8. Check yourself #

1. You roll two fair dice. What is the probability that the sum is 7?
   Answer

   There are $6 \cdot 6 = 36$ equally likely pairs. The pairs with sum 7 are $(1,6), (2,5), (3,4), (4,3), (5,2), (6,1)$: 6 pairs. So the probability is $6/36 = 1/6$.
2. $P(A) = 0.3$, $P(B) = 0.5$, and $A$ and $B$ are independent. Compute $P(A \text{ and } B)$ and $P(A \text{ or } B)$.
   Answer

   $P(A \text{ and } B) = 0.3 \cdot 0.5 = 0.15$. $P(A \text{ or } B) = 0.3 + 0.5 - 0.15 = 0.65$.
3. Test scores are about $\mathcal N(60, 15^2)$. About what fraction of students score above 90?
   Answer

   $z = (90 - 60)/15 = 2$. The two tails beyond $\pm 2$ hold $1 - 0.9545 = 0.0455$, so one tail holds about $0.023$: about 2.3 %.
4. One measurement has standard deviation 5. You average 25 independent measurements. What is the standard deviation of the average?
   Answer

   $\sigma/\sqrt n = 5/\sqrt{25} = 1$. To halve it again, you need $4 \cdot 25 = 100$ measurements.
5. You make a histogram of 50 values with bin width 0.05. Why does it look noisy, and what should you do?
   Answer

   There are many more bins than the data can fill: most bins hold 0 or 1 value, so the heights jump. Use wider bins (fewer bins), or collect more data.
6. A model has accuracy 0.90 on 1000 test examples. A second model has 0.905. Is the second model better?
   Answer

   Not clearly. The standard error of each accuracy is about $\sqrt{0.9 \cdot 0.1 / 1000} \approx 0.0095$, so a difference of 0.005 is well inside the noise. (A paired comparison on the same test set is more precise, but 0.005 is still small.)

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. In the frequency view, what does $P(\text{heads}) = 0.3$ mean?

   Show answer

2. How does the typical gap between a relative frequency and $p$ depend on the number of trials $n$?

   Show answer

3. Write the variance of data $x_1, \dots, x_n$ (dividing by $n$).

   Show answer

4. Why do we usually report the standard deviation and not the variance?

   Show answer

5. Compute the mean and standard deviation (divide by $n$) of $2, 4, 4, 4, 5, 5, 7, 9$.

   Show answer

6. What divisor do np.std and pandas .std() use by default?

   Show answer

7. All data are multiplied by $a$ and then $b$ is added. What happens to the standard deviation?

   Show answer

8. What fractions of a normal distribution lie within 1, 2 and 3 standard deviations of the mean?

   Show answer

9. Why can a probability density be larger than 1?

   Show answer

10. What does the central limit theorem say?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Why divide by $n - 1$ in the sample variance?" — The deviations are measured from $\bar x$, which is fitted to the same sample, so they are too small on average. With $n - 1$, $\mathbb E[s^2] = \sigma^2$. Know the defaults: NumPy ddof=0, pandas ddof=1.
• "What is the variance of the mean of $n$ i.i.d. variables?" — $\sigma^2/n$; the std is $\sigma/\sqrt n$. Use it for error bars on a metric and for the noise of a mini-batch gradient.
• "State the central limit theorem. When does it fail?" — Averages of i.i.d. variables with finite variance become normal. It fails without a finite variance (Cauchy) and can be slow for very skewed data; it says nothing about the data themselves.
• "Mean or median?" — The median is robust to outliers (for example house prices, latencies); the mean is the right summary when you care about totals, and it is what squared-error losses estimate.
• "Is a density value a probability?" — No; probability is the area under the density, and $p(x)$ can be larger than 1.

10. Practice #

The notebook for this lesson runs in Colab or in your browser. It has five exercises, each with a check cell and a folded solution:

1. Simulate coin flips with a seeded generator and plot the running frequency of several runs with the $\pm 2\sqrt{p(1-p)/n}$ band.
2. Implement the mean, variance and standard deviation with a ddof argument, and compare with NumPy and pandas.
3. Compute a density histogram with np.histogram and check that the bar areas add up to 1.
4. Standardize a dataset to z-scores, then count the fraction of normal samples within 1 and 2 standard deviations.
5. The CLT in action: average $k$ uniform numbers many times and compare the std of the averages with $\sqrt{1/12}/\sqrt k$.

11. Watch and read #

Calculating the Mean, Variance and Standard Deviation, Clearly Explained!!!StatQuest with Josh Starmer · 14 min

Why we divide by n − 1 for a sample, with a simple picture.

StatQuest: Histograms, Clearly ExplainedStatQuest with Josh Starmer · 4 min

The Normal Distribution, Clearly Explained!!!StatQuest with Josh Starmer · 5 min

But what is the Central Limit Theorem?3Blue1Brown · 31 min

Dice sums turning into a bell curve, and the role of the mean and the standard deviation.

• Daniel Kunin and others, Seeing Theory (Brown University), interactive chapters 1–3: chance events, expectation and variance, distributions, and the CLT.
• Illowsky and Dean, Introductory Statistics 2e (OpenStax, free): chapter 2 (descriptive statistics, histograms), chapter 3 (probability rules), chapter 6 (the normal distribution) and chapter 7 (the central limit theorem).
• Grinstead and Snell, Introduction to Probability (free PDF): chapter 1 (simulation and frequencies), chapter 6 (expected value and variance) and chapter 9 (CLT).
• Allen Downey, Think Stats (free online): the same ideas with Python and real data.