Random variables: PMF, PDF, CDF, expectation and variance

Before this lesson: lesson 01 (events, conditional probability, independence). For the continuous part you need integrals and the substitution rule in one variable (calculus, lesson 10). Time: about 70 minutes, plus the notebook.

---

Why this matters #

Almost every quantity in machine learning is an expectation. The risk of a model is $\mathbb{E}[\ell(f(X), Y)]$, the expected loss on a new example. Cross-entropy, the ELBO of a variational autoencoder and the value function of reinforcement learning are expectations too.

Linearity of expectation is the reason why a minibatch gradient is an unbiased estimate of the full gradient (optimization, lesson 5). It needs no independence, so it holds even when the examples of a batch are related.

Every sampler, from a Monte Carlo study to a diffusion model, starts from simple noise and transforms it. The density of the result follows the change-of-variables rule of this lesson. The reparameterisation trick of a VAE, $z = \mu + \sigma \varepsilon$, is the simplest case. The probability integral transform of section 3 is the standard calibration check for probabilistic forecasts, from weather ensembles to the predictive distributions of a neural network.

1. Predict first #

2. The picture #

A random variable is a function #

Roll two fair dice. The sample space $\Omega$ has 36 equally likely outcomes $(i, j)$ (lesson 01). Let $X$ be the sum of the two dice. Then $X$ is a function on $\Omega$: it maps $(1, 6)$ to 7, $(4, 4)$ to 8, and so on.

A random variable is a function $X : \Omega \to \mathbb{R}$. It is not random in itself; the outcome is random, and $X$ reads a number from it. We write $\{X = 7\}$ for the event "the sum is 7", the set of outcomes that $X$ maps to 7. It has 6 outcomes, so $P(X = 7) = 6/36$.

The distribution of $X$ says how the probability mass on $\Omega$ lands on the real line. All 36 outcomes carry mass $1/36$. The function $X$ moves each of them to a point of the line, and the masses pile up: 6 outcomes land on 7, only one lands on 2.

There are two simple cases.

• Discrete: the mass sits on separate points. We draw a bar chart, and the bar heights are probabilities. The sum of two dice is discrete.
• Continuous: the mass is spread over an interval. We draw a curve, and areas under the curve are probabilities. The height is probability per unit length, so it can be larger than 1. The waiting time until the next customer is continuous.

The cumulative distribution function (CDF) $F_X(x) = P(X \le x)$ is the mass collected from the left up to $x$. It works for both cases: it is a staircase for a discrete $X$ and a smooth curve for a continuous one. The mass between $a$ and $b$ is the rise of the CDF:

Interactive figure: loading…

For the dice, $P(4 < X \le 7) = P(X \in \{5, 6, 7\}) = (4 + 5 + 6)/36 = 15/36 \approx 0.417$. For a continuous $X$, a single point has no width, so it has no area: $P(X = x) = 0$ for every $x$. This is why $P(a < X \le b)$ and $P(a \le X \le b)$ are the same number for a continuous $X$, but not for the dice.

The expectation is a balance point #

Put the probability masses on a light beam, at their values. The expectation $\mathbb{E}[X]$ is the point where the beam balances. For a discrete $X$,

For two dice, the distribution is symmetric around 7, so the beam balances at 7, and indeed $\mathbb{E}[X] = 7$. A far mass has a long lever arm: it pulls the balance point toward itself, even when its probability is small. This is why the mean is sensitive to outliers.

Interactive figure: loading…

The torque about a point $f$ is $\sum_i p_i (x_i - f) = \mathbb{E}[X] - f$, because the $p_i$ add up to 1. It is zero only at $f = \mathbb{E}[X]$. The variance is the average squared lever arm, $\operatorname{Var}(X) = \sum_i p_i (x_i - \mathbb{E}[X])^2$. Squaring makes far masses count much more.

Reading the CDF backwards #

The CDF maps every value $x$ to a level $u = F(x)$ between 0 and 1. Now go the other way. Pick a level $u$ uniformly at random in $(0, 1)$ and find the $x$ where the CDF reaches $u$. This $x$ is a sample from the distribution. This method is called inverse-CDF sampling.

Why does it work? Where the CDF is steep, the density is high. A steep part of the CDF covers a long range of levels $u$ over a short range of $x$. Many uniform levels therefore land on that short range, and the samples pile up exactly where the density is high.

Interactive figure: loading…

3. Definitions and main results #

PMF, PDF and CDF #

• The probability mass function (PMF) of a discrete $X$ is $p_X(x) = P(X = x)$. It satisfies $p_X(x) \ge 0$ and $\sum_x p_X(x) = 1$.
• A continuous $X$ has a probability density function (PDF) $p_X \ge 0$ with $\int_{-\infty}^{\infty} p_X(x)\,dx = 1$ and $P(a \le X \le b) = \int_a^b p_X(x)\,dx$ for all $a \le b$. Then $P(X = x) = 0$ for every $x$.
• The cumulative distribution function (CDF) is $F_X(x) = P(X \le x)$ for every real $x$.

Properties of a CDF. $F_X$ is nondecreasing, it tends to 0 at $-\infty$ and to 1 at $+\infty$, and it is right-continuous. For all $a < b$, $P(a < X \le b) = F_X(b) - F_X(a)$: the event $\{X \le b\}$ is the disjoint union of $\{X \le a\}$ and $\{a < X \le b\}$. For a continuous $X$, $p_X = F_X'$ wherever $p_X$ is continuous. For a discrete $X$, $F_X$ jumps by $p_X(x)$ at each value $x$.

We use $p_X(x)$, or just $p(x)$, for both the PMF and the PDF. The context says which one.

Expectation, LOTUS and linearity #

Definition. The expectation (or mean) of $X$ is

It exists when $\sum_x |x|\,p_X(x) < \infty$ (or the integral version is finite). Most distributions in this track have a mean, but not all of them.

A distribution without a mean: the Cauchy distribution

The Cauchy density is $p(x) = \dfrac{1}{\pi(1 + x^2)}$. It is symmetric around 0 and looks like a normal curve with heavy tails. But $\int_{-c}^{c} |x|\, p(x)\,dx = \frac{1}{\pi} \ln(1 + c^2)$ grows without bound as $c \to \infty$, so $\mathbb{E}[X]$ does not exist. The consequence is real: the average of $n$ independent Cauchy samples is again Cauchy, with the same spread. It never settles down, however large $n$ is. The law of large numbers (lesson 06) needs a finite mean.

We often need the mean of a function of $X$, such as $X^2$ or a cost $c(X)$. The next theorem says that we do not need the distribution of $g(X)$ for it.

Theorem (LOTUS, the law of the unconscious statistician). For any function $g$,

whenever the sum (or integral) of $|g(x)|\,p_X(x)$ is finite.

Proof (discrete).

1. $Y = g(X)$ takes the value $y$ when $X$ takes any $x$ with $g(x) = y$. So $P(Y = y) = \sum_{x:\, g(x) = y} p_X(x)$.
2. By the definition of the mean, $\mathbb{E}[Y] = \sum_y y\,P(Y = y) = \sum_y \sum_{x:\, g(x) = y} g(x)\,p_X(x)$, because $y = g(x)$ inside the inner sum.
3. Every $x$ belongs to exactly one group (the group of $y = g(x)$). So the double sum is a sum over all $x$: $\sum_x g(x)\,p_X(x)$. $\blacksquare$

Theorem (linearity of expectation). For random variables $X, Y$ with finite means and constants $a, b, c$,

No independence is needed.

Proof (discrete).

1. Let $p(x, y) = P(X = x, Y = y)$ be the joint PMF (lesson 04 studies it). LOTUS for the pair $(X, Y)$ gives $\mathbb{E}[X + Y] = \sum_{x, y} (x + y)\,p(x, y)$.
2. Split the sum into two parts: $\sum_x x \sum_y p(x, y) + \sum_y y \sum_x p(x, y)$.
3. The inner sums are the PMFs of $X$ and of $Y$ alone: $\sum_y p(x, y) = p_X(x)$ by the law of total probability. So the total is $\mathbb{E}[X] + \mathbb{E}[Y]$.
4. LOTUS with $g(x) = ax + c$ gives $\mathbb{E}[aX + c] = a\,\mathbb{E}[X] + c$. Combine the two results. $\blacksquare$

We never wrote $p(x, y)$ as a product $p_X(x)\,p_Y(y)$. That is why the result holds for dependent variables.

A warning. In general $\mathbb{E}[g(X)] \ne g(\mathbb{E}[X])$. It is equal for every $X$ only when $g$ is linear plus a constant. For a convex $g$ (such as $x^2$ or $e^x$), Jensen's inequality says $\mathbb{E}[g(X)] \ge g(\mathbb{E}[X])$ (information theory, lesson 3). For example, $\mathbb{E}[X^2] \ge \mathbb{E}[X]^2$, and the gap is the variance.

Indicator variables #

The indicator of an event $A$ is the random variable $I_A$ that is 1 when $A$ happens and 0 otherwise. Its mean is a probability:

The indicator method. To find the mean of a count, write the count as a sum of indicators and add their probabilities. Linearity does the rest. You never need to know how the indicators depend on each other.

Example 1: fixed points. Shuffle $n$ cards numbered $1, \dots, n$. A fixed point is a card that ends in its own position. Let $I_i$ be the indicator of "card $i$ is in position $i$". By symmetry, card $i$ is equally likely to be in each of the $n$ positions, so $P(I_i = 1) = 1/n$. The number of fixed points is $N = I_1 + \dots + I_n$, and

The indicators are dependent (if cards $1, \dots, n-1$ are fixed, card $n$ must be fixed too), but linearity does not care.

Example 2: records. Daily values $X_1, \dots, X_n$ are independent and identically distributed (i.i.d.) and continuous, so there are no ties. Day $k$ is a record if $X_k$ is larger than all earlier values. By symmetry, each of the first $k$ days is equally likely to hold the largest of the first $k$ values, so $P(\text{day } k \text{ is a record}) = 1/k$. Hence

For a year of daily temperatures with no trend ($n = 365$), this is $H_{365} \approx 6.48$ records. Many more records than this are evidence of a trend.

Variance and standardising #

Definition. With $\mu = \mathbb{E}[X]$, the variance is $\operatorname{Var}(X) = \mathbb{E}[(X - \mu)^2]$, and the standard deviation is $\sigma = \sqrt{\operatorname{Var}(X)}$. The standard deviation has the same unit as $X$.

Theorem. (a) $\operatorname{Var}(X) = \mathbb{E}[X^2] - \mu^2$. (b) $\operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X)$.

Proof.

1. (a) Expand the square: $(X - \mu)^2 = X^2 - 2\mu X + \mu^2$.
2. Use linearity: $\mathbb{E}[X^2] - 2\mu\,\mathbb{E}[X] + \mu^2 = \mathbb{E}[X^2] - 2\mu^2 + \mu^2 = \mathbb{E}[X^2] - \mu^2$.
3. (b) $aX + b$ has mean $a\mu + b$, so $aX + b - (a\mu + b) = a(X - \mu)$. Square it and take the mean: $a^2\,\mathbb{E}[(X - \mu)^2] = a^2 \operatorname{Var}(X)$. $\blacksquare$

A shift $b$ moves the distribution but does not change its spread. A scale $a$ multiplies the spread by $|a|$, so the variance by $a^2$.

Standardising. The z-score $Z = (X - \mu)/\sigma$ has mean 0 and variance 1, by (b) with $a = 1/\sigma$ and $b = -\mu/\sigma$. It measures a value in standard deviations from the mean. If exam scores have mean 70 and standard deviation 10, a score of 85 has $z = (85 - 70)/10 = 1.5$. Feature standardisation in ML ("StandardScaler") applies this to every input feature.

Sums. Random variables $X$ and $Y$ are independent if $P(X \in A, Y \in B) = P(X \in A)\,P(Y \in B)$ for all sets $A, B$ of values. For independent $X$ and $Y$, the variances add: $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$. For dependent variables there is an extra covariance term; lesson 04 proves both. One consequence: for i.i.d. $X_1, \dots, X_n$ with variance $\sigma^2$, the sample mean $\bar X_n = \frac1n \sum_i X_i$ has variance $\sigma^2/n$ (use (b) with $a = 1/n$). Averages are less noisy than single values.

The density of $Y = g(X)$ #

We often transform a random variable: $Y = X^2$, $Y = e^X$, $Y = \mu + \sigma X$. What is the density of $Y$? The answer is not $p_X(g^{-1}(y))$. There is an extra factor, because $g$ stretches or squeezes the axis.

Interactive figure: loading…

The rule is conservation of probability. A short interval of length $|dx|$ around $x$ holds probability $p_X(x)\,|dx|$. The map $g$ sends it to an interval of length $|dy|$ around $y = g(x)$ that holds the same probability. So $p_Y(y)\,|dy| = p_X(x)\,|dx|$, and $p_Y(y) = p_X(x)\,|dx/dy|$.

Theorem (change of variables in one dimension). Let $X$ have density $p_X$, and let $g$ be strictly monotone and differentiable, with $g' \ne 0$ on the set where $p_X > 0$. Then $Y = g(X)$ has density

for $y$ in the image of that set, and $p_Y(y) = 0$ elsewhere.

Proof.

1. Let $g$ be increasing. Then $g(X) \le y$ exactly when $X \le g^{-1}(y)$. So $F_Y(y) = F_X(g^{-1}(y))$.
2. Differentiate with the chain rule: $p_Y(y) = p_X(g^{-1}(y))\,(g^{-1})'(y)$. Here $(g^{-1})' > 0$.
3. Let $g$ be decreasing. Then $g(X) \le y$ exactly when $X \ge g^{-1}(y)$. So $F_Y(y) = 1 - F_X(g^{-1}(y))$ ($P(X = g^{-1}(y)) = 0$ because $X$ is continuous).
4. Differentiate: $p_Y(y) = -p_X(g^{-1}(y))\,(g^{-1})'(y)$. This is positive, because $(g^{-1})' < 0$. Both cases give the absolute value in the theorem. $\blacksquare$

Example. $X \sim U(0, 1)$ and $Y = X^2$. Then $g^{-1}(y) = \sqrt y$ and $(g^{-1})'(y) = 1/(2\sqrt y)$, so $p_Y(y) = 1 \cdot \frac{1}{2\sqrt y}$ for $0 < y \le 1$. The density is large near 0: the map $x \mapsto x^2$ squeezes the interval $(0, 0.5)$ into $(0, 0.25)$.

Example (affine map). $Y = \mu + \sigma X$ with $\sigma > 0$. Then $g^{-1}(y) = (y - \mu)/\sigma$, so $p_Y(y) = \frac1\sigma\, p_X\big(\frac{y - \mu}{\sigma}\big)$. With $X \sim \mathcal N(0, 1)$, this gives the density of $\mathcal N(\mu, \sigma^2)$. This is the reparameterisation trick.

When $g$ is not monotone, use the CDF method. Compute $F_Y(y) = P(g(X) \le y)$ directly, then differentiate. Let $X$ be uniform on $(-1, 1)$, with density $1/2$, and $Y = X^2$. For $0 < y < 1$:

In general, add the contributions of all branches: for $Y = X^2$, $p_Y(y) = \big[p_X(\sqrt y) + p_X(-\sqrt y)\big] / (2\sqrt y)$. Lesson 05 extends the rule to vectors, where the factor $|dx/dy|$ becomes the absolute determinant of a Jacobian matrix.

Inverse-CDF sampling and the probability integral transform #

Theorem. Let $F$ be a continuous, strictly increasing CDF, and let $U \sim U(0, 1)$.

• (a) Inverse-CDF sampling: $X = F^{-1}(U)$ has CDF $F$.
• (b) Probability integral transform (PIT): if $X$ has CDF $F$, then $F(X) \sim U(0, 1)$.

Proof.

1. $F$ is strictly increasing, so $F^{-1}(u) \le x$ exactly when $u \le F(x)$.
2. (a) $P(F^{-1}(U) \le x) = P(U \le F(x)) = F(x)$, because $P(U \le u) = u$ for $u \in [0, 1]$.
3. (b) For $u \in (0, 1)$: $P(F(X) \le u) = P(X \le F^{-1}(u)) = F(F^{-1}(u)) = u$. This is the CDF of $U(0, 1)$. $\blacksquare$

Example. $\text{Exp}(\lambda)$ has CDF $F(x) = 1 - e^{-\lambda x}$ for $x \ge 0$. Solve $u = F(x)$: $x = -\ln(1 - u)/\lambda$. So $-\ln(1 - U)/\lambda$ is an exponential sample. With $\lambda = 1$ and $u = 0.6$, it gives $x = -\ln 0.4 \approx 0.916$.

Discrete distributions. For a CDF with jumps or flat parts, use the generalised inverse $F^{-1}(u) = \min\{x : F(x) \ge u\}$. Step 1 still holds, because $F$ is right-continuous, so (a) still holds. In practice: compute the cumulative sums of the probabilities and return the first value whose cumulative sum reaches $u$. A binary search does this in $O(\log K)$ steps for $K$ values (np.searchsorted). Part (b) needs a continuous $F$: for a discrete $X$, $F(X)$ takes only the values $F(x_k)$, so it cannot be uniform.

PIT histograms check calibration. A model predicts a full distribution, with CDF $F_t$, for each future value $y_t$ (tomorrow's temperature, a delivery time, the next sales number). If the model is calibrated (its predicted distributions are correct), the values $F_t(y_t)$ are uniform by part (b). So you plot a histogram of the PIT values:

• flat: consistent with calibration (necessary, but not sufficient);
• U-shaped: the observations fall too often in the tails, so the predicted distributions are too narrow (overconfident);
• hump-shaped: the predicted distributions are too wide (underconfident).

For example, if the observations are $\mathcal N(0, 1)$ but the model predicts $\mathcal N(0, 0.5^2)$, the five bins of the PIT histogram have masses $0.337, 0.113, 0.101, 0.113, 0.337$: a clear U.

Answers to section 1. (1) No. $P(Y \le 0.25) = P(X \le 0.5) = 0.5$, and $p_Y(y) = 1/(2\sqrt y)$ piles up near 0. (2) $50 \cdot 0.3 = 15$. Write the count as a sum of 50 indicators and use linearity. Independence is not needed: dependence changes the variance of the count, not its mean. (3) Yes. A density is probability per unit length; $U(0, 1/3)$ has density 3 on its interval.

4. Worked example #

A. Discrete. A small support team gets $X$ urgent tickets per day, with

$x$	0	1	2	3
$p_X(x)$	0.4	0.3	0.2	0.1

1. CDF. Add from the left: $F(0) = 0.4$, $F(1) = 0.7$, $F(2) = 0.9$, $F(3) = 1$ (a staircase). So $P(X \ge 2) = 1 - F(1) = 0.3$.
2. Mean. $\mathbb{E}[X] = 0 \cdot 0.4 + 1 \cdot 0.3 + 2 \cdot 0.2 + 3 \cdot 0.1 = 0 + 0.3 + 0.4 + 0.3 = 1.0$.
3. Second moment (LOTUS with $g(x) = x^2$). $\mathbb{E}[X^2] = 0 \cdot 0.4 + 1 \cdot 0.3 + 4 \cdot 0.2 + 9 \cdot 0.1 = 0.3 + 0.8 + 0.9 = 2.0$.
4. Variance. $\operatorname{Var}(X) = 2.0 - 1.0^2 = 1.0$, so $\sigma = 1.0$.
5. A cost. Each day costs $c(X) = 10X^2$ (overtime grows fast). LOTUS gives $\mathbb{E}[c(X)] = 10 \cdot 2.0 = 20$. But $c(\mathbb{E}[X]) = 10 \cdot 1^2 = 10$. Planning with the average number of tickets underestimates the average cost by half (Jensen, $x^2$ is convex).

This is the "worked example" preset of the balance-point figure in section 2.

B. Continuous. $X = \max(U_1, U_2)$ for two independent uniforms on $[0, 1]$ (the better of two random scores).

1. CDF. $X \le x$ exactly when both $U_1 \le x$ and $U_2 \le x$. By independence, $F(x) = x \cdot x = x^2$ for $0 \le x \le 1$.
2. Density. $p(x) = F'(x) = 2x$ on $[0, 1]$. Large values are more likely, as expected for a maximum.
3. Mean. $\mathbb{E}[X] = \int_0^1 x \cdot 2x\,dx = \left[\tfrac{2x^3}{3}\right]_0^1 = \tfrac23$.
4. Variance. $\mathbb{E}[X^2] = \int_0^1 x^2 \cdot 2x\,dx = \left[\tfrac{x^4}{2}\right]_0^1 = \tfrac12$, so $\operatorname{Var}(X) = \tfrac12 - \tfrac49 = \tfrac{1}{18} \approx 0.0556$.
5. Inverse CDF. Solve $u = x^2$: $F^{-1}(u) = \sqrt u$. So $\sqrt U$ has the same distribution as $\max(U_1, U_2)$: one uniform number is enough. (The "density 2x" preset of the inverse-CDF figure.)
6. PIT check. $Y = F(X) = X^2$. With $g^{-1}(y) = \sqrt y$ and $(g^{-1})'(y) = 1/(2\sqrt y)$, the change-of-variables rule gives $p_Y(y) = 2\sqrt y \cdot \frac{1}{2\sqrt y} = 1$ on $(0, 1)$. Uniform, as the theorem says.

5. In code #

SciPy checks the hand computations of the worked example. Then a simulation checks the indicator results and the PIT histogram. Simulated numbers carry a Monte Carlo error: for the mean number of records, about $\pm 0.01$ with 20,000 simulated years.

import numpy as np
from scipy import stats, integrate

# A. discrete: a distribution from a table
X = stats.rv_discrete(values=([0, 1, 2, 3], [0.4, 0.3, 0.2, 0.1]))
print(X.cdf([0, 1, 2, 3]), X.mean(), X.moment(2), X.var())
# [0.4 0.7 0.9 1. ] 1.0 2.0 1.0
print(X.expect(lambda t: 10 * t**2))              # 20.0 (LOTUS), while c(E[X]) = 10

# B. continuous: density 2x on [0, 1]
m1 = integrate.quad(lambda t: t * 2 * t, 0, 1)[0]
m2 = integrate.quad(lambda t: t**2 * 2 * t, 0, 1)[0]
print(round(m1, 4), round(m2 - m1**2, 4))         # 0.6667 0.0556 (2/3 and 1/18)

rng = np.random.default_rng(1)
u = rng.random(100_000)
mx = np.maximum(rng.random(100_000), rng.random(100_000))   # max of two uniforms
print(stats.ks_2samp(np.sqrt(u), mx).pvalue > 0.01)        # True: same law

# inverse CDF of Exp(lam): -log(1 - U) / lam
lam = 2.0
x = -np.log(1 - rng.random(100_000)) / lam
print(round(x.mean(), 3), round(x.var(), 3))      # 0.502 0.252 (exact 0.5, 0.25)

# indicators: fixed points of 20,000 random permutations of 10 items
perms = np.argsort(rng.random((20_000, 10)), axis=1)
print(round((perms == np.arange(10)).sum(axis=1).mean(), 2))   # 1.0 (exact 1)

# indicators: records in 365 i.i.d. daily values, 20,000 simulated years
temps = rng.standard_normal((20_000, 365))
records = (temps >= np.maximum.accumulate(temps, axis=1)).sum(axis=1)
print(round(records.mean(), 2), round((1 / np.arange(1, 366)).sum(), 2))
# 6.47 6.48

# PIT histograms: observations y ~ N(0, 1), two forecasts
y = rng.standard_normal(100_000)
for sd in (1.0, 0.5):
    pit = stats.norm(0, sd).cdf(y)
    print(sd, np.round(np.histogram(pit, bins=5, range=(0, 1))[0] / y.size, 2))
# 1.0 [0.2 0.2 0.2 0.2 0.2]      calibrated: flat
# 0.5 [0.34 0.11 0.1 0.11 0.34]  too narrow: U-shaped

The ks_2samp line is a two-sample Kolmogorov–Smirnov test: a p-value above 0.01 means the two samples are consistent with one distribution (tests are lesson 11).

6. Where it shows up #

• Risk and minibatch gradients. Training minimises the average loss $\frac1n \sum_i \ell_i$, an estimate of the risk $\mathbb{E}[\ell]$. By linearity, the gradient on a random minibatch has the full gradient as its mean, whatever the dependence between the examples.
• Reparameterisation and flows. A VAE samples $z = \mu + \sigma \varepsilon$ with $\varepsilon \sim \mathcal N(0, 1)$, a monotone change of variables, so gradients can flow through the sample (generative models, lesson 2). Normalising flows chain many invertible maps and track the density with the multivariate version of the rule.
• Transport maps. In one dimension, $T = F_Y^{-1} \circ F_X$ sends a continuous $X$ to a variable with CDF $F_Y$ (PIT, then inverse CDF). Quantile normalisation of features and quantile mapping of climate-model output use this map. Diffusion and flow-matching models learn such noise-to-data maps in high dimension.
• Forecast calibration. PIT histograms (and rank histograms of ensembles) are the standard check of probabilistic weather and climate forecasts, and of the predictive distributions of regression models. A U-shape means overconfidence.
• A/B tests. A conversion is an indicator; the conversion rate is a mean of indicators, with mean $p$ and, for $n$ independent users, variance $p(1 - p)/n$.
• Records and extremes. In a series without trend, the expected number of records grows only like $\ln n$. Counting records is a simple test for a trend in temperature series.

7. Common mistakes #

• $\mathbb{E}[g(X)] = g(\mathbb{E}[X])$. False in general. For a convex $g$, $\mathbb{E}[g(X)] \ge g(\mathbb{E}[X])$ (Jensen). In the worked example: $20 \ge 10$.
• Reading a density value as a probability. $U(0, 0.5)$ has density 2. Only areas under the density are probabilities.
• Forgetting the factor $|dg^{-1}/dy|$ when you transform a density. Check that the new density integrates to 1.
• Using the change-of-variables formula for a non-monotone $g$. For $Y = X^2$ with $X$ on both sides of 0, add both branches, or use the CDF method.
• Adding variances of dependent variables. $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$ needs zero covariance. And for independent $X, Y$, $\operatorname{Var}(X - Y)$ is also the sum $\operatorname{Var}(X) + \operatorname{Var}(Y)$, not the difference.
• $\mathbb{E}[X^2] - \mathbb{E}[X]^2$ in floating point. With values near $10^8$, both terms are huge and nearly equal, and the difference loses all its digits. Compute $\mathbb{E}[(X - \mu)^2]$ instead (two passes over the data).
• $\log U$ with $U = 0$. rng.random() returns values in $[0, 1)$ and can return exactly 0. Use $-\log(1 - U)$, which is always finite.
• Expecting $F(X)$ to be uniform for a discrete $X$. It takes only the values $F(x_k)$.

8. Check yourself #

1. $X \sim U(0, 2)$. Give the density, $P(X \le 0.5)$, $\mathbb{E}[X]$ and $\operatorname{Var}(X)$.
   Answer

   The density is $1/2$ on $[0, 2]$. $P(X \le 0.5) = 0.5 \cdot 1/2 = 0.25$. $\mathbb{E}[X] = 1$ (the middle). $\mathbb{E}[X^2] = \int_0^2 x^2/2\,dx = 4/3$, so $\operatorname{Var}(X) = 4/3 - 1 = 1/3$ (in general $(b - a)^2/12$ for $U(a, b)$).
2. $\operatorname{Var}(X) = 4$. Find $\operatorname{Var}(3X - 2)$, and $\operatorname{Var}(X - Y)$ for an independent copy $Y$ of $X$.
   Answer

   $\operatorname{Var}(3X - 2) = 9 \cdot 4 = 36$. $\operatorname{Var}(X - Y) = \operatorname{Var}(X) + (-1)^2 \operatorname{Var}(Y) = 4 + 4 = 8$.
3. 30 people have independent birthdays, uniform over 365 days. What is the expected number of pairs of people with the same birthday?
   Answer

   One indicator per pair. Each pair shares a birthday with probability $1/365$. There are $\binom{30}{2} = 435$ pairs, so the mean is $435/365 \approx 1.19$. The pair indicators are dependent, but linearity does not need independence.
4. $X \sim \mathcal N(0, 1)$ and $Y = e^X$. Find the density of $Y$.
   Answer

   $g(x) = e^x$ is increasing, $g^{-1}(y) = \ln y$ and $(g^{-1})'(y) = 1/y$. So $p_Y(y) = \varphi(\ln y)/y$ for $y > 0$, where $\varphi$ is the standard normal density. This is the **log-normal** distribution.
5. Observations come from $\mathcal N(0, 1)$, but a model predicts $\mathcal N(0, 2^2)$. What shape does the PIT histogram have?
   Answer

   A hump: the predicted distribution is too wide, so the observations fall mostly in its middle, where $F$ is near 0.5. With 5 bins the masses are about $0.05, 0.26, 0.39, 0.26, 0.05$.
6. You want samples of a discrete $X$ with $P(X = 0, 1, 2, 3) = 0.4, 0.3, 0.2, 0.1$, and you draw $u = 0.75$. Which value does the inverse-CDF method return?
   Answer

   The cumulative sums are $0.4, 0.7, 0.9, 1.0$. The first value whose cumulative sum reaches $0.75$ is $x = 2$ (because $0.7 < 0.75 \le 0.9$).

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. What is the difference between a PMF value and a PDF value?

   Show answer

2. How do you get $P(a < X \le b)$ from the CDF?

   Show answer

3. State LOTUS.

   Show answer

4. Does linearity of expectation need independence? Give the formula.

   Show answer

5. What is the expected number of fixed points of a random permutation of $n$ items, and why?

   Show answer

6. Give the shortcut formula for the variance, and $\operatorname{Var}(aX + b)$.

   Show answer

7. What is the density of $Y = g(X)$ for a strictly monotone, differentiable $g$?

   Show answer

8. How do you sample $\text{Exp}(\lambda)$ from $U \sim U(0, 1)$?

   Show answer

9. What is the probability integral transform, and what is it used for?

   Show answer

10. A PIT histogram is U-shaped. What is wrong with the forecasts?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "You only have a uniform random number generator. How do you sample from a distribution with CDF $F$?" — Inverse CDF: return $F^{-1}(U)$. For a discrete distribution, binary search in the cumulative sums, $O(\log K)$ per draw. If $F^{-1}$ has no closed form: a transformation such as Box–Muller (lesson 05) or rejection sampling (lesson 14).
• "What is $\mathbb{E}[\max(U_1, U_2)]$ for two independent uniforms?" — $2/3$, from the CDF $x^2$. In general, the maximum of $n$ uniforms has CDF $x^n$ and mean $n/(n + 1)$.
• "Expected number of fixed points, records, empty boxes or shared birthdays?" — Write the count as a sum of indicators, and use linearity.
• "Is $\mathbb{E}[1/X] = 1/\mathbb{E}[X]$?" — No. For a positive $X$, $1/x$ is convex, so $\mathbb{E}[1/X] \ge 1/\mathbb{E}[X]$ by Jensen, with equality only for a constant $X$.
• "Why does a minibatch gradient work?" — It is an unbiased estimate of the full gradient by linearity of expectation; its variance shrinks like $1/(\text{batch size})$ for independent samples.

10. Practice #

The notebook for this lesson runs in Colab or in your browser. It has five exercises, each with a check cell and a folded solution:

1. mean_var(xs, ps): the mean and the variance from a PMF table, checked with scipy.stats.rv_discrete.
2. sample_exponential(lam, n, rng): an inverse-CDF sampler for $\text{Exp}(\lambda)$, checked with a Kolmogorov–Smirnov test against SciPy.
3. sample_categorical(p, n, rng): the discrete inverse CDF with cumulative sums and np.searchsorted.
4. density_x_squared(y): the density of $Y = X^2$ for $X \sim U(-1, 1)$, compared with a histogram.
5. mean_fixed_points(n, trials, rng): simulate random permutations and compare the mean number of fixed points with 1.

11. Watch and read #

Lecture 9: Expectation, Indicator Random Variables, Linearity | Statistics 110Harvard University · Joe Blitzstein, Statistics 110 · 50 min

Linearity of expectation and the indicator method, with several counting examples like those of section 3.

Why “probability of 0” does not mean “impossible” | Probabilities of probabilities, part 23Blue1Brown · 10 min

Why a continuous variable needs a density, and why single points have probability 0 (section 2).

8. Continuous Random VariablesMIT OpenCourseWare · John Tsitsiklis, MIT 6.041 · 50 min

PDFs, CDFs, expectation and variance for continuous random variables, at the level of section 3.

• Joseph Blitzstein and Jessica Hwang, Introduction to Probability, 2nd edition, free PDF: ch. 3 (random variables, PMF, CDF), ch. 4 (expectation, linearity, indicators, LOTUS, variance), ch. 5 (continuous random variables, "universality of the uniform" = inverse-CDF sampling) and §8.1 (change of variables).
• Art Owen, Monte Carlo theory, methods and examples, ch. 4 "Non-uniform random variables", §4.1–4.4: inversion, also for discrete distributions.
• Tilmann Gneiting, Fadoua Balabdaoui and Adrian Raftery, Probabilistic forecasts, calibration and sharpness, JRSS B 2007, §3.1: PIT histograms and their shapes.
• MIT 6.041, Probabilistic Systems Analysis and Applied Probability (OpenCourseWare): lectures 5–9 cover discrete and continuous random variables.