Joint and conditional distributions, conditional expectation and covariance

Before this lesson: lesson 02 (expectation, variance, LOTUS) and lesson 03 (common distributions). For the last part: symmetric matrices and the spectral theorem (linear algebra 06) and positive semidefinite matrices (linear algebra 07). Time: about 80 minutes, plus the notebook.

---

Why this matters #

• Regression and the MSE loss. A model trained with the squared error learns (in the limit of a lot of data and a flexible model) the conditional expectation $\mathbb{E}[Y \mid X = x]$. Section 3 proves that it is the best predictor in mean squared error.
• Uncertainty of ensembles. The predictive variance of a deep ensemble or a mixture model splits, by the law of total variance, into "average noise of the members" plus "disagreement between the members".
• Averaging and SGD. The variance of a sum needs the covariances. A mini-batch gradient averages $B$ per-example gradients; its variance drops like $1/B$ only when they are uncorrelated.
• PCA, whitening, Kalman filters. All of them are statements about a covariance matrix: its eigenvectors, its square root, or how it changes under a linear map.

1. Predict first #

2. The picture #

A joint table. Take two discrete random variables. In our running example, a student does two homework problems. $X$ is the number of problems the student solves (0, 1 or 2), and $Y = 1$ if the student went to the tutorial, $Y = 0$ if not. The joint PMF is a table: cell $(x, y)$ holds $P(X = x, Y = y)$, and all cells add up to 1.

	$Y = 0$ (no tutorial)	$Y = 1$ (tutorial)	row sum $p_X(x)$
$X = 0$	0.2	0.1	0.3
$X = 1$	0.2	0.2	0.4
$X = 2$	0.0	0.3	0.3
column sum $p_Y(y)$	0.4	0.6	1

Three operations on the table give everything in this lesson:

• Marginal = add up. The row sums are the distribution of $X$ alone; the column sums are the distribution of $Y$ alone. They are written in the margins of the table, hence the name.
• Conditional = keep one column and rescale it. Keep the column $Y = 1$: $(0.1, 0.2, 0.3)$. It sums to $0.6$, so divide by $0.6$: $p(x \mid Y = 1) = (1/6, 1/3, 1/2)$. This is the distribution of $X$ in the world where you know that the student went to the tutorial.
• Independence = every column has the same shape. If $X$ and $Y$ are independent, knowing $Y$ changes nothing, so every rescaled column equals the marginal $p_X$. Then the table is the outer product $p_X\,p_Y^\top$ of its two marginals, a matrix of rank 1.

Interactive figure: loading…

The conditional mean and the two parts of the spread. The conditional mean $\mathbb{E}[X \mid Y = y]$ is the balance point of column $y$. In the table it is $0.5$ for $Y = 0$ and $4/3$ for $Y = 1$. It is a number for each $y$, so it is a function of $y$. Averaging it over $y$ (with weights $p_Y$) gives back the overall mean.

The overall spread of $X$ has two sources. Inside each group the values scatter around the group mean (the within-group part). And the group means themselves differ (the between-group part). The law of total variance in section 3 says that the variance is exactly the sum of these two parts.

Interactive figure: loading…

Covariance as a picture. Draw a cloud of points $(x, y)$ and the cross through the means $(\bar x, \bar y)$. For each point, the product $(x - \bar x)(y - \bar y)$ is positive in the upper-right and lower-left quadrants and negative in the other two. The covariance is the average of these products. When most points lie in the positive quadrants, the cloud tilts up and the covariance is positive.

Interactive figure: loading…

3. Definitions and main results #

Joint, marginal and conditional distributions #

Let $X$ and $Y$ be two random variables defined on the same sample space.

• The joint PMF (discrete case) is $p(x, y) = P(X = x, Y = y)$.
• A joint PDF (continuous case) is a function $p(x, y) \ge 0$ with $P((X, Y) \in B) = \iint_B p(x, y)\,dx\,dy$ for every (measurable) set $B \subseteq \mathbb{R}^2$. As in one dimension, $p(x, y)$ is a density, not a probability: it can be larger than 1.
• The marginals are $p_X(x) = \sum_y p(x, y)$ and $p_Y(y) = \sum_x p(x, y)$; for densities, replace the sums by integrals, $p_X(x) = \int p(x, y)\,dy$.
• The conditional PMF or PDF of $X$ given $Y = y$ (where $p_Y(y) > 0$) is

• Multiplying back gives the chain rule: $p(x, y) = p(x \mid y)\,p_Y(y) = p(y \mid x)\,p_X(x)$.
• $X$ and $Y$ are independent if $p(x, y) = p_X(x)\,p_Y(y)$ for all $x$ and $y$. Equivalently, $p(x \mid y) = p_X(x)$ for every $y$ with $p_Y(y) > 0$: knowing $Y$ does not change the distribution of $X$.

Factorisation criterion. If $p(x, y) = g(x)\,h(y)$ for all $x, y$, for some functions $g$ and $h$, then $X$ and $Y$ are independent.

Proof (densities; for PMFs replace integrals by sums).

1. Let $G = \int g(x)\,dx$ and $H = \int h(y)\,dy$. The joint density integrates to 1, so $GH = 1$.
2. Integrate out one variable: $p_X(x) = \int g(x)h(y)\,dy = g(x)\,H$, and in the same way $p_Y(y) = h(y)\,G$.
3. So $p_X(x)\,p_Y(y) = g(x)h(y)\,GH = p(x, y)$. $\blacksquare$

For a table this says: $X$ and $Y$ are independent exactly when the joint table has rank 1. A rank-1 table is an outer product $uv^\top$, so $p(x, y) = u_x v_y$ has the form $g(x)h(y)$; conversely an independent table is $p_X\,p_Y^\top$.

The support matters. The condition "for all $x, y$" includes the points where the density is 0, so the set where $p > 0$ must be a rectangle (a product set). Take $p(x, y) = 2$ on the triangle $0 < x < y < 1$ and 0 elsewhere. The formula looks constant, but $X$ and $Y$ are dependent: $X > 0.9$ forces $Y > 0.9$. For this density:

• $p_X(x) = \int_x^1 2\,dy = 2(1 - x)$ and $p_Y(y) = \int_0^y 2\,dx = 2y$, both on $(0, 1)$;
• given $X = x$, $p(y \mid x) = 2 / (2(1 - x)) = 1/(1 - x)$ on $(x, 1)$: $Y$ is uniform on $(x, 1)$.

Conditional expectation, the tower rule and total variance #

The conditional expectation $\mathbb{E}[X \mid Y = y] = \sum_x x\,p(x \mid y)$ (or $\int x\,p(x \mid y)\,dx$) is a number for each $y$. Call this function $m(y)$. Now put the random variable $Y$ into it:

This is a random variable, a function of $Y$. It takes the value $m(y)$ when $Y = y$. In the same way, $\operatorname{Var}(X \mid Y = y) = \mathbb{E}[X^2 \mid Y = y] - m(y)^2$, and $\operatorname{Var}(X \mid Y)$ is that function evaluated at $Y$.

Theorem (tower rule, also called Adam's law or the law of iterated expectations). If $\mathbb{E}|X| < \infty$, then $\mathbb{E}\big[\mathbb{E}[X \mid Y]\big] = \mathbb{E}[X]$.

Proof (discrete case; the continuous case is the same with integrals).

1. $m(Y)$ is a function of $Y$, so by LOTUS $\mathbb{E}[m(Y)] = \sum_y p_Y(y)\,m(y) = \sum_y p_Y(y) \sum_x x\,\dfrac{p(x, y)}{p_Y(y)}$.
2. The factors $p_Y(y)$ cancel: $\sum_y \sum_x x\,p(x, y)$.
3. Sum over $y$ first: $\sum_x x\,p_X(x) = \mathbb{E}[X]$. $\blacksquare$

Theorem (law of total variance, also called Eve's law). If $\mathbb{E}[X^2] < \infty$, then

Proof. Write $m(Y) = \mathbb{E}[X \mid Y]$.

1. Use $\operatorname{Var}(X \mid Y) = \mathbb{E}[X^2 \mid Y] - m(Y)^2$ and the tower rule for $X^2$: $\mathbb{E}[\operatorname{Var}(X \mid Y)] = \mathbb{E}[X^2] - \mathbb{E}[m(Y)^2]$.
2. Use the definition of variance and the tower rule for $X$: $\operatorname{Var}(m(Y)) = \mathbb{E}[m(Y)^2] - (\mathbb{E}[m(Y)])^2 = \mathbb{E}[m(Y)^2] - (\mathbb{E}[X])^2$.
3. Add the two lines. The terms $\mathbb{E}[m(Y)^2]$ cancel, and what is left is $\mathbb{E}[X^2] - (\mathbb{E}[X])^2 = \operatorname{Var}(X)$. $\blacksquare$

Both parts are $\ge 0$. So, on average, knowing $Y$ cannot increase the variance: $\mathbb{E}[\operatorname{Var}(X \mid Y)] \le \operatorname{Var}(X)$.

The conditional expectation is the best predictor

Claim. For every function $h$, $\mathbb{E}[(X - m(Y))^2] \le \mathbb{E}[(X - h(Y))^2]$, where $m(Y) = \mathbb{E}[X \mid Y]$.

Proof.

1. Write $X - h(Y) = (X - m(Y)) + (m(Y) - h(Y))$ and expand the square.
2. The cross term is $2\,\mathbb{E}[(X - m(Y))\,(m(Y) - h(Y))]$. Condition on $Y$: given $Y$, the factor $m(Y) - h(Y)$ is a constant, and $\mathbb{E}[X - m(Y) \mid Y] = 0$. By the tower rule the cross term is 0.
3. So $\mathbb{E}[(X - h(Y))^2] = \mathbb{E}[(X - m(Y))^2] + \mathbb{E}[(m(Y) - h(Y))^2] \ge \mathbb{E}[(X - m(Y))^2]$. $\blacksquare$

This is why a regression model trained with the squared loss estimates $\mathbb{E}[\text{target} \mid \text{input}]$, and why the smallest possible MSE is $\mathbb{E}[\operatorname{Var}(X \mid Y)]$, the within-group part.

Covariance and correlation #

The covariance of $X$ and $Y$, with means $\mu_X$ and $\mu_Y$, is

(Expand the product and use linearity: $\mathbb{E}[XY] - \mu_X\mu_Y - \mu_X\mu_Y + \mu_X\mu_Y$.) The correlation is the covariance of the standardised variables,

Properties you will use all the time:

• $\operatorname{Cov}(X, X) = \operatorname{Var}(X)$, $\operatorname{Cov}(X, Y) = \operatorname{Cov}(Y, X)$, and $\operatorname{Cov}(X, c) = 0$ for a constant $c$.
• Covariance is bilinear: $\operatorname{Cov}(aX + bY, Z) = a\operatorname{Cov}(X, Z) + b\operatorname{Cov}(Y, Z)$, and the same in the second slot.
• So $\operatorname{Cov}(aX + b, cY + d) = ac\operatorname{Cov}(X, Y)$, and $\rho(aX + b, cY + d) = \operatorname{sign}(ac)\,\rho(X, Y)$: correlation has no units and ignores shifts and scales, but a negative factor flips its sign.

Theorem. $|\rho| \le 1$, that is, $\operatorname{Cov}(X, Y)^2 \le \operatorname{Var}(X)\operatorname{Var}(Y)$. Equality holds if and only if $Y = aX + b$ with probability 1, for some constants $a \ne 0$ and $b$.

Proof.

1. Standardise: $\tilde X = (X - \mu_X)/\sigma_X$ and $\tilde Y = (Y - \mu_Y)/\sigma_Y$. Both have variance 1, and $\operatorname{Cov}(\tilde X, \tilde Y) = \rho$.
2. A variance is never negative. By bilinearity, $0 \le \operatorname{Var}(\tilde X + \tilde Y) = 1 + 1 + 2\rho$ and $0 \le \operatorname{Var}(\tilde X - \tilde Y) = 1 + 1 - 2\rho$.
3. So $-1 \le \rho \le 1$.
4. If $\rho = 1$, then $\operatorname{Var}(\tilde X - \tilde Y) = 0$, so $\tilde X - \tilde Y$ is a constant with probability 1, and $Y$ is an affine function of $X$ with $a > 0$. The case $\rho = -1$ is the same with $\tilde X + \tilde Y$ and gives $a < 0$. $\blacksquare$

Uncorrelated does not mean independent #

Independent implies uncorrelated. If $X$ and $Y$ are independent, then $\mathbb{E}[XY] = \sum_{x, y} xy\,p_X(x)\,p_Y(y) = \big(\sum_x x\,p_X(x)\big)\big(\sum_y y\,p_Y(y)\big) = \mathbb{E}[X]\,\mathbb{E}[Y]$, so $\operatorname{Cov}(X, Y) = 0$.

The converse is false. Let $X$ be uniform on $\{-1, 0, 1\}$ and $Y = X^2$.

1. $\mathbb{E}[X] = 0$ and $\mathbb{E}[XY] = \mathbb{E}[X^3] = \tfrac13(-1 + 0 + 1) = 0$.
2. So $\operatorname{Cov}(X, Y) = 0 - 0 \cdot \mathbb{E}[Y] = 0$: uncorrelated.
3. But $Y$ is a function of $X$. For example $P(Y = 0 \mid X = 0) = 1$, while $P(Y = 0) = 1/3$: dependent.

The same happens for $Z \sim \mathcal N(0, 1)$ and $Z^2$, since all odd moments of a symmetric density are 0. Correlation measures only the linear part of a dependence. A U-shape, a circle or an X-shape can have correlation 0.

Interactive figure: loading…

The variance of a sum #

Theorem.

Proof.

1. $\operatorname{Var}(S) = \operatorname{Cov}(S, S)$ for $S = \sum_i X_i$.
2. By bilinearity, $\operatorname{Cov}\big(\sum_i X_i, \sum_j X_j\big) = \sum_i \sum_j \operatorname{Cov}(X_i, X_j)$: $n^2$ terms.
3. The $n$ terms with $i = j$ are the variances. Every pair $i < j$ appears twice, as $(i, j)$ and $(j, i)$. $\blacksquare$

For uncorrelated $X_i$ the covariances vanish and the variances simply add. For two variables: $\operatorname{Var}(X \pm Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) \pm 2\operatorname{Cov}(X, Y)$.

The covariance matrix #

For a random vector $X = (X_1, \dots, X_d)^\top$ with mean $\mu = \mathbb{E}[X]$, the covariance matrix is

Its diagonal holds the variances. The correlation matrix is $D^{-1/2}\,\Sigma\,D^{-1/2}$ with $D = \operatorname{diag}(\Sigma_{11}, \dots, \Sigma_{dd})$; its entries are the correlations $\rho_{ij}$.

Theorem (affine maps). For a fixed matrix $A \in \mathbb{R}^{m \times d}$ and vector $b \in \mathbb{R}^m$,

Proof.

1. By linearity, $\mathbb{E}[AX + b] = A\mu + b$. So $(AX + b) - \mathbb{E}[AX + b] = A(X - \mu)$: the shift $b$ drops out.
2. $\operatorname{Cov}(AX + b) = \mathbb{E}\big[A(X - \mu)(X - \mu)^\top A^\top\big] = A\,\mathbb{E}\big[(X - \mu)(X - \mu)^\top\big]\,A^\top = A\Sigma A^\top$, because expectation is linear, entry by entry. $\blacksquare$

With one row, $A = a^\top$, this gives the variance of any linear combination: $\operatorname{Var}(a^\top X) = a^\top \Sigma\, a$. With $a = (1, \dots, 1)$ it is again the variance of a sum.

Theorem. Every covariance matrix is symmetric and positive semidefinite (PSD): $a^\top \Sigma a \ge 0$ for every $a$. It is singular if and only if some combination $a^\top X$ with $a \ne 0$ is constant with probability 1.

Proof.

1. Symmetric: $\Sigma_{ij} = \operatorname{Cov}(X_i, X_j) = \operatorname{Cov}(X_j, X_i) = \Sigma_{ji}$.
2. PSD: $a^\top \Sigma a = \operatorname{Var}(a^\top X) \ge 0$, by the previous theorem.
3. Singular: for a PSD matrix, $\Sigma a = 0$ exactly when $a^\top \Sigma a = 0$ (write $a^\top \Sigma a = \|\Sigma^{1/2} a\|^2$, linear algebra 07). And $\operatorname{Var}(a^\top X) = 0$ exactly when $a^\top X$ is constant: the data lie on a hyperplane. $\blacksquare$

So a symmetric matrix with a negative eigenvalue is never a covariance matrix: it would give a "variance" $a^\top \Sigma a < 0$.

Bessel's correction: why the sample variance divides by $n - 1$ #

Theorem. Let $X_1, \dots, X_n$ ($n \ge 2$) be i.i.d. with mean $\mu$ and variance $\sigma^2$. Let $\bar X = \frac1n \sum_i X_i$ and

Then $\mathbb{E}[S^2] = \sigma^2$: $S^2$ is unbiased.

Proof.

1. Write $X_i - \bar X = (X_i - \mu) - (\bar X - \mu)$, square, and sum. Use $\sum_i (X_i - \mu) = n(\bar X - \mu)$:

2. Each term $(X_i - \mu)^2$ has mean $\sigma^2$, so $\mathbb{E}\big[\sum_i (X_i - \mu)^2\big] = n\sigma^2$.
3. The $X_i$ are independent, so by the variance of a sum $\operatorname{Var}(\bar X) = \frac{1}{n^2} \cdot n\sigma^2 = \sigma^2/n$. Hence $\mathbb{E}[n(\bar X - \mu)^2] = n\operatorname{Var}(\bar X) = \sigma^2$.
4. So $\mathbb{E}\big[\sum_i (X_i - \bar X)^2\big] = n\sigma^2 - \sigma^2 = (n - 1)\sigma^2$. Divide by $n - 1$. $\blacksquare$

Why. $\bar X$ is the number $c$ that makes $\sum_i (X_i - c)^2$ smallest. So the deviations from $\bar X$ are, on average, smaller than the deviations from the true $\mu$. Dividing by $n$ gives the mean $\frac{n-1}{n}\sigma^2$, which is too small: "one degree of freedom is spent to estimate $\mu$". The same proof, entry by entry, shows that the sample covariance matrix $C = \frac{1}{n-1} \sum_i (X_i - \bar X)(X_i - \bar X)^\top$ is unbiased for $\Sigma$.

Note: $S$ is a biased estimator of $\sigma$. The square root is concave, so $\mathbb{E}[S] < \sigma$ by Jensen's inequality (information theory 03) (unless $S^2$ is constant).

The covariance matrix and PCA #

$\Sigma$ is symmetric, so by the spectral theorem (linear algebra 06) $\Sigma = Q \Lambda Q^\top$, with orthonormal eigenvectors $q_1, \dots, q_d$ (the columns of $Q$) and eigenvalues $\lambda_1 \ge \dots \ge \lambda_d \ge 0$ (they are $\ge 0$ because $\Sigma$ is PSD). In this section only, $\Lambda$ is the diagonal matrix of eigenvalues; from lesson 05 on, $\Lambda$ is the precision matrix.

• The direction of largest variance. For a unit vector $u$, write $c = Q^\top u$; it is also a unit vector. Then $\operatorname{Var}(u^\top X) = u^\top \Sigma u = \sum_k \lambda_k c_k^2 \le \lambda_1 \sum_k c_k^2 = \lambda_1$, with equality at $u = q_1$. So $q_1$ is the direction of largest variance, and that variance is $\lambda_1$.
• The principal components are uncorrelated. $Y = Q^\top (X - \mu)$ has $\operatorname{Cov}(Y) = Q^\top \Sigma Q = \Lambda$, a diagonal matrix.
• Total variance. $\sum_i \operatorname{Var}(X_i) = \operatorname{tr}\Sigma = \sum_k \lambda_k$. The first $k$ components explain the fraction $(\lambda_1 + \dots + \lambda_k)/\operatorname{tr}\Sigma$.

In two dimensions, $\Sigma$ is drawn as an ellipse: its axes point along $q_1$ and $q_2$, with half-lengths $\sqrt{\lambda_1}$ and $\sqrt{\lambda_2}$ (one standard deviation). PCA on data does the same with the sample covariance $C$ (linear algebra 09).

Interactive figure: loading…

Answers to section 1. (1) No. The average is $a^\top X$ with $a = (\tfrac12, \tfrac12)$, so its variance is $a^\top \Sigma a = \tfrac14 (5 + 5 + 2 \cdot 4) = 4.5$, only a little less than 5. The difference has variance $5 + 5 - 2 \cdot 4 = 2$. Most of the variance is shared by the two readings, and averaging cannot remove a shared part. (2) (a) Yes, by the tower rule. (b) No. The weighted average of the class variances is only $\mathbb{E}[\operatorname{Var}(X \mid Y)]$. It misses the between part $\operatorname{Var}(\mathbb{E}[X \mid Y])$, which is positive unless all class means are equal.

4. Worked example #

(a) The homework table. We use the table of section 2: rows $X = 0, 1, 2$, columns $Y = 0, 1$.

1. Marginals. Row sums: $p_X = (0.3, 0.4, 0.3)$. Column sums: $p_Y = (0.4, 0.6)$.
2. Mean and variance of $X$. $\mathbb{E}[X] = 0 \cdot 0.3 + 1 \cdot 0.4 + 2 \cdot 0.3 = 1$. $\mathbb{E}[X^2] = 0 + 1 \cdot 0.4 + 4 \cdot 0.3 = 1.6$. So $\operatorname{Var}(X) = 1.6 - 1^2 = 0.6$.
3. Not independent. $p(2, 0) = 0$, but $p_X(2)\,p_Y(0) = 0.3 \cdot 0.4 = 0.12$. One cell is enough to show dependence.
4. Conditionals. Column $Y = 0$ divided by $0.4$: $p(x \mid Y = 0) = (0.5, 0.5, 0)$. Its mean is $0.5$, and its variance is $\mathbb{E}[X^2 \mid Y = 0] - 0.5^2 = 0.5 - 0.25 = 0.25$. Column $Y = 1$ divided by $0.6$: $p(x \mid Y = 1) = (1/6, 1/3, 1/2)$. Its mean is $1/3 + 2 \cdot 1/2 = 4/3$. $\mathbb{E}[X^2 \mid Y = 1] = 1/3 + 4 \cdot 1/2 = 7/3$, so the variance is $7/3 - 16/9 = 5/9$.
5. Tower rule. $\mathbb{E}[\mathbb{E}[X \mid Y]] = 0.4 \cdot 0.5 + 0.6 \cdot \tfrac43 = 0.2 + 0.8 = 1 = \mathbb{E}[X]$. ✓
6. Total variance. Within: $\mathbb{E}[\operatorname{Var}(X \mid Y)] = 0.4 \cdot 0.25 + 0.6 \cdot \tfrac59 = 0.1 + \tfrac13 = \tfrac{13}{30} \approx 0.433$. Between: $\operatorname{Var}(\mathbb{E}[X \mid Y]) = 0.4 \cdot 0.5^2 + 0.6 \cdot (\tfrac43)^2 - 1^2 = 0.1 + \tfrac{16}{15} - 1 = \tfrac16 \approx 0.167$. Sum: $\tfrac{13}{30} + \tfrac{5}{30} = \tfrac{18}{30} = 0.6 = \operatorname{Var}(X)$. ✓
7. Covariance and correlation. $\mathbb{E}[XY]$ only gets contributions from the column $Y = 1$: $\mathbb{E}[XY] = 1 \cdot 1 \cdot 0.2 + 2 \cdot 1 \cdot 0.3 = 0.8$. With $\mathbb{E}[Y] = 0.6$: $\operatorname{Cov}(X, Y) = 0.8 - 1 \cdot 0.6 = 0.2$. $Y$ is Bernoulli(0.6), so $\operatorname{Var}(Y) = 0.6 \cdot 0.4 = 0.24$, and $\rho = 0.2/\sqrt{0.6 \cdot 0.24} \approx 0.527$. Going to the tutorial goes together with solving more problems. (This is an association, not a proof that the tutorial causes it.)

(b) Two thermometers. Model: the true temperature $T$ has variance 4, and thermometer $i$ reads $X_i = T + e_i$, where the errors $e_1, e_2$ have variance 1 and are independent of each other and of $T$.

1. $\operatorname{Var}(X_i) = \operatorname{Var}(T) + \operatorname{Var}(e_i) = 4 + 1 = 5$.
2. $\operatorname{Cov}(X_1, X_2) = \operatorname{Cov}(T + e_1, T + e_2) = \operatorname{Var}(T) = 4$, by bilinearity (all other terms are 0).
3. So $\Sigma = \begin{pmatrix} 5 & 4 \\ 4 & 5 \end{pmatrix}$ and $\rho = 4/5 = 0.8$.
4. Average and difference at once. Take $A = \begin{pmatrix} 1/2 & 1/2 \\ 1 & -1 \end{pmatrix}$. Then $A\Sigma = \begin{pmatrix} 4.5 & 4.5 \\ 1 & -1 \end{pmatrix}$ and $A\Sigma A^\top = \begin{pmatrix} 4.5 & 0 \\ 0 & 2 \end{pmatrix}$. The average has variance 4.5, the difference 2, and they are uncorrelated. The average removes half of the error variance ($4 + \tfrac12$), but none of the variance of $T$: that part is signal, not noise.
5. PCA. The eigenvalues of $\Sigma$ are $9$ (eigenvector $(1, 1)/\sqrt 2$) and $1$ (eigenvector $(1, -1)/\sqrt 2$). Check: $\Sigma (1, 1)^\top = (9, 9)^\top$ and $\Sigma (1, -1)^\top = (1, -1)^\top$. The first component, the "common temperature" direction, explains $9/10 = 90\%$ of the total variance $\operatorname{tr}\Sigma = 10$.

(c) Bessel's correction by hand. Data $2, 4, 9$. The mean is $\bar x = 15/3 = 5$. The squared deviations are $9, 1, 16$, with sum $26$. So $S^2 = 26/2 = 13$. Dividing by $n = 3$ would give $8.67$.

5. In code #

NumPy checks the hand computations. The notebook for this lesson has these cells and the exercises.

import numpy as np

# (a) joint PMF: rows X = 0, 1, 2; columns Y = 0, 1
P = np.array([[0.2, 0.1],
              [0.2, 0.2],
              [0.0, 0.3]])
x, y = np.array([0, 1, 2]), np.array([0, 1])
pX, pY = P.sum(axis=1), P.sum(axis=0)        # marginals: row and column sums
print(pX, pY, np.linalg.matrix_rank(P))      # [0.3 0.4 0.3] [0.4 0.6] 2  -> rank 2: dependent
cond = P / pY                                # column j is p(x | Y = y_j)
m = x @ cond                                 # E[X | Y = y]
v = x**2 @ cond - m**2                       # Var(X | Y = y)
print(m.round(4), pY @ m)                    # [0.5    1.3333] 1.0   (tower rule)
print(round(pY @ v, 4), round(pY @ m**2 - (pY @ m)**2, 4))   # 0.4333 0.1667  (sum 0.6)
EX = x @ pX
print(round(x @ P @ y - EX * (pY @ y), 4))   # 0.2 = Cov(X, Y)

# (b) Cov(AX + b) = A Σ Aᵀ, exactly and on samples
Sigma = np.array([[5.0, 4.0], [4.0, 5.0]])
A, b = np.array([[0.5, 0.5], [1.0, -1.0]]), np.array([3.0, -1.0])
print(A @ Sigma @ A.T)                       # [[4.5 0. ] [0.  2. ]]
rng = np.random.default_rng(1)
X = rng.multivariate_normal([0.0, 0.0], Sigma, size=200_000)   # rows are samples
print(np.cov((X @ A.T + b).T).round(2))      # about [[4.5 0.] [0. 2.]]
print(np.linalg.eigvalsh(Sigma))             # [1. 9.]: the PCA variances

# uncorrelated but dependent: X uniform on {-1, 0, 1}, Y = X^2
xs = rng.choice([-1, 0, 1], size=200_000)
print(round(np.corrcoef(xs, xs**2)[0, 1], 2))   # about 0.0, yet Y is a function of X

# (c) Bessel: divide by n - 1
d = np.array([2.0, 4.0, 9.0])
print(d.var(ddof=1), round(d.var(ddof=0), 2))   # 13.0 8.67

np.cov treats each row as one variable unless you pass rowvar=False; that is why the code passes the transpose. It divides by $n - 1$ by default.

6. Where it shows up #

• Deep ensembles and mixtures. An ensemble of $M$ Gaussian heads with means $\mu_m$ and variances $\sigma_m^2$ predicts with variance $\frac1M \sum_m \sigma_m^2 + \frac1M \sum_m (\mu_m - \bar\mu)^2$. This is the law of total variance with $Y$ = the member. The first part is often called aleatoric (noise in the data), the second epistemic (disagreement of the models).
• Regression with the squared loss estimates the conditional mean $\mathbb{E}[Y \mid X]$ (the best predictor above). The irreducible error is $\mathbb{E}[\operatorname{Var}(Y \mid X)]$.
• Mini-batch gradients. The mean of $B$ uncorrelated per-example gradients has $1/B$ times the variance of one. Correlated examples (duplicates, consecutive frames of a video) reduce less, because of the covariance terms.
• PCA, whitening and EOFs. PCA uses the eigenvectors of the (sample) covariance. Whitening multiplies by $\Lambda^{-1/2} Q^\top$, which gives the identity covariance by $\operatorname{Cov}(AX) = A\Sigma A^\top$. In climate science, empirical orthogonal functions (EOFs) are PCA of space–time anomalies.
• Ensemble Kalman filter. Weather forecasting estimates the forecast covariance by the sample covariance of an ensemble of model runs, with $1/(N - 1)$ (ML for science 06). With few members it has rank at most $N - 1$ and is noisy.
• Sums of correlated errors. The error of a total (a portfolio, rainfall over a region, a sum of pixel errors) needs the covariance terms. Positive correlation makes the total much more uncertain than the sum of variances suggests.

7. Common mistakes #

• Factorising the formula but not the support. $p(x, y) = 2$ on $0 < x < y < 1$ looks like a product, but $X$ and $Y$ are dependent. The set where $p > 0$ must be a rectangle.
• Treating $\mathbb{E}[X \mid Y]$ as a number. It is a random variable, a function of $Y$. Only $\mathbb{E}[X \mid Y = y]$, for a fixed $y$, is a number.
• Averaging only the group variances. This misses the between part $\operatorname{Var}(\mathbb{E}[X \mid Y])$.
• Reading zero correlation as independence. $X$ and $X^2$ are uncorrelated (for symmetric $X$) and fully dependent. A correlation near 0 means "no linear relation", not "no relation".
• Forgetting the covariance terms in $\operatorname{Var}(X + Y)$. The variances add only for uncorrelated variables.
• Dividing by $n$ and calling the result unbiased, or claiming that $S$ (not $S^2$) is unbiased for $\sigma$.
• The rowvar default of np.cov. A data matrix of shape (samples, features) needs rowvar=False or a transpose. Otherwise you get an $n \times n$ matrix of covariances between samples.

8. Check yourself #

1. $(X, Y)$ has density $p(x, y) = 2$ on $0 < x < y < 1$. Find $p_X$ and $\mathbb{E}[Y \mid X = x]$, and check the tower rule for $\mathbb{E}[Y]$.
   Answer

   $p_X(x) = \int_x^1 2\,dy = 2(1 - x)$ on $(0, 1)$. Given $X = x$, $Y$ is uniform on $(x, 1)$, so $\mathbb{E}[Y \mid X = x] = (1 + x)/2$. $\mathbb{E}[X] = \int_0^1 x \cdot 2(1 - x)\,dx = 1/3$, so $\mathbb{E}[(1 + X)/2] = (1 + 1/3)/2 = 2/3$. Directly: $p_Y(y) = 2y$ and $\mathbb{E}[Y] = \int_0^1 y \cdot 2y\,dy = 2/3$. ✓
2. A deep ensemble has 5 members with predicted means $1.0, 1.2, 0.8, 1.1, 0.9$, each with predicted variance $0.04$. What is the variance of the ensemble's predictive mixture?
   Answer

   Within part: $\mathbb{E}[\operatorname{Var}] = 0.04$. The means average to $1.0$; their variance (weights $1/5$) is $(0 + 0.04 + 0.04 + 0.01 + 0.01)/5 = 0.02$. Total: $0.04 + 0.02 = 0.06$.
3. $\operatorname{Var}(X) = 1$, $\operatorname{Var}(Y) = 2$ and $\operatorname{Cov}(X, Y) = 0.5$. Find $\operatorname{Var}(2X - 3Y + 5)$.
   Answer

   The shift 5 does not matter. $\operatorname{Var}(2X - 3Y) = 4 \cdot 1 + 9 \cdot 2 + 2 \cdot 2 \cdot (-3) \cdot 0.5 = 4 + 18 - 6 = 16$. Or with $a = (2, -3)$: $a^\top \Sigma a$.
4. $\operatorname{Var}(X) = \operatorname{Var}(Y) = 1$ and $\rho = 0.9$. Find $\operatorname{Var}(X - Y)$.
   Answer

   $1 + 1 - 2 \cdot 0.9 = 0.2$. The difference of two strongly correlated readings removes most of the shared variance.
5. Can $\begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix}$ be a covariance matrix? Show it with one vector $a$.
   Answer

   No. With $a = (1, -1)$: $a^\top \Sigma a = 1 + 1 - 2 \cdot 2 = -2$. This would be $\operatorname{Var}(X_1 - X_2) < 0$, which is impossible. (Its eigenvalues are $3$ and $-1$; and "$\rho$" would be $2 > 1$.)
6. You compute the sample variance of 5 numbers with np.var(x). On average, by what factor is it too small?
   Answer

   `np.var` divides by $n$ by default (`ddof=0`), so $\mathbb{E} = \frac{n-1}{n}\sigma^2 = 0.8\,\sigma^2$. Use `np.var(x, ddof=1)` for the unbiased $S^2$.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. How do you get the marginal $p_X$ and the conditional $p(x \mid y)$ from a joint PMF table?

   Show answer

2. What does independence look like in a joint PMF table?

   Show answer

3. Why is $\mathbb{E}[X \mid Y]$ a random variable and not a number?

   Show answer

4. State the tower rule.

   Show answer

5. State the law of total variance and name its two parts.

   Show answer

6. Give a pair of variables that are uncorrelated but dependent.

   Show answer

7. Write $\operatorname{Var}(X + Y)$ and $\operatorname{Var}(X - Y)$.

   Show answer

8. What is $\operatorname{Cov}(AX + b)$ when $\operatorname{Cov}(X) = \Sigma$?

   Show answer

9. Why is every covariance matrix positive semidefinite?

   Show answer

10. Why does the sample variance divide by $n - 1$?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Are uncorrelated variables independent?" — No: $X$ and $X^2$ for symmetric $X$. They are for jointly Gaussian variables (lesson 05); Gaussian marginals alone are not enough.
• "Why divide by $n - 1$?" — The deviations from $\bar X$ are on average smaller than those from $\mu$; $\mathbb{E}\sum_i (X_i - \bar X)^2 = (n - 1)\sigma^2$.
• "$n$ variables all have variance $\sigma^2$ and the same pairwise correlation $\rho$. How negative can $\rho$ be?" — $\operatorname{Var}(\sum_i X_i) = n\sigma^2\big(1 + (n - 1)\rho\big) \ge 0$, so $\rho \ge -1/(n - 1)$.
• "Your ensemble members agree but each is unsure. What does the predictive variance look like?" — Large within part, small between part; the law of total variance gives the split.
• "What does PCA compute, and what does the top eigenvalue mean?" — The eigenvectors of the sample covariance; $\lambda_1$ is the largest variance of any unit-length projection $u^\top X$.

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution:

1. From a joint PMF array: marginals, the conditional columns, and an independence test.
2. Verify the law of total variance on simulated grouped data (three classes).
3. Write the sample covariance matrix from scratch (no np.cov) and check Bessel's correction on 20,000 samples of size 5.
4. Check $\operatorname{Cov}(AX + b) = A\Sigma A^\top$ on samples.
5. PCA from the covariance matrix: eigen-decomposition compared with sklearn.decomposition.PCA.

11. Watch and read #

Lecture 19: Joint, Conditional, and Marginal Distributions | Statistics 110Harvard University · Joe Blitzstein, Statistics 110 · 50 min

Joint, marginal and conditional distributions, discrete and continuous, with many small examples.

12. Iterated ExpectationsMIT OpenCourseWare · 6.041 Probabilistic Systems Analysis · 48 min

The conditional expectation as a random variable, the tower rule and the law of total variance.

Lecture 21: Covariance and Correlation | Statistics 110Harvard University · Joe Blitzstein, Statistics 110 · 49 min

Properties of covariance, the variance of a sum, and why correlation lies between −1 and 1.

The Covariance Matrix : Data Science Basicsritvikmath · 11 min

A short visual summary of the covariance matrix and how it describes the shape of a data cloud.

• Joseph Blitzstein and Jessica Hwang, Introduction to Probability, 2nd edition (free PDF from the authors): §7.1 (joint, marginal and conditional), §7.3 (covariance and correlation), §9.1–9.5 (conditional expectation, Adam's and Eve's laws).
• Kevin Murphy, Probabilistic Machine Learning: An Introduction, free PDF: §3.1 (covariance, correlation, uncorrelated does not imply independent).
• NumPy documentation of np.cov (rowvar and ddof).