Fisher information, the Cramér–Rao bound and asymptotic normality of the MLE

Before this lesson: lesson 08 (likelihood, MLE, bias and MSE, the exponential family), lesson 06 (law of large numbers, central limit theorem, the delta method, confidence intervals) and lesson 04 (covariance and the Cauchy–Schwarz inequality $\operatorname{Cov}(X, Y)^2 \le \operatorname{Var}(X)\operatorname{Var}(Y)$). From calculus: second derivatives and Hessians (calculus 03). Time: about 70 minutes, plus the notebook.

---

Why this matters #

• Error bars of fitted models. The "std. error" column of a logistic or Poisson regression, the error bars on the parameters of a physical model, the interval for a conversion rate: all are $1/\sqrt{\text{information}}$.
• Planning experiments. Before you collect data you can ask how many users an A/B test needs, or where to put a sensor, by computing the Fisher information of the planned design.
• Deep learning. The Fisher information matrix is the local metric of the KL divergence between models. The natural gradient, K-FAC (optimization 12), the trust region of TRPO, elastic weight consolidation for continual learning and Laplace approximations of network posteriors all use it.
• Knowing the limits. The theory explains why rare events need huge samples and why the usual interval breaks for a rate near 0.

1. Predict first #

2. The picture #

Curvature is precision #

One Gaussian reading. A kitchen scale reports $x \sim \mathcal N(\theta, \sigma^2)$, where $\theta$ is the true weight and $\sigma$ the known noise. As a function of $\theta$, the log-likelihood is a downward parabola:

A small $\sigma$ gives a sharp peak: the reading pins $\theta$ down. A large $\sigma$ gives a flat peak. The Fisher information of one reading is this curvature: $I(\theta) = 1/\sigma^2$.

More data, more curvature. For $n$ independent readings, the log-likelihoods add, so the curvatures add: $-\ell_n''(\theta) = n/\sigma^2$. The width of the peak, $1/\sqrt{\text{curvature}} = \sigma/\sqrt n$, is exactly the standard error of the sample mean. In this case the whole theory is one sentence: precision = curvature of the log-likelihood.

Interactive figure: loading…

The score: how sensitive is the data to $\theta$? #

The score is the derivative of the log-likelihood of one observation, $s(\theta; x) = \partial_\theta \log p(x \mid \theta)$. For the scale, $s = (x - \theta)/\sigma^2$. Before the data are drawn, the score at the true $\theta$ is a random variable. Two facts:

• its mean is 0: on average, the data do not push $\theta$ up or down from the truth;
• its variance is $\operatorname{Var}(X)/\sigma^4 = 1/\sigma^2$, the same number as the curvature.

If a small change of $\theta$ changes the probabilities of the outcomes a lot, the score swings widely from sample to sample: the data are informative. Section 3 shows that "variance of the score = expected curvature" holds for every regular model.

Interactive figure: loading…

Information adds over independent sources #

Two scales with noise standard deviations $\sigma_1 = 1$ g and $\sigma_2 = 2$ g weigh the same object. Their information is $1/1 + 1/4 = 1.25$, so (section 3) no unbiased estimate can have a variance below $1/1.25 = 0.8$ g². Weight each reading by its precision: $\hat\theta = \frac{1 \cdot x_1 + 0.25 \cdot x_2}{1.25} = 0.8\,x_1 + 0.2\,x_2$. Its variance is $0.8^2 \cdot 1 + 0.2^2 \cdot 4 = 0.64 + 0.16 = 0.8$ g²: it attains the bound. The plain average $\frac12(x_1 + x_2)$ has variance $\frac14(1 + 4) = 1.25$ g², worse than the first scale alone.

3. Definitions and main results #

Setting. A model $p(x \mid \theta)$ with $\theta \in \Theta \subseteq \mathbb{R}$ for now. The data $X_1, \dots, X_n$ are i.i.d. from $p(\cdot \mid \theta)$ for one true $\theta$, and $\mathbb{E}_\theta$ is the expectation under it.

• The score of one observation is $s(\theta; x) = \partial_\theta \log p(x \mid \theta)$. The score of the sample is $s_n(\theta) = \sum_{i=1}^n s(\theta; X_i) = \ell_n'(\theta)$, the derivative of the log-likelihood $\ell_n$.
• The Fisher information of one observation is $I(\theta) = \mathbb{E}_\theta[s(\theta; X)^2]$.
• Regularity conditions, used in every result below: (R1) the support $\{x : p(x \mid \theta) > 0\}$ does not depend on $\theta$; (R2) $\log p(x \mid \theta)$ is twice differentiable in $\theta$; (R3) $\int p(x \mid \theta)\,dx$ can be differentiated twice under the integral sign; (R4) $\theta$ is an interior point of $\Theta$, and $0 < I(\theta) < \infty$.

The two information identities #

Lemma. Under (R1)–(R4): $\mathbb{E}_\theta[s(\theta; X)] = 0$, so $I(\theta) = \operatorname{Var}_\theta(s)$; and $I(\theta) = -\mathbb{E}_\theta[\partial_\theta^2 \log p(X \mid \theta)]$.

Proof.

1. $s = \partial_\theta \log p = \frac{\partial_\theta p}{p}$, so $\mathbb{E}[s] = \int \frac{\partial_\theta p}{p}\, p\, dx = \int \partial_\theta p\, dx$.
2. By (R3), swap the derivative and the integral: $\int \partial_\theta p\, dx = \partial_\theta \int p\, dx = \partial_\theta 1 = 0$. Hence $I = \mathbb{E}[s^2] = \operatorname{Var}(s)$.
3. Differentiate $\log p$ twice (quotient rule): $\partial_\theta^2 \log p = \partial_\theta\Big(\frac{\partial_\theta p}{p}\Big) = \frac{\partial_\theta^2 p}{p} - \Big(\frac{\partial_\theta p}{p}\Big)^2 = \frac{\partial_\theta^2 p}{p} - s^2$.
4. Take expectations. As in steps 1–2, $\mathbb{E}\big[\frac{\partial_\theta^2 p}{p}\big] = \partial_\theta^2 \int p\, dx = 0$. So $\mathbb{E}[\partial_\theta^2 \log p] = -\mathbb{E}[s^2] = -I(\theta)$. $\blacksquare$

Information adds. The sample score $s_n$ is a sum of $n$ independent terms with mean 0 and variance $I(\theta)$, so $\operatorname{Var}(s_n) = nI(\theta)$. Likewise $-\mathbb{E}[\ell_n''] = nI(\theta)$.

Model	Score $s(\theta; x)$	$I(\theta)$	Bound $1/(nI(\theta))$
Bernoulli($p$)	$\frac{x}{p} - \frac{1 - x}{1 - p}$	$\frac{1}{p(1-p)}$	$\frac{p(1-p)}{n}$
Poisson($\lambda$)	$\frac{x}{\lambda} - 1$	$\frac{1}{\lambda}$	$\frac{\lambda}{n}$
Exponential, rate $\lambda$	$\frac{1}{\lambda} - x$	$\frac{1}{\lambda^2}$	$\frac{\lambda^2}{n}$
$\mathcal N(\mu, \sigma^2)$, $\sigma^2$ known	$\frac{x - \mu}{\sigma^2}$	$\frac{1}{\sigma^2}$	$\frac{\sigma^2}{n}$

Check the Bernoulli row with the second identity: $\partial_p^2 \log p(x \mid p) = -\frac{x}{p^2} - \frac{1 - x}{(1 - p)^2}$. With $\mathbb{E}[X] = p$, minus its expectation is $\frac{1}{p} + \frac{1}{1 - p} = \frac{1}{p(1-p)}$ ✓.

Interactive figure: loading…

Several parameters. For $\theta \in \mathbb{R}^d$ the score is the gradient $\nabla_\theta \log p$, and the Fisher information matrix is $I(\theta) = \mathbb{E}[s s^\top] = -\mathbb{E}[\nabla_\theta^2 \log p]$, a $d \times d$ covariance matrix, so symmetric and positive semidefinite. For $\mathcal N(\mu, \sigma^2)$ with $\theta = (\mu, \sigma^2)$: $I = \operatorname{diag}\big(\frac{1}{\sigma^2},\ \frac{1}{2\sigma^4}\big)$.

Reparametrisation. If $\theta = g(\phi)$ with $g$ smooth, the chain rule gives $s_\phi = s_\theta\, g'(\phi)$, so $I_\phi(\phi) = I_\theta(g(\phi))\, g'(\phi)^2$. For the Bernoulli in log-odds $\eta = \log\frac{p}{1-p}$, $\frac{dp}{d\eta} = p(1-p)$, so $I(\eta) = \frac{1}{p(1-p)}\,p^2(1-p)^2 = p(1-p)$. This is $A''(\eta) = \operatorname{Var}(X)$ from lesson 08: for a natural parameter, $\partial_\eta^2 \log p = -A''(\eta)$ does not depend on $x$, so the Fisher information is the Hessian of the log-partition function.

The Cramér–Rao bound #

Theorem (Cramér–Rao). Let $T = T(X_1, \dots, X_n)$ be an unbiased estimator of $\theta$ ($\mathbb{E}_\theta[T] = \theta$ for all $\theta$) with finite variance. Assume (R1)–(R4), and that $\int T\, p_n\, dx$ can be differentiated under the integral, where $p_n(x \mid \theta) = \prod_i p(x_i \mid \theta)$. Then

Proof.

1. Unbiasedness says $\int T(x)\, p_n(x \mid \theta)\, dx = \theta$ for every $\theta$.
2. Differentiate both sides in $\theta$. On the left use $\partial_\theta p_n = s_n\, p_n$ (the score of the sample): $\int T\, s_n\, p_n\, dx = 1$, that is, $\mathbb{E}[T s_n] = 1$.
3. Since $\mathbb{E}[s_n] = 0$ (lemma), $\operatorname{Cov}(T, s_n) = \mathbb{E}[T s_n] - \mathbb{E}[T]\,\mathbb{E}[s_n] = 1$.
4. Cauchy–Schwarz for covariances (lesson 04): $\operatorname{Cov}(T, s_n)^2 \le \operatorname{Var}(T)\operatorname{Var}(s_n)$, so $1 \le \operatorname{Var}(T) \cdot nI(\theta)$.
5. Divide by $nI(\theta) > 0$. $\blacksquare$

Why it is true, in words. An unbiased estimator must move one-for-one with $\theta$ (step 2). The only way the data "see" a change of $\theta$ is through the score. If the score is noisy and weak (small $I$), the estimator must be noisy too.

Efficiency. An unbiased estimator whose variance equals $1/(nI(\theta))$ is efficient. Equality in Cauchy–Schwarz means that $T - \theta$ is a multiple of $s_n$. For every $\theta$ at once this happens only in a one-parameter exponential family, when $T$ is the sample mean of the sufficient statistic. So $\bar X$ is efficient for a Bernoulli $p$, a Poisson $\lambda$ and a Gaussian $\mu$. For an exponential rate $\lambda$, no unbiased estimator attains $\lambda^2/n$: the best one, $\frac{n-1}{n\bar X}$, has variance $\frac{\lambda^2}{n-2}$.

Biased estimators. If $\mathbb{E}_\theta[T] = \theta + b(\theta)$, step 2 gives $\operatorname{Cov}(T, s_n) = 1 + b'(\theta)$, and

A shrinkage estimator $c\bar X$ with $c < 1$ has $b'(\theta) = c - 1 < 0$, so its variance can go below $1/(nI)$ without breaking any rule.

Several parameters. For an unbiased $T \in \mathbb{R}^d$: $\operatorname{Cov}(T) \succeq I(\theta)^{-1}/n$ (in the Loewner order of linear algebra 07). For one coordinate the bound is $[I(\theta)^{-1}]_{kk}/n$, and $[I^{-1}]_{kk} \ge 1/I_{kk}$: not knowing the other parameters costs precision. For the gamma shape $k$ at $(k, \beta) = (2, 3)$, $[I^{-1}]_{kk} \approx 6.90$ but $1/I_{kk} \approx 1.55$. With an unknown rate, the shape is more than four times harder to estimate.

Asymptotic normality of the MLE #

Theorem. Assume (R1)–(R4), a correct and identifiable model, a consistent MLE $\hat\theta_n$, and that $\partial_\theta^2 \log p$ is continuous in $\theta$ and bounded by an integrable function near the true $\theta$. Then

In words: for large $n$, $\hat\theta_n \approx \mathcal N\big(\theta, \frac{1}{nI(\theta)}\big)$. The MLE is approximately unbiased, Gaussian, and reaches the Cramér–Rao bound. It is asymptotically efficient.

Proof sketch: a Taylor expansion of the score

1. $\hat\theta_n$ solves $\ell_n'(\hat\theta_n) = 0$. Expand around the true $\theta$ (mean value theorem): $0 = \ell_n'(\theta) + \ell_n''(\tilde\theta)(\hat\theta_n - \theta)$, with $\tilde\theta$ between $\hat\theta_n$ and $\theta$.
2. Rearrange: $\sqrt n(\hat\theta_n - \theta) = \dfrac{\ell_n'(\theta)/\sqrt n}{-\ell_n''(\tilde\theta)/n}$.
3. Numerator: $\ell_n'(\theta)$ is a sum of $n$ i.i.d. scores with mean 0 and variance $I(\theta)$. By the central limit theorem (lesson 06), $\ell_n'(\theta)/\sqrt n \to \mathcal N(0, I(\theta))$ in distribution.
4. Denominator: by the law of large numbers, $-\ell_n''(\theta)/n \to -\mathbb{E}[\partial_\theta^2 \log p] = I(\theta)$ in probability. The same limit holds at $\tilde\theta$, because $\tilde\theta \to \theta$ and $\partial_\theta^2 \log p$ is continuous and dominated near $\theta$.
5. Slutsky's lemma: the ratio converges to $\mathcal N(0, I)/I$, which is $\mathcal N(0, I^{-1})$. $\blacksquare$

Interactive figure: loading…

Standard errors and Wald intervals #

For large $n$, $\hat\theta \approx \mathcal N(\theta, 1/(nI(\theta)))$. Replace the unknown $\theta$ by $\hat\theta$ in the variance:

with $z_{0.975} = 1.96$ for 95%. This is the Wald interval. The observed information $-\ell_n''(\hat\theta)$ is the curvature of the actual log-likelihood at its peak; the expected information $nI(\hat\theta)$ is the formula evaluated at $\hat\theta$. Both are valid; software usually reports the observed one, because it is what a numerical optimiser gives you (the Hessian of the negative log-likelihood).

• Several parameters. $\widehat{\operatorname{SE}}_k = \sqrt{[H^{-1}]_{kk}}$, where $H$ is the Hessian of the negative log-likelihood at $\hat\theta$ (or $nI(\hat\theta)$).
• Functions of $\theta$. The MLE of $g(\theta)$ is $g(\hat\theta)$ (lesson 08), and the delta method (lesson 06) gives $\widehat{\operatorname{SE}}(g(\hat\theta)) = |g'(\hat\theta)|\,\widehat{\operatorname{SE}}(\hat\theta)$.

Interactive figure: loading…

When the theory fails #

• The support depends on $\theta$ (R1 fails). For Uniform($0, \theta$), the MLE is $\max_i X_i$. Its error $n(\theta - \hat\theta)$ converges to an Exponential distribution with mean $\theta$: the error is of order $1/n$, not $1/\sqrt n$, and it is not Gaussian. The "score" $\partial_\theta \log(1/\theta) = -1/\theta$ has mean $-1/\theta \ne 0$, so the lemma fails too.
• The true $\theta$ is on the boundary (R4 fails): a variance component equal to 0, a mixture weight equal to 0. The limit is half a point mass at 0 and half a half-Gaussian.
• A singular or nearly singular $I(\theta)$: directions the data cannot identify, such as label switching in mixtures, over-parametrised neural networks, or "sloppy" parameter combinations of physical models.
• Small samples. The Gaussian approximation is poor when $n$ is small or $\theta$ is near the edge, as the coverage figure shows. Use Wilson, likelihood-ratio or exact intervals there.
• A misspecified model. The MLE converges to the KL-closest $\theta^*$ (lesson 08), and its variance is the "sandwich" $H^{-1} J H^{-1}/n$ with $H = -\mathbb{E}[\nabla^2 \log p_\theta]$ and $J = \operatorname{Var}(\nabla \log p_\theta)$ under the true distribution. In general $H \ne J$; this gives the robust standard errors of statistics software.

Fisher information as the metric of the KL divergence

Expand $\mathrm{KL}(p_\theta \,\|\, p_{\theta + \delta})$ to second order in $\delta$. The zeroth-order term is 0 (same distribution), the first-order term is $-\delta^\top \mathbb{E}_\theta[s] = 0$, and the second-order term is $\frac12 \delta^\top I(\theta)\, \delta$:

$$\mathrm{KL}(p_\theta \,\|\, p_{\theta + \delta}) \approx \tfrac12\, \delta^\top I(\theta)\, \delta .$$

So the Fisher matrix measures how far apart two nearby models are, in the units that matter for prediction, and it does not depend on how you parametrise the model. The natural gradient step $-\eta\, I(\theta)^{-1}\nabla L(\theta)$ is the steepest-descent direction in this geometry. K-FAC approximates $I(\theta)$ for a neural network by Kronecker products per layer; TRPO constrains each policy update by this quadratic KL. The Laplace approximation of a posterior, $\mathcal N(\hat\theta, H^{-1})$ with $H$ the Hessian of the negative log-posterior at its mode, is the Bayesian twin of the Wald interval (calculus 11). The "empirical Fisher" $\frac1n\sum_i g_i g_i^\top$, built from gradients at the observed labels, is a different matrix (Kunstner et al., 2019).

Answers to section 1. (1) Yes, the same $p$. $\operatorname{Var}(\hat p) = p(1-p)/n$ is largest at $p = 1/2$, and $I(p) = 1/(p(1-p))$ is smallest there ($I = 4$). They are reciprocals (up to $n$) because $\hat p = \bar X$ attains the Cramér–Rao bound. Near $p = 0$ the information is large, but the relative error $\operatorname{SE}/p = \sqrt{(1-p)/(np)}$ grows: at $p = 0.01$ and $n = 100$ it is about 1. Rare events need many trials. (2) Yes, but only with bias. The constant estimator $T = 0$ has variance 0, and $c\bar X$ with $c < 1$ has variance $c^2\sigma^2/n$. The bound $\sigma^2/n$ holds only for unbiased estimators.

4. Worked example #

A help desk counts its support tickets per day. Model: the daily counts $X_i$ are i.i.d. Poisson($\lambda$). Data: $n = 50$ days with $\sum_i x_i = 150$ tickets.

1. Score. $\log p(x \mid \lambda) = x \log\lambda - \lambda - \log x!$, so $s = \frac{x}{\lambda} - 1$, and $\mathbb{E}[s] = \frac{\lambda}{\lambda} - 1 = 0$ ✓.
2. Information, two routes.
   • Curvature: $-\partial_\lambda^2 \log p = \frac{x}{\lambda^2}$, with expectation $\frac{\lambda}{\lambda^2} = \frac{1}{\lambda}$.
   • Variance of the score: $\operatorname{Var}\big(\frac{X}{\lambda}\big) = \frac{\lambda}{\lambda^2} = \frac{1}{\lambda}$ ✓.
3. MLE and the bound. $\ell_n'(\lambda) = \frac{\sum_i x_i}{\lambda} - n = 0$ gives $\hat\lambda = \bar x = 150/50 = 3$. The Cramér–Rao bound is $\frac{1}{nI} = \frac{\lambda}{n}$. $\bar X$ is unbiased with variance exactly $\frac{\lambda}{n}$: it is efficient at every $n$, not only for large $n$.
4. Wald interval. $\widehat{\operatorname{SE}} = \sqrt{\hat\lambda/n} = \sqrt{3/50} = \sqrt{0.06} \approx 0.245$. The 95% interval is $3 \pm 1.96 \cdot 0.245 = 3 \pm 0.480$, that is $[2.52,\ 3.48]$ tickets per day.
5. Observed information. $-\ell_n''(\hat\lambda) = \frac{\sum_i x_i}{\hat\lambda^2} = \frac{150}{9} \approx 16.7$, and $\frac{n}{\hat\lambda} = \frac{50}{3} \approx 16.7$: both versions of the standard error agree here.
6. Delta method. The probability of a day without tickets is $g(\lambda) = e^{-\lambda}$. Its MLE is $e^{-3} \approx 0.0498$, and $|g'(\hat\lambda)| = e^{-3}$, so $\widehat{\operatorname{SE}} = e^{-3} \cdot 0.245 \approx 0.0122$.
7. Design. To get $\operatorname{SE} \le 0.1$ at $\lambda \approx 3$ you need $\lambda/n \le 0.01$, so $n \ge 300$ days.

5. In code #

SciPy and finite differences check the two identities for the Poisson model, then the exact coverage of a Wald interval and the failure for Uniform($0, \theta$). The notebook has these cells and the exercises.

import numpy as np
from scipy import stats

rng = np.random.default_rng(0)
lam, h = 3.0, 1e-4
x = rng.poisson(lam, size=1_000_000)               # one million "days" of ticket counts
logp = lambda l: stats.poisson.logpmf(x, l)
score = (logp(lam + h) - logp(lam - h)) / (2 * h)                 # d/dλ log p(x | λ)
curv = (logp(lam + h) - 2 * logp(lam) + logp(lam - h)) / h**2     # d²/dλ² log p(x | λ)
print(round(score.mean(), 3))                         # 0.0: E[score] = 0
print(round(score.var(), 3), round(-curv.mean(), 3))  # 0.333 0.333: both are 1/λ

def wald_coverage(n, p):
    """Exact probability that the Wald 95% interval for a Bernoulli rate contains p."""
    k = np.arange(n + 1)
    p_hat = k / n
    half = 1.96 * np.sqrt(p_hat * (1 - p_hat) / n)
    covered = (p_hat - half <= p) & (p <= p_hat + half)
    return float(stats.binom.pmf(k, n, p)[covered].sum())

print(round(wald_coverage(400, 0.12), 3))   # 0.95: the asymptotics work here
print(round(wald_coverage(20, 0.05), 3))    # 0.639: n p = 1, far from Gaussian

# Uniform(0, θ): the error of the MLE max(x) is of order 1/n and exponential, not Gaussian
theta, n = 1.0, 1000
err = n * (theta - rng.uniform(0, theta, size=(5000, n)).max(axis=1))
print(round(err.mean(), 2), round(stats.skew(err), 1))   # 1.03 2.0: mean ≈ θ, skewness ≈ 2 (Exponential)

At $n = 20$ and $p = 0.05$, the outcome $k = 0$ has probability $0.95^{20} \approx 0.36$, and for $k = 0$ the Wald interval is $[0, 0]$. The Wilson interval inverts the test $|\hat p - p| \le 1.96\sqrt{p(1-p)/n}$: it uses the variance at the candidate $p$, not at $\hat p$, and its coverage here is $0.925$.

6. Where it shows up #

• Regression tables. The standard errors of a logistic regression are $\sqrt{[(X^\top W X)^{-1}]_{kk}}$ with $W = \operatorname{diag}(\hat p_i(1 - \hat p_i))$: the inverse Fisher matrix. Huge standard errors signal a nearly singular matrix: collinear features, or separable data where the MLE runs off to infinity.
• A/B tests and sample sizes. The usual standard error $\sqrt{\hat p(1 - \hat p)/n}$ is $1/\sqrt{nI(\hat p)}$. Low baseline rates need a large $n$ for a useful relative precision (section 1). Sample-size formulas (interview 09) are this calculation.
• Optimisation of neural networks. The natural gradient, K-FAC and Shampoo-style preconditioners approximate the Fisher matrix; Adam's second-moment estimate is a crude diagonal version of the empirical Fisher.
• Continual learning and pruning. Elastic weight consolidation adds $\frac{\lambda}{2}\sum_j F_{jj}(\theta_j - \theta_j^{\text{old}})^2$ with the diagonal Fisher $F$, to protect the weights that mattered for an old task. Fisher and Hessian traces rank the sensitivity of layers for pruning and for choosing quantisation bit widths.
• ML for science. Experimental design chooses sensor positions or measurement times to maximise $\det I$ (D-optimal) or minimise $\operatorname{tr} I^{-1}$ (A-optimal). In seismology, magnitudes above a threshold are roughly exponential with rate $\beta$, and $I(\beta) = 1/\beta^2$ gives the standard error $\hat b/\sqrt n$ of the Gutenberg–Richter $b$-value. A nearly singular Fisher matrix of a physical model shows which parameter combinations the data cannot constrain.

7. Common mistakes #

• Forgetting $n$. $I(\theta)$ is per observation; the bound for $n$ observations is $1/(nI(\theta))$.
• Applying the bound to biased estimators. Shrinkage, MAP and ridge estimators can have a variance below $1/(nI)$. Compare MSEs, not variances.
• Thinking the MLE attains the bound at finite $n$. In general only asymptotically: $1/\bar X$ for an exponential rate is biased at every finite $n$.
• Using $1/I_{kk}$ instead of $[I^{-1}]_{kk}$ when other parameters are estimated too. The interval comes out too narrow.
• Ignoring the regularity conditions: a support that depends on $\theta$, a parameter on the boundary, a singular Fisher matrix.
• Trusting Wald intervals when $n\hat p$ is small. At $\hat p = 0$ the interval is $[0, 0]$. Use a Wilson, likelihood-ratio or exact interval.
• Calling the empirical Fisher "the Fisher". They agree in expectation only for a correct model at the true parameter.

8. Check yourself #

1. $X \sim \mathcal N(\mu, \sigma^2)$ with $\mu$ known; the parameter is $v = \sigma^2$. Compute $I(v)$ and the Cramér–Rao bound. Does $\frac1n \sum_i (X_i - \mu)^2$ attain it?
   Answer

   $\log p = -\frac12 \log v - \frac{(x-\mu)^2}{2v} + \text{const}$, so $\partial_v^2 \log p = \frac{1}{2v^2} - \frac{(x-\mu)^2}{v^3}$. Minus its expectation is $-\frac{1}{2v^2} + \frac{v}{v^3} = \frac{1}{2v^2}$. So $I(\sigma^2) = 1/(2\sigma^4)$ and the bound is $2\sigma^4/n$. The estimator is unbiased ($\mu$ is known) with variance $\operatorname{Var}((X-\mu)^2)/n = 2\sigma^4/n$: it attains the bound. It is the sample mean of the sufficient statistic $(X - \mu)^2$.
2. Why does the Cramér–Rao bound not apply to Uniform($0, \theta$)? Which proof step breaks?
   Answer

   The support $[0, \theta]$ depends on $\theta$, so differentiating $\int_0^\theta p\,dx = 1$ produces a boundary term, and step 2 of the lemma fails. Here $s = \partial_\theta(-\log\theta) = -1/\theta$ for every $x$ in the support, so $\mathbb{E}[s] = -1/\theta \ne 0$. Indeed the unbiased estimator $\frac{n+1}{n}\max_i X_i$ has variance $\theta^2/(n(n+2))$, of order $1/n^2$.
3. Show that the Fisher information of a Bernoulli in log-odds $\eta$ is $p(1-p)$, and give the Wald standard error of $\hat\eta$.
   Answer

   Chain rule: $I(\eta) = I(p)\,(dp/d\eta)^2 = \frac{1}{p(1-p)}\, p^2(1-p)^2 = p(1-p)$. So $\widehat{\operatorname{SE}}(\hat\eta) = 1/\sqrt{n\hat p(1-\hat p)}$. The delta method gives the same: $\sqrt{\hat p(1-\hat p)/n}\cdot|d\eta/dp|$ with $d\eta/dp = 1/(p(1-p))$.
4. An ad gets 0 clicks in 50 views. Give the Wald 95% interval for the click rate $p$, say what is wrong with it, and give a better upper bound.
   Answer

   $\hat p = 0$, so $\widehat{\operatorname{SE}} = 0$ and the interval is $[0, 0]$: it claims certainty. The Gaussian approximation fails when $n\hat p = 0$, and $I(\hat p) = \infty$ there. An exact one-sided 95% upper bound solves $(1-p)^{50} = 0.05$: $p_u = 1 - 0.05^{1/50} \approx 0.058$, close to the "rule of three" $3/n = 0.06$.
5. For Gaussian data, the sample median has asymptotic variance $\frac{\pi\sigma^2}{2n}$. How many observations does it need to match the precision of the sample mean with $n = 1000$?
   Answer

   Its efficiency relative to the mean (which attains the bound) is $\frac{\sigma^2/n}{\pi\sigma^2/(2n)} = 2/\pi \approx 0.64$. So it needs about $1000 \cdot \pi/2 \approx 1571$ observations.
6. For a natural parameter $\eta$ of an exponential family, why is the observed information $-\ell_n''(\eta)$ equal to the expected information $nI(\eta)$ for every data set?
   Answer

   $\log p(x \mid \eta) = \log h(x) + \eta T(x) - A(\eta)$, so $\partial_\eta^2 \log p = -A''(\eta)$, which does not depend on $x$. Then $-\ell_n''(\eta) = nA''(\eta) = nI(\eta)$ for every data set. In other parametrisations (such as $\lambda$ for the Poisson) the two can differ away from the MLE.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Define the score.

   Show answer

2. Why is $\mathbb{E}_\theta[s] = 0$?

   Show answer

3. Give the two formulas for the Fisher information $I(\theta)$.

   Show answer

4. Fisher information of one Bernoulli($p$) trial?

   Show answer

5. State the Cramér–Rao bound.

   Show answer

6. What is the key step in the proof of the Cramér–Rao bound?

   Show answer

7. State the asymptotic normality of the MLE.

   Show answer

8. Write the 95% Wald interval.

   Show answer

9. How does information combine over independent observations or sensors?

   Show answer

10. Why does the theory fail for Uniform($0, \theta$)?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "What is Fisher information, intuitively?" — The expected curvature of the log-likelihood, equal to the variance of the score: how much one observation tells you about $\theta$.
• "Derive the Cramér–Rao bound." — $\operatorname{Cov}(T, s_n) = 1$ for unbiased $T$, then Cauchy–Schwarz.
• "Why is the standard error of the MLE $1/\sqrt{nI}$?" — CLT on the score, LLN on the second derivative, then Slutsky.
• "Can an estimator beat the Cramér–Rao bound?" — Yes, if it is biased (ridge, shrinkage, James–Stein) or if the model is not regular (Uniform($0, \theta$)).
• "Fisher, Hessian, empirical Fisher: what is the difference?" — The Fisher is the expected Hessian of the negative log-likelihood under the model, always positive semidefinite; the Hessian at the observed data can be indefinite; the empirical Fisher uses the observed labels and is a different matrix.
• "How many users does an A/B test need to estimate a 2% conversion rate within ±0.2 percentage points?" — Half-width $1.96\sqrt{p(1-p)/n} \le 0.002$ gives $n \ge 1.96^2 \cdot 0.0196/0.002^2 \approx 18{,}800$.

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution:

1. The score and the Fisher information matrix of the gamma distribution: closed form against finite differences of scipy.stats.gamma.logpdf.
2. The empirical variance of the MLE of an exponential rate against the Cramér–Rao value $\lambda^2/n$, for several $n$.
3. Exact coverage of the Wald interval for a Bernoulli rate at $p = 0.5$ and $p = 0.02$, and the Wilson interval.
4. Standard errors of a two-parameter model from the numerical Hessian of the negative log-likelihood.
5. Uniform($0, \theta$): show that $n(\theta - \hat\theta)$, not $\sqrt n(\theta - \hat\theta)$, has a stable distribution.

11. Watch and read #

A Visual Introduction to Fisher Information and the Cramér-Rao Lower BoundJacob Seifert · 9 min

The picture of this lesson: a sharper log-likelihood means more information and a smaller bound.

The Fisher InformationMutual Information · 17 min

The score, the two formulas for the information and why they agree, with clear animations.

6. Maximum Likelihood Estimation (cont.) and the Method of MomentsMIT OpenCourseWare · Philippe Rigollet, 18.650 Statistics for Applications · 79 min

The first half proves the asymptotic normality of the MLE with the Fisher information.

• Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul Grasman and Eric-Jan Wagenmakers, A Tutorial on Fisher Information (2017): §2 (design, tests and intervals from the MLE), Appendix B (asymptotic normality, efficiency and the Cramér–Rao bound), Appendix E (regularity conditions).
• Kevin Murphy, Probabilistic Machine Learning: Advanced Topics, free PDF: §3.3 (the Fisher information matrix and the asymptotic normality of the MLE).
• Frederik Kunstner, Lukas Balles and Philipp Hennig, Limitations of the Empirical Fisher Approximation for Natural Gradient Descent (2019): the Fisher, the generalised Gauss–Newton matrix and the empirical Fisher.