Bayesian inference: priors, posteriors, MAP and prediction

Before this lesson: lesson 01 (Bayes' rule), lesson 03 (the beta, gamma and Poisson distributions), lesson 05 (Gaussian conditioning, completing the square) and lesson 08 (likelihood, MLE, bias–variance, the exponential family). Lesson 09 helps: the data precision of a Gaussian mean is $nI(\mu)$. Time: about 75 minutes, plus the notebook.

Why this matters #

• Decisions need uncertainty. Is variant B better than A? How wide is a forecast band? Where should active learning or Bayesian optimisation sample next? All need a distribution over the unknowns, not only a point.
• Regularisation is a prior. Ridge regression and weight decay are MAP estimates with a Gaussian prior; the lasso uses a Laplace prior. Add-one smoothing of counts is a posterior mean.
• Exploration. Thompson sampling draws a parameter from each arm's Beta posterior and plays the best draw (reinforcement learning 03).
• Science. The analysis step of a Kalman filter in weather forecasting is a Gaussian–Gaussian update (ML for science 06); Gaussian processes are Bayesian regression (ML for science 07).
• Ground truth for approximate methods. Conjugate models have exact posteriors, so they are the test cases for MCMC (lesson 15) and variational inference.

1. Predict first #

2. The picture #

Bayes' rule for a parameter #

Treat the unknown parameter $\theta$ as a random variable with a prior density $p(\theta)$. After the data $\mathcal D = (x_1, \dots, x_n)$, Bayes' rule (lesson 01) gives

• The likelihood $p(\mathcal D \mid \theta)$ is the same function as in lesson 08.
• The evidence (or marginal likelihood) $p(\mathcal D)$ does not depend on $\theta$. It only rescales the product so that the posterior integrates to 1. So we often write $p(\theta \mid \mathcal D) \propto p(\mathcal D \mid \theta)\, p(\theta)$.
• In logs, $\log p(\theta \mid \mathcal D) = \log p(\theta) + \sum_i \log p(x_i \mid \theta)$ + const. The prior is one term, and each data point adds one more.

On a grid. For one parameter you can do this with a computer and nothing else: put 1000 grid points on $[0, 1]$, multiply the prior and the likelihood at each point, and divide by the sum (times the grid step). The result is the posterior, up to the grid error. A coin with 7 heads in 10 flips and a uniform prior: the product is $p^7(1-p)^3$, and after normalisation it is the Beta(8, 4) density.

The coin: pseudo-counts #

A Beta$(a, b)$ prior on a probability $p$ has density $\propto p^{a-1}(1-p)^{b-1}$, mean $\frac{a}{a+b}$, and acts like $a$ earlier heads and $b$ earlier tails. Observing $k$ heads in $n$ flips just adds counts: the posterior is Beta$(a + k,\ b + n - k)$.

Sequential = batch. Today's posterior is tomorrow's prior. Update Beta($a, b$) with $k_1$ heads in $n_1$ flips, then with $k_2$ in $n_2$: you get Beta($a + k_1 + k_2,\ b + n_1 + n_2 - k_1 - k_2$), the same as one update with all the data, in any order. A running A/B dashboard and a Kalman filter work this way.

The Gaussian mean: precisions add #

For a Gaussian mean with known noise, the prior and the data each contribute a precision (inverse variance). The posterior precision is their sum, and the posterior mean is the precision-weighted average of the prior mean and $\bar x$: the more precise source gets more weight.

3. Definitions and main results #

A prior family $\mathcal F$ is conjugate to a likelihood if a prior in $\mathcal F$ always gives a posterior in $\mathcal F$. Then the update is a rule for the parameters of the prior (the hyperparameters).

Beta–Bernoulli #

Data: $x_1, \dots, x_n \in \{0, 1\}$ i.i.d. Bernoulli($p$), with $k = \sum_i x_i$. Prior: $p \sim \text{Beta}(a, b)$ with $a, b > 0$.

1. Likelihood: $p(\mathcal D \mid p) = p^k (1-p)^{n-k}$.
2. Posterior $\propto p^k (1-p)^{n-k} \cdot p^{a-1}(1-p)^{b-1} = p^{a+k-1}(1-p)^{b+n-k-1}$.
3. This is the kernel of a Beta density, so $p \mid \mathcal D \sim \text{Beta}(a', b')$ with $a' = a + k$, $b' = b + n - k$. $\blacksquare$

Summaries of the posterior:

• Posterior mean $\frac{a'}{a'+b'} = w\,\frac{a}{a+b} + (1-w)\,\frac{k}{n}$ with $w = \frac{a+b}{a+b+n}$: a weighted average of the prior mean and the MLE. The prior counts like $a + b$ observations.
• MAP (the mode) $\frac{a'-1}{a'+b'-2}$, for $a', b' > 1$. With $a = b = 1$ it equals the MLE $k/n$.
• Credible interval: the 2.5% and 97.5% quantiles of Beta($a', b'$) (no closed form; scipy.stats.beta.ppf).

Point estimates: posterior mean, MAP and the regularised MLE #

The maximum a posteriori (MAP) estimate is the mode of the posterior. Take logs of Bayes' rule; the evidence does not depend on $\theta$, so

L2 = Gaussian prior. Linear regression $y = Xw + \varepsilon$ with noise $\varepsilon \sim \mathcal N(0, \sigma^2 I)$ and prior $w \sim \mathcal N(0, \sigma_w^2 I)$:

Multiply by $2\sigma^2$: this is ridge regression, $\|y - Xw\|^2 + \lambda\|w\|^2$ with $\lambda = \sigma^2/\sigma_w^2$. A Laplace prior $p(w_j) \propto e^{-|w_j|/b}$ gives the lasso, $\|y - Xw\|^2 + \lambda\|w\|_1$ with $\lambda = 2\sigma^2/b$. Weight decay in plain SGD is an L2 penalty; with Adam the two differ, which is why AdamW decouples them (optimization 06).

Which summary? Each point estimate is the best answer to a different loss: the posterior mean minimises the expected squared error, the posterior median the expected absolute error, and the MAP is the limit for a 0–1 loss. For a Gaussian posterior all three coincide.

The MAP depends on the parametrisation. A density changes with a Jacobian (lesson 05); a likelihood does not. So the MLE is invariant (lesson 08), but the MAP is not. Example: a help desk records 5 waiting times between calls with sum $S = 10$ minutes; model Exponential($\lambda$); flat prior on the rate $\lambda$.

1. In $\lambda$: the posterior is $\propto \lambda^5 e^{-10\lambda}$, with mode $\lambda_{\text{MAP}} = 5/10 = 0.5$, equal to the MLE.
2. In the mean waiting time $\theta = 1/\lambda$: $|d\lambda/d\theta| = 1/\theta^2$, so the same prior becomes $p(\theta) \propto 1/\theta^2$. The posterior is $\propto \theta^{-5}e^{-10/\theta} \cdot \theta^{-2} = \theta^{-7}e^{-10/\theta}$.
3. Set the derivative of $-7\log\theta - 10/\theta$ to zero: $-7/\theta + 10/\theta^2 = 0$, so $\theta_{\text{MAP}} = 10/7 \approx 1.43$ minutes, not $1/0.5 = 2$ minutes.

A "flat prior" is not neutral: flat in $\lambda$ is not flat in $1/\lambda$. Posterior quantiles (such as the median) move correctly under a monotone change of variable; the mode and the mean in general do not.

Shrinkage: when a prior beats the MLE #

The posterior mean and the MAP pull the MLE toward the prior: they are shrinkage estimators. Judge them as estimators, with the bias–variance decomposition of lesson 08, at a fixed true $\theta$.

Gaussian mean. Data $x_i \sim \mathcal N(\mu, \sigma^2)$, $\sigma^2$ known; prior $\mathcal N(\mu_0, \tau_0^2)$. The posterior mean is $\hat\mu_{\text{B}} = w\bar x + (1 - w)\mu_0$ with $w = \frac{n}{n + \sigma^2/\tau_0^2}$ (derived below).

1. $\mathbb{E}[\hat\mu_{\text{B}}] = w\mu + (1 - w)\mu_0$, so the bias is $(1 - w)(\mu_0 - \mu)$.
2. $\operatorname{Var}(\hat\mu_{\text{B}}) = w^2\sigma^2/n$, smaller than the variance $\sigma^2/n$ of $\bar x$.
3. $\text{MSE}(\hat\mu_{\text{B}}) = w^2\frac{\sigma^2}{n} + (1 - w)^2(\mu - \mu_0)^2$, while $\text{MSE}(\bar x) = \sigma^2/n$ for every $\mu$.
4. Solving $\text{MSE}(\hat\mu_{\text{B}}) < \sigma^2/n$ gives exactly $(\mu - \mu_0)^2 < \frac{\sigma^2}{n} + 2\tau_0^2$.

So the prior helps when its guess $\mu_0$ is roughly right and hurts when it is far off. It trades a little bias for a larger cut in variance.

Gaussian–Gaussian (known variance) #

Data: $x_1, \dots, x_n$ i.i.d. $\mathcal N(\mu, \sigma^2)$ with $\sigma^2$ known. Prior: $\mu \sim \mathcal N(\mu_0, \tau_0^2)$.

1. $\log p(\mu \mid \mathcal D) = -\frac{(\mu - \mu_0)^2}{2\tau_0^2} - \sum_i \frac{(x_i - \mu)^2}{2\sigma^2} + \text{const}$.
2. Collect the powers of $\mu$: $-\tfrac12 \mu^2 \Big(\frac{1}{\tau_0^2} + \frac{n}{\sigma^2}\Big) + \mu\Big(\frac{\mu_0}{\tau_0^2} + \frac{n\bar x}{\sigma^2}\Big) + \text{const}$.
3. Complete the square: $-\tfrac12 A\mu^2 + B\mu = -\tfrac{A}{2}(\mu - B/A)^2 + \text{const}$, the log-density of a Gaussian with precision $A$ and mean $B/A$.
4. So $\mu \mid \mathcal D \sim \mathcal N(\mu_n, \tau_n^2)$ with

Posterior precision = prior precision + data precision. The data precision $n/\sigma^2$ is $nI(\mu)$, the Fisher information of lesson 09. The prior is worth $\sigma^2/\tau_0^2$ observations. The posterior is Gaussian, so its mean, median and mode (the MAP) coincide.

Conjugate priors and the exponential family #

Every exponential family has a conjugate prior. As a function of $\eta$, the likelihood of $n$ observations is $\exp\big(\eta^\top s - nA(\eta)\big)$ with $s = \sum_i T(x_i)$ (lesson 08). Take a prior of the same form, with hyperparameters $\tau$ and $\nu > 0$: $p(\eta) \propto \exp\big(\eta^\top\tau - \nu A(\eta)\big)$. The posterior is $\propto \exp\big(\eta^\top(\tau + s) - (\nu + n)A(\eta)\big)$, the same family with the update $\tau \leftarrow \tau + \sum_i T(x_i)$, $\nu \leftarrow \nu + n$. The prior acts like $\nu$ pseudo-observations.

Gamma–Poisson. Counts $x_i \sim$ Poisson($\lambda$), prior $\lambda \sim$ Gamma($a, b$) (shape $a$, rate $b$). The posterior is $\propto \lambda^{S}e^{-n\lambda} \cdot \lambda^{a-1}e^{-b\lambda}$ with $S = \sum_i x_i$, that is Gamma($a + S,\ b + n$): mean $\frac{a + S}{b + n}$, MAP $\frac{a + S - 1}{b + n}$. The prior is worth $b$ observation periods with $a$ events in total.

The posterior predictive distribution #

To predict a new observation $x^*$, average the model over the posterior:

It has two sources of spread: the noise of $x^*$ for a fixed $\theta$, and the remaining uncertainty about $\theta$. By the law of total variance (lesson 04),

A plug-in prediction $p(x^* \mid \hat\theta)$ keeps only the first term, so it is too narrow.

• Gaussian. $x^* = \mu + \varepsilon^*$ with $\mu \mid \mathcal D \sim \mathcal N(\mu_n, \tau_n^2)$ and independent noise $\varepsilon^* \sim \mathcal N(0, \sigma^2)$. A sum of independent Gaussians is Gaussian: $x^* \mid \mathcal D \sim \mathcal N(\mu_n,\ \sigma^2 + \tau_n^2)$.
• One coin flip. $P(x^* = 1 \mid \mathcal D) = \int_0^1 p\; \text{Beta}(p; a', b')\, dp = \frac{a'}{a'+b'}$, the posterior mean.
• $m$ future flips. The number of heads is beta-binomial, with mean $m\bar p$ and variance $m\bar p(1-\bar p)\,\frac{a'+b'+m}{a'+b'+1}$, where $\bar p = \frac{a'}{a'+b'}$. The last factor is above 1 for $m > 1$: all $m$ flips share the same unknown $p$, so their errors do not cancel.

Credible intervals and confidence intervals #

• A 95% credible interval $C$ satisfies $P(\theta \in C \mid \mathcal D) = 0.95$: a probability statement about $\theta$, given the observed data and the prior.
• A 95% confidence interval is a random interval $C(X)$ with $P_\theta(\theta \in C(X)) = 0.95$ for every $\theta$ (lesson 06): a statement about the procedure over repeated data sets. For one realised interval, $\theta$ is either in it or not.

They can agree numerically (a flat prior on a Gaussian mean gives $\bar x \pm 1.96\sigma/\sqrt n$ for both), but they answer different questions. Averaged over $\theta$ drawn from the prior, a credible interval covers $\theta$ exactly 95% of the time. At a fixed $\theta$ far from the prior it can cover much less: with the prior Beta(1, 19) (mean 0.05) and a true rate of 0.12, the 95% credible interval covers the truth with probability $0.72$ at $n = 10$, $0.92$ at $n = 100$ and $0.95$ at $n = 1000$ (exact sums over $k$).

The prior's influence vanishes (usually). The log-likelihood is a sum of $n$ terms and grows like $n$; the log-prior stays fixed. In the conjugate formulas, the prior weights $\frac{a+b}{a+b+n}$ and $\frac{1/\tau_0^2}{1/\tau_0^2 + n/\sigma^2}$ both go to 0.

Answers to section 1. (1) The prior precision is $1/1 = 1$ and the data precision is $n/\sigma^2 = 8/4 = 2$. The posterior precision is 3, so the variance is $1/3$ and the mean is $\frac{1 \cdot 0 + 2 \cdot 3}{3} = 2$: two thirds of the way from 0 to 3, because the data are twice as precise as the prior. (2) Wider: $\mathcal N(\mu_n, \sigma^2 + \tau_n^2) = \mathcal N(2, 4 + 1/3) = \mathcal N(2, 13/3)$, against the plug-in variance $\sigma^2 = 4$. The plug-in drops the parameter-uncertainty term $\tau_n^2$. Here $\tau_n^2 \approx \sigma^2/n \to 0$, so the gap vanishes as $n$ grows.

4. Worked example #

A/B test with a historical prior. Past experiments put conversion rates near 5%, so the team uses the prior Beta(1, 19): mean $0.05$, worth 20 users. Variant B converts $k = 12$ of $n = 100$ users; the control A converts 8 of 100.

1. Posterior of B. Beta($1 + 12,\ 19 + 88$) = Beta(13, 107).
2. Posterior mean. $13/120 \approx 0.108$. Check with the weights: $w = 20/120 = 1/6$ on the prior, and $\frac16 \cdot 0.05 + \frac56 \cdot 0.12 = 0.0083 + 0.1 \approx 0.108$ ✓.
3. MAP and MLE. MAP $= \frac{13 - 1}{120 - 2} = 12/118 \approx 0.102$. MLE $= 0.12$.
4. Credible interval. The 2.5% and 97.5% quantiles of Beta(13, 107) are $0.059$ and $0.170$. The Wald interval of lesson 09 is $0.12 \pm 1.96\sqrt{0.12 \cdot 0.88/100} = [0.056,\ 0.184]$: centred on the MLE, with no pull toward the prior.
5. Predictive. The next user converts with probability $13/120 \approx 0.108$. For the next 50 users, the beta-binomial variance is $50 \cdot 0.108 \cdot 0.892 \cdot \frac{170}{121} \approx 6.79$; the plug-in Binomial(50, 0.12) has variance $50 \cdot 0.12 \cdot 0.88 = 5.28$.
6. Is B better than A? The posterior of A is Beta($1 + 8,\ 19 + 92$) = Beta(9, 111). Draw many pairs $(p_A, p_B)$ from the two posteriors and count how often $p_B > p_A$: about $0.82$. (Numerical integration of $\int f_B(p)\,F_A(p)\,dp$, with the density $f_B$ and the CDF $F_A$, gives $0.819$.) The posterior mean of the lift $p_B - p_A$ is $\frac{13}{120} - \frac{9}{120} \approx 0.033$.
7. Expected loss. If you ship B and B is worse, you lose $p_A - p_B$. The expected loss $\mathbb{E}[\max(p_A - p_B, 0)]$ is about $0.004$ (0.4 percentage points); for shipping A it is about $0.037$. A common rule: ship when the expected loss is below a threshold you accept.

5. In code #

SciPy checks the conjugate update on a grid, the worked example and the A/B probability. The notebook has these cells and the exercises.

import numpy as np
from scipy import stats, integrate

a, b = 1, 19                       # prior Beta(1, 19): mean 0.05, worth 20 users
k, n = 12, 100                     # variant B: 12 conversions in 100 users

# grid posterior = normalised prior x likelihood; compare with the conjugate Beta(a + k, b + n - k)
p = np.linspace(1e-6, 1 - 1e-6, 200_001)
unnorm = stats.beta.pdf(p, a, b) * stats.binom.pmf(k, n, p)
grid_post = unnorm / (unnorm.sum() * (p[1] - p[0]))
print(np.abs(grid_post - stats.beta.pdf(p, a + k, b + n - k)).max() < 1e-6)   # True

post_B = stats.beta(a + k, b + n - k)                                          # Beta(13, 107)
print(round(post_B.mean(), 4), post_B.ppf([0.025, 0.975]).round(4))            # 0.1083 [0.0595 0.1695]
print(round(stats.betabinom(50, a + k, b + n - k).var(), 2), round(stats.binom(50, k / n).var(), 2))  # 6.79 5.28

# Bayesian A/B: control A had 8 conversions in 100 users, same prior
post_A = stats.beta(a + 8, b + 92)
rng = np.random.default_rng(0)
pA, pB = post_A.rvs(1_000_000, random_state=rng), post_B.rvs(1_000_000, random_state=rng)
print(round((pB > pA).mean(), 3), round(np.maximum(pA - pB, 0).mean(), 4))    # 0.82 0.0036
exact = integrate.quad(lambda q: post_B.pdf(q) * post_A.cdf(q), 0, 1)[0]
print(round(exact, 3))                                                         # 0.819

# Gaussian-Gaussian: prior N(0, 1), 8 readings with variance 4 and mean 3
prec = 1 / 1 + 8 / 4
print(round(1 / prec, 3), (0 / 1 + 8 * 3 / 4) / prec, round(4 + 1 / prec, 3))  # 0.333 2.0 4.333

The Monte Carlo probability $0.82$ has a standard error of about $\sqrt{0.82 \cdot 0.18/10^6} \approx 0.0004$.

6. Where it shows up #

• Regularisation. Ridge regression and weight decay are MAP estimates under a Gaussian prior ($\lambda = \sigma^2/\sigma_w^2$); the lasso is the MAP under a Laplace prior (its posterior mean is not sparse). Add-one (Laplace) smoothing of counts in n-gram language models and naive Bayes is the posterior mean under a uniform Dirichlet prior.
• Bandits and recommender systems. Thompson sampling keeps a Beta posterior per arm (ad, article, variant), draws one $p$ from each, and shows the arm with the largest draw. Arms with uncertain posteriors get explored automatically.
• Empirical Bayes for many small groups. Click rates of thousands of small segments, batting averages, ratings of products with few reviews: fit one Beta prior to all groups, then shrink each raw rate toward the overall rate. The shrunk rates have a lower total MSE than the raw MLEs (the James–Stein effect).
• Data assimilation and Kalman filters. Forecast = prior, observations = likelihood, analysis = posterior; with Gaussians, the update is the precision-weighted average above, in many dimensions (ML for science 06).
• Uncertainty in deep learning. Laplace approximations, Bayesian last layers and deep ensembles approximate a posterior predictive; Gaussian processes give it exactly (ML for science 07). Bayesian linear regression is in lesson 13.
• Bayesian A/B tests. Report $P(p_B > p_A \mid \mathcal D)$, the posterior of the lift, and the expected loss. Caution: a rule like "stop when $P(p_B > p_A \mid \mathcal D) > 0.95$", checked every day, still declares a winner more often than 5% of the time when A and B are equal (lesson 12).

7. Common mistakes #

• Plug-in predictions. $p(x^* \mid \hat\theta)$ ignores the parameter uncertainty and gives prediction intervals that are too narrow, most of all with little data.
• Adding variances instead of precisions. Posterior precision = prior precision + data precision.
• The wrong ridge constant. $\lambda = \sigma^2/\sigma_w^2$ (noise variance over prior variance), not $\sigma_w^2$.
• Treating the MAP as "the Bayesian answer". It is one point: it ignores the posterior width, it depends on the parametrisation, and it can sit on the boundary (a Beta($1, b$) prior and zero successes give $\hat p_{\text{MAP}} = 0$).
• "There is a 95% probability that $\theta$ is in this confidence interval." That is the credible-interval statement, and it needs a prior.
• A prior that excludes the truth. Zero prior density can never be repaired by data. A prior centred far from the truth dominates small samples (coverage 0.72 at $n = 10$ above).
• A "non-informative" flat prior. Flat in one parametrisation is not flat in another ($\lambda$ vs $1/\lambda$), so it still carries information.

8. Check yourself #

1. Uniform prior Beta(1, 1); a new variant has 0 conversions in 30 users. Give the posterior, its mean and MAP, and the 95% upper credible bound.
   Answer

   Beta(1, 31). Mean $1/32 \approx 0.031$; MAP 0. The CDF of Beta(1, $b$) is $1 - (1-p)^b$, so the upper bound solves $(1-p)^{31} = 0.05$: $p_u = 1 - 0.05^{1/31} \approx 0.092$. Unlike the Wald interval of lesson 09, it does not collapse to a point.
2. Show that updating a Gaussian mean with batch 1 and then batch 2 gives the same posterior as one update with both batches.
   Answer

   Each update adds $n_j/\sigma^2$ to the precision and $n_j\bar x_j/\sigma^2$ to "precision times mean". Sums do not depend on order or grouping, so in both cases the final precision is $1/\tau_0^2 + (n_1 + n_2)/\sigma^2$ and the final precision times mean is $\mu_0/\tau_0^2 + (n_1\bar x_1 + n_2\bar x_2)/\sigma^2$.
3. For the kitchen scale of lesson 08 ($n = 4$ readings with $\bar x = 1.3$, noise variance $\sigma^2 = 4$) and the prior $\mathcal N(0, 1)$, compute the posterior mean and variance. For which true offsets $\mu$ does the posterior mean have a smaller MSE than $\bar x$?
   Answer

   Data precision $4/4 = 1$, prior precision 1, so $w = 0.5$: posterior mean $0.5 \cdot 1.3 = 0.65$, variance $1/2$. MSE of the posterior mean: $w^2\sigma^2/n + (1-w)^2\mu^2 = 0.25 + 0.25\mu^2$; MSE of $\bar x$: $1$. It wins when $\mu^2 < 3$, that is $|\mu| < 1.73$, in agreement with $(\mu - \mu_0)^2 < \sigma^2/n + 2\tau_0^2 = 3$.
4. Does the MAP of the mean waiting time $\theta = 1/\lambda$ equal $1/\lambda_{\text{MAP}}$? Explain with the example of section 3.
   Answer

   No. With a flat prior on $\lambda$, $\lambda_{\text{MAP}} = 0.5$, so $1/\lambda_{\text{MAP}} = 2$. In $\theta$ the prior picks up the Jacobian $1/\theta^2$, and the posterior $\theta^{-7}e^{-10/\theta}$ has its mode at $10/7 \approx 1.43$. Densities change with a Jacobian; the mode moves. The MLE has no Jacobian and is invariant.
5. For one future binary outcome the posterior predictive is Bernoulli($\bar p$). In what sense is the Bayesian predictive "wider" than the plug-in Bernoulli($\hat p$)?
   Answer

   For a single trial both are Bernoulli distributions; they differ only in the probability ($\bar p$ is pulled toward the prior mean), and a Bernoulli's variance is fixed by its mean. The extra spread appears for $m > 1$ future trials: the beta-binomial variance $m\bar p(1-\bar p)\frac{a'+b'+m}{a'+b'+1}$ is larger than the binomial $m\bar p(1-\bar p)$, because all $m$ trials share the same unknown $p$.
6. "The 95% credible interval for the conversion rate is [0.059, 0.170]." Write one correct sentence that explains it to a product manager, and one sentence that a confidence interval would justify instead.
   Answer

   Credible: "Given our prior and these data, there is a 95% probability that the true rate is between 5.9% and 17.0%." Confidence: "If we repeated this experiment many times, intervals built this way would contain the true rate in 95% of the repetitions."

9. Interview angle #

• "A coin shows 3 heads in 3 tosses. What is P(heads next)?" — The MLE says 1. With a uniform prior, the posterior is Beta(4, 1) and Laplace's rule gives $4/5$.
• "Derive ridge regression from a Bayesian view." — Gaussian likelihood and Gaussian prior; the negative log-posterior is $\frac{1}{2\sigma^2}\|y - Xw\|^2 + \frac{1}{2\sigma_w^2}\|w\|^2$, so $\lambda = \sigma^2/\sigma_w^2$.
• "MLE, MAP, full Bayes: what is the difference?" — Maximise the likelihood; maximise the posterior (a regularised MLE, still one point); keep the whole posterior and average predictions over it.
• "How do you compute $P(B \text{ better than } A)$?" — Sample both Beta posteriors and count, or compute $\int f_B(p)\,F_A(p)\,dp$.
• "When does the prior still matter with lots of data?" — When it excludes the truth, when the number of parameters grows with $n$ (hierarchical models, many small groups), and in directions the data do not inform.
• "Credible vs confidence interval?" — A probability about $\theta$ given the data and prior, vs a coverage property of a procedure.

10. Practice #

The notebook for this lesson has five exercises, each with a check cell and a folded solution:

1. A grid posterior for a coin, compared with the conjugate Beta posterior (maximum absolute difference).
2. map_gaussian_mean and the Gaussian–Gaussian posterior, checked against the precision formula and a numerical optimiser.
3. MSE of the MLE and of the MAP for a coin, by simulation over a grid of true $p$, against the exact sums.
4. The beta-binomial predictive for the next 50 users against the plug-in binomial.
5. A Bayesian A/B test: $P(p_B > p_A)$ by Monte Carlo and by numerical integration, and the expected loss.

11. Watch and read #

• Kevin Murphy, Probabilistic Machine Learning: An Introduction, free PDF: §4.5 (MAP estimation and regularisation), §4.6 (Bayesian statistics: conjugate priors, beta-binomial, Gaussian, credible intervals) and §4.7.5 (confidence intervals are not credible).
• Allen Downey, Think Bayes 2 (free online, with code): grid posteriors, conjugate priors and Bayesian A/B tests.
• Kevin Murphy, Probabilistic Machine Learning: Advanced Topics, free PDF: §3.4 (conjugate priors, including the exponential family).