Probability and Statistics · 256

Glossary

The technical terms of this track, with short definitions (and the Persian term where available). Each term links to the lesson where it is introduced.

A/A test: An experiment where both arms get the same version. It checks the pipeline: about $\alpha$ of A/A tests should be significant. Introduced in: Lesson 12
A/B test: A randomised experiment that compares a control (A) and a treatment (B) on a metric, with users assigned to the arms at random. Introduced in: Lesson 12
acceptance probability: The probability $\alpha(x, x')$ that a Metropolis–Hastings proposal is accepted. Introduced in: Lesson 15
almost sure convergence: $X_n \to X$ almost surely if $P(X_n \to X) = 1$ : on almost every outcome, the sequence of numbers converges. Introduced in: Lesson 6
alternative hypothesis: $H_1$ : the claim that we accept when we reject $H_0$ , for example 'the means differ'. Introduced in: Lesson 11
antithetic variables: Use pairs of negatively correlated samples, such as $U$ and $1 - U$ . For a monotone $f$ the average of the pair has less variance than two independent samples. Introduced in: Lesson 14
aperiodic: The chain does not cycle with a fixed period; the period of a state is the greatest common divisor of the possible return times, and aperiodic means it is 1. Introduced in: Lesson 15
asymptotic normality of the MLE: Under regularity conditions, $\sqrt n(\hat\theta - \theta) \to \mathcal N(0, I(\theta)^{-1})$ , so the MLE is asymptotically efficient and $\widehat{\operatorname{SE}} = 1/\sqrt{nI(\hat\theta)}$ . Introduced in: Lesson 9
autocorrelation: The correlation $\rho_t$ between states of a stationary chain $t$ steps apart. Introduced in: Lesson 15
base rate: The probability of a condition in the population before any test; for a disease it is also called the prevalence. Introduced in: Lesson 1
base-rate fallacy: Ignoring the base rate and reading $P(E \mid H)$ as $P(H \mid E)$ . For a rare condition, most positives of an accurate test can still be false. Introduced in: Lesson 1
Bayes factor: The likelihood ratio of the data for two hypotheses or models. It is the factor that turns prior odds into posterior odds. Introduced in: Lesson 1
Bayes' rule: $P(H \mid E) = P(E \mid H)\,P(H) / P(E)$ . It turns the probability of the evidence given a hypothesis into the probability of the hypothesis given the evidence. Introduced in: Lesson 1
Bayesian linear regression: A Gaussian prior $\beta \sim \mathcal N(0, \tau^2 I)$ with the Gaussian linear model. The posterior is Gaussian, and its mean is the ridge estimate with penalty $\lambda = \sigma^2/\tau^2$ . Introduced in: Lesson 13
Benjamini–Hochberg procedure: Sort the $m$ p-values, find the largest $k$ with $p_{(k)} \le k\alpha/m$ , and reject the $k$ smallest. It controls the FDR at level $\alpha$ for independent tests. Introduced in: Lesson 11
Bernoulli distribution: A random variable with values 1 (success) with probability $p$ and 0 otherwise. Mean $p$ , variance $p(1 - p)$ . Introduced in: Lesson 3
Bessel's correction: Dividing the sum of squared deviations by $n - 1$ instead of $n$ . It makes the sample variance an unbiased estimate of $\sigma^2$ , because the deviations are measured from $\bar x$ , not from $\mu$ . Introduced in: Lesson 4
beta distribution: $\text{Beta}(\alpha, \beta)$ : a distribution on $(0, 1)$ with density proportional to $x^{\alpha - 1}(1 - x)^{\beta - 1}$ , mean $\alpha/(\alpha + \beta)$ . It models an unknown probability. Introduced in: Lesson 3
beta-binomial distribution: The posterior predictive of the number of successes in $m$ new trials when $p$ has a beta distribution. It is wider than a binomial with a fixed $p$ . Introduced in: Lesson 10
bias: $\operatorname{bias}(\hat\theta) = \mathbb{E}[\hat\theta] - \theta$ . An estimator with zero bias is unbiased. Introduced in: Lesson 8
binomial distribution: $\text{Bin}(n, p)$ : the number of successes in $n$ independent $\text{Bernoulli}(p)$ trials. $P(X = k) = \binom{n}{k}p^k(1 - p)^{n - k}$ , mean $np$ , variance $np(1 - p)$ . Introduced in: Lesson 3
Bonferroni correction: Test each of $m$ hypotheses at level $\alpha/m$ . By the union bound the FWER is at most $\alpha$ . Introduced in: Lesson 11
bootstrap: Estimate the sampling distribution of a statistic by resampling the data with replacement many times and recomputing the statistic on each resample. Introduced in: Lesson 12
bootstrap standard error: The standard deviation of the statistic over the bootstrap resamples. Introduced in: Lesson 12
calibrated: A probabilistic forecast is calibrated when its stated probabilities match observed frequencies. For continuous forecasts, the PIT values $F(y)$ of the outcomes are then uniform. Introduced in: Lesson 2
Cantelli's inequality: The one-sided Chebyshev bound $P(X - \mu \ge t) \le \sigma^2/(\sigma^2 + t^2)$ for $t > 0$ . Introduced in: Lesson 7
categorical distribution: One of $K$ classes, class $k$ with probability $p_k$ . It is the $K$ -class version of the Bernoulli; outcomes are often written as one-hot vectors. Introduced in: Lesson 3
central limit theorem (CLT): For i.i.d. variables with mean $\mu$ and variance $0 < \sigma^2 < \infty$ , $\sqrt n(\bar X_n - \mu)/\sigma \to \mathcal N(0, 1)$ in distribution. So $\bar X_n$ is approximately $\mathcal N(\mu, \sigma^2/n)$ for large $n$ . Introduced in: Lesson 6
chain rule of probability: $P(A \cap B) = P(A \mid B)\,P(B)$ ; for more events, $P(A_1 \cap \dots \cap A_n) = P(A_1)\,P(A_2 \mid A_1) \cdots P(A_n \mid A_1, \dots, A_{n-1})$ . Introduced in: Lesson 1
characteristic function: $\varphi_X(t) = \mathbb{E}[e^{itX}]$ . Unlike the MGF it exists for every distribution, and it determines the distribution. Introduced in: Lesson 3
Chebyshev's inequality: For $X$ with mean $\mu$ and variance $\sigma^2$ , $P(|X - \mu| \ge k) \le \sigma^2/k^2$ for every $k > 0$ . Introduced in: Lesson 6
Chernoff bound: $P(S \ge t) \le \inf_{\lambda > 0} e^{-\lambda t}\,\mathbb{E}[e^{\lambda S}]$ : Markov's inequality applied to $e^{\lambda S}$ , optimised over $\lambda$ . For sums of independent variables it gives exponential tails. Introduced in: Lesson 7
Cholesky factor: The lower-triangular matrix $L$ with positive diagonal and $\Sigma = LL^\top$ , for positive definite $\Sigma$ . It is used to sample ( $x = \mu + Lz$ ) and to compute the log-density stably. Introduced in: Lesson 5
coefficient of determination: $R^2 = 1 - \text{RSS}/\text{TSS}$ , the fraction of the variance of $y$ explained by the model. The adjusted $R^2$ penalises extra features. Introduced in: Lesson 13
collider: A common effect of two variables. Conditioning on a collider can make independent causes dependent. Introduced in: Lesson 1
common cause: An event or variable that influences two others. It can make them dependent even when they are independent given the cause. Introduced in: Lesson 1
concentration inequality: A non-asymptotic bound on the probability that a random variable (often a sum or mean) is far from its mean, valid for every $n$ . Introduced in: Lesson 7
conditional distribution: The distribution of $X$ when $Y = y$ is known: $p(x \mid y) = p(x, y)/p(y)$ for $p(y) > 0$ . Introduced in: Lesson 4
conditional expectation: $\mathbb{E}[X \mid Y = y]$ is the mean of the conditional distribution. $\mathbb{E}[X \mid Y]$ is the random variable $g(Y)$ with $g(y) = \mathbb{E}[X \mid Y = y]$ ; it is the best predictor of $X$ from $Y$ in mean squared error. Introduced in: Lesson 4
conditional probability: $P(A \mid B) = P(A \cap B) / P(B)$ for $P(B) > 0$ : the probability of $A$ when we know that $B$ happened. Introduced in: Lesson 1
conditionally independent: $A$ and $B$ are conditionally independent given $C$ if $P(A \cap B \mid C) = P(A \mid C)\,P(B \mid C)$ . It neither implies nor follows from independence. Introduced in: Lesson 1
confidence band: The set of intervals for the mean response $x^\top\beta$ at each $x$ . Its half-width is $t\,\hat\sigma\sqrt{x^\top(X^\top X)^{-1}x}$ . Introduced in: Lesson 13
confidence interval: A random interval computed from the data that contains the true parameter with probability $1 - \alpha$ (for example 95%) over repeated samples. The probability is about the procedure, not about one computed interval. Introduced in: Lesson 6
confounder: A variable that affects both a feature and the response. Leaving it out of a regression biases the coefficient of the feature (omitted-variable bias). Introduced in: Lesson 13
conjugate prior: A prior family such that the posterior is in the same family, for example beta prior + binomial likelihood gives a beta posterior. Every exponential family has one. Introduced in: Lesson 8
conservation of probability: The 1-D change-of-variables rule: for $Y = g(X)$ with $g$ monotone, the mass in matching small intervals is equal, so $f_Y(y) = f_X(x)/|g'(x)|$ with $x = g^{-1}(y)$ . Introduced in: Lesson 2
consistent: An estimator is consistent if $\hat\theta_n \to \theta$ in probability as $n \to \infty$ . MSE $\to 0$ is a sufficient condition. Introduced in: Lesson 8
control variates: Reduce Monte Carlo variance by subtracting $c\,(g(X) - \mathbb{E}[g(X)])$ , where $g$ has a known mean and is correlated with $f$ . The best $c$ removes the fraction $\rho^2$ of the variance. Introduced in: Lesson 14
convergence in distribution: $X_n \to X$ in distribution if the CDFs converge, $F_n(x) \to F(x)$ , at every point where $F$ is continuous. Introduced in: Lesson 6
convergence in probability: $X_n \to X$ in probability if $P(|X_n - X| > \varepsilon) \to 0$ for every $\varepsilon > 0$ . Introduced in: Lesson 6
correlation: $\rho = \operatorname{Cov}(X, Y)/(\sigma_X\sigma_Y) \in [-1, 1]$ . It measures linear dependence only; $|\rho| = 1$ means $Y = a + bX$ exactly (with probability 1). Introduced in: Lesson 4
correlation matrix: The matrix of correlations, $R = D^{-1}\Sigma D^{-1}$ with $D = \operatorname{diag}(\sigma_1, \dots, \sigma_d)$ . Its diagonal is all ones. Introduced in: Lesson 4
countable additivity: The axiom $P(\bigcup_i A_i) = \sum_i P(A_i)$ for disjoint events $A_1, A_2, \dots$ . Introduced in: Lesson 1
covariance: $\operatorname{Cov}(X, Y) = \mathbb{E}[(X - \mu_X)(Y - \mu_Y)] = \mathbb{E}[XY] - \mu_X\mu_Y$ . It is positive when $X$ and $Y$ tend to be large together. Introduced in: Lesson 4
covariance matrix: For a random vector $X$ with mean $\mu$ , $\Sigma = \mathbb{E}[(X - \mu)(X - \mu)^\top]$ , with $\Sigma_{ij} = \operatorname{Cov}(X_i, X_j)$ . Under an affine map, $\operatorname{Cov}(AX + b) = A\Sigma A^\top$ . Introduced in: Lesson 4
coverage: The probability that an interval procedure contains the true parameter. A 95% interval has the right coverage if 95% of its intervals over repeated samples contain the truth. Introduced in: Lesson 6
Cramér–Rao bound: For an unbiased estimator $T$ of $\theta$ from $n$ i.i.d. observations, under regularity conditions, $\operatorname{Var}(T) \ge 1/(nI(\theta))$ . Introduced in: Lesson 9
credible interval: A Bayesian interval that contains the parameter with posterior probability $1 - \alpha$ , given the observed data and the prior. Introduced in: Lesson 6
critical value: The threshold for the test statistic: reject $H_0$ when the statistic is beyond it, for example $|z| > 1.96$ at $\alpha = 0.05$ . Introduced in: Lesson 11
cross-entropy: $H(p, q) = -\mathbb{E}_p[\log q(X)]$ . The average negative log-likelihood of a model $q$ on data is the cross-entropy from the empirical distribution to $q$ . Introduced in: Lesson 8
cumulant generating function: $K_X(t) = \log M_X(t)$ . $K'(0)$ is the mean and $K''(0)$ is the variance; cumulants of independent sums add. Introduced in: Lesson 3
cumulative distribution function (CDF): $F(x) = P(X \le x)$ . It is nondecreasing and right-continuous, with limits 0 at $-\infty$ and 1 at $+\infty$ . Introduced in: Lesson 2
CUPED: Controlled-experiment using pre-experiment data: subtract $\theta(X - \bar X)$ from the metric $Y$ , where $X$ is a pre-experiment covariate and $\theta = \operatorname{Cov}(X, Y)/\operatorname{Var}(X)$ . It keeps the effect estimate unbiased and reduces the variance by the factor $1 - \rho^2$ . Introduced in: Lesson 12
degrees of freedom: The parameter $\nu$ of the $t$ and $\chi^2$ distributions. For the sample variance it is $n - 1$ : the number of deviations from $\bar x$ that can vary freely. Introduced in: Lesson 6
delta method: If $\sqrt n(\hat\theta - \theta) \to \mathcal N(0, \sigma^2)$ and $g$ is differentiable with $g'(\theta) \ne 0$ , then $\sqrt n(g(\hat\theta) - g(\theta)) \to \mathcal N(0, g'(\theta)^2\sigma^2)$ . So $\operatorname{SE}(g(\hat\theta)) \approx |g'(\hat\theta)|\operatorname{SE}(\hat\theta)$ . Introduced in: Lesson 6
design matrix: The $n \times p$ matrix $X$ whose rows are the feature vectors of the observations, usually with a column of ones for the intercept. Introduced in: Lesson 13
detailed balance: $\pi_i P_{ij} = \pi_j P_{ji}$ for all states: the flow from $i$ to $j$ equals the flow back. It implies that $\pi$ is stationary; such a chain is called reversible. Introduced in: Lesson 15
Dirichlet distribution: $\text{Dir}(\alpha_1, \dots, \alpha_K)$ : a distribution on probability vectors with density proportional to $\prod_k p_k^{\alpha_k - 1}$ . It generalises the beta to $K$ classes. Introduced in: Lesson 3
distribution: The rule that gives $P(X \in A)$ for all sets $A$ . A PMF, a PDF or a CDF describes it. Introduced in: Lesson 2
divergence (HMC): A leapfrog trajectory whose energy error becomes very large, usually in a region of high curvature. Divergences show that the sampler misses part of the posterior. Introduced in: Lesson 15
duality of tests and intervals: A $1 - \alpha$ confidence interval contains exactly the parameter values $\theta_0$ that a level- $\alpha$ test of $H_0: \theta = \theta_0$ does not reject. Introduced in: Lesson 11
effective sample size (Kish): $n_e = (\sum_i w_i)^2/\sum_i w_i^2 \in [1, n]$ : the number of equally weighted samples that give the same variance as the weighted sample. A small $n_e$ means a few weights dominate. Introduced in: Lesson 14
effective sample size (MCMC): $\text{ESS} = n/\tau$ : the number of independent draws that give the same Monte Carlo error as $n$ correlated draws. Introduced in: Lesson 15
efficient: An unbiased estimator is efficient if its variance equals the Cramér–Rao bound. Introduced in: Lesson 9
empirical Bayes: Estimating the prior's hyperparameters from many related groups of data, then using that prior for each group. Introduced in: Lesson 10
empirical distribution: The distribution that puts mass $1/n$ on each observed data point. Introduced in: Lesson 8
empirical risk: The average loss of a model on the $n$ training or test examples, $\hat R(h) = \frac1n\sum_i \ell(h(x_i), y_i)$ . Introduced in: Lesson 7
envelope: A function $Mq(x)$ that lies above the target density everywhere, built from a proposal $q$ that is easy to sample. Introduced in: Lesson 14
ergodic theorem: For an irreducible chain with stationary distribution $\pi$ , the time average $\frac1n\sum_t f(X_t)$ converges to $\mathbb{E}_\pi[f]$ from any starting state. Introduced in: Lesson 15
estimator: A function of the data, $\hat\theta = T(X_1, \dots, X_n)$ , used to estimate a parameter. It is a random variable. Introduced in: Lesson 8
event: A subset of the sample space. The event happens when the outcome is in this subset. Introduced in: Lesson 1
evidence: $p(D) = \int p(D \mid \theta)\,p(\theta)\,d\theta$ , the normalising constant of the posterior. Also called the marginal likelihood; it is used to compare models. Introduced in: Lesson 10
exchangeable: Random variables are exchangeable if their joint distribution does not change when we permute them. Under $H_0$ this makes permutation tests exact. Introduced in: Lesson 12
expectation: The probability-weighted average $\mathbb{E}[X] = \sum_x x\,p(x)$ or $\int x f(x)\,dx$ . It exists when the sum or integral of $|x|$ is finite. Also called the mean. Introduced in: Lesson 2
explaining away: The effect of conditioning on a collider: given the common effect, evidence for one cause lowers the probability of the other cause. Introduced in: Lesson 1
exponential distribution: $\text{Exp}(\lambda)$ : density $\lambda e^{-\lambda x}$ for $x \ge 0$ , mean $1/\lambda$ . It models the waiting time to the next event of a Poisson process and is memoryless. Introduced in: Lesson 3
exponential family: A family with density $p(x \mid \eta) = h(x)\exp\big(\eta^\top T(x) - A(\eta)\big)$ . Bernoulli, Poisson, Gaussian, gamma, beta and categorical are examples. Introduced in: Lesson 8
F-test for nested models: Tests whether $q$ extra features help: $F = \frac{(\text{RSS}_0 - \text{RSS}_1)/q}{\text{RSS}_1/(n - p)}$ , compared with the $F_{q, n - p}$ distribution. Introduced in: Lesson 13
false discovery rate (FDR): The expected fraction of false rejections among all rejections, $\mathbb{E}[V/\max(R, 1)]$ . Introduced in: Lesson 11
family-wise error rate (FWER): The probability of at least one false rejection among all the tests. Introduced in: Lesson 11
Fisher information: $I(\theta) = \operatorname{Var}_\theta(s(\theta)) = -\mathbb{E}_\theta\big[\frac{\partial^2}{\partial\theta^2}\log p(X \mid \theta)\big]$ : the expected curvature of the log-likelihood. For $n$ i.i.d. observations it is $nI(\theta)$ ; for several parameters it is a matrix. Introduced in: Lesson 9
full conditional: The distribution of one coordinate given all the other coordinates, $\pi(x_i \mid x_{-i})$ . Introduced in: Lesson 15
gamma distribution: $\text{Gamma}(k, \lambda)$ (shape $k$ , rate $\lambda$ ): density $\lambda^k x^{k - 1}e^{-\lambda x}/\Gamma(k)$ for $x > 0$ , mean $k/\lambda$ . For integer $k$ it is the sum of $k$ independent $\text{Exp}(\lambda)$ times (the Erlang distribution). Introduced in: Lesson 3
Gaussian conditioning: For jointly Gaussian $(x_1, x_2)$ : $x_1 \mid x_2 \sim \mathcal N\big(\mu_1 + \Sigma_{12}\Sigma_{22}^{-1}(x_2 - \mu_2),\ \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}\big)$ . The mean shifts linearly with $x_2$ ; the covariance does not depend on $x_2$ . Introduced in: Lesson 5
Gaussian distribution: $\mathcal N(\mu, \sigma^2)$ : density $\frac{1}{\sqrt{2\pi\sigma^2}}\exp\big(-\frac{(x - \mu)^2}{2\sigma^2}\big)$ . Also called the normal distribution. Introduced in: Lesson 3
Gauss–Markov theorem: If the errors have mean 0, equal variance and no correlation, OLS has the smallest variance among all linear unbiased estimators: it is the BLUE (best linear unbiased estimator). Introduced in: Lesson 13
generalisation bound: A bound on the gap between true and empirical risk. For a finite class $\mathcal H$ and losses in $[0, 1]$ , Hoeffding plus the union bound give $|R(h) - \hat R(h)| \le \sqrt{\log(2|\mathcal H|/\delta)/(2n)}$ for all $h \in \mathcal H$ with probability at least $1 - \delta$ . Introduced in: Lesson 7
generalised linear model (GLM): A regression model where the response has an exponential-family distribution and a link function of its mean is linear in the features. Linear, logistic and Poisson regression are GLMs. Introduced in: Lesson 8
geometric distribution: $\text{Geom}(p)$ : the number of trials up to and including the first success. $P(X = k) = (1 - p)^{k - 1}p$ for $k \ge 1$ , mean $1/p$ . Introduced in: Lesson 3
Gibbs sampling: MCMC that updates one coordinate (or block) at a time by sampling from its full conditional distribution given all the others. Every move is accepted. Introduced in: Lesson 15
Hamiltonian Monte Carlo (HMC): MCMC that adds a momentum variable, simulates Hamiltonian dynamics with the leapfrog integrator to make long moves along the target, and corrects the integration error with a Metropolis step. Introduced in: Lesson 15
Hastings correction: The factor $q(x \mid x')/q(x' \mid x)$ in the acceptance probability. It is needed when the proposal is not symmetric. Introduced in: Lesson 15
hat matrix: $H = X(X^\top X)^{-1}X^\top$ , the projection that maps $y$ to the fitted values $\hat y = Hy$ . Introduced in: Lesson 13
HC3: A heteroscedasticity-consistent (robust, sandwich) standard error that uses $e_i^2/(1 - h_{ii})^2$ as the variance estimate of observation $i$ . Introduced in: Lesson 13
Hoeffding's inequality: For independent $X_i \in [a, b]$ with mean $\mu$ , $P(|\bar X_n - \mu| \ge \varepsilon) \le 2\exp(-2n\varepsilon^2/(b - a)^2)$ . Introduced in: Lesson 7
Hoeffding's lemma: If $a \le X \le b$ , then $\mathbb{E}[e^{\lambda(X - \mathbb{E}X)}] \le e^{\lambda^2(b - a)^2/8}$ for all $\lambda$ : a bounded variable is sub-Gaussian with variance proxy $(b - a)^2/4$ . Introduced in: Lesson 7
Holm's method: A step-down version of Bonferroni: compare the $k$ -th smallest p-value with $\alpha/(m - k + 1)$ and stop at the first failure. It controls the FWER and rejects at least as much as Bonferroni. Introduced in: Lesson 11
homoscedastic: All errors have the same variance $\sigma^2$ . Heteroscedastic errors have variances that change between observations. Introduced in: Lesson 13
hyperparameters: Parameters of the prior, such as $\alpha$ and $\beta$ of a beta prior. Introduced in: Lesson 10
hypothesis test: A rule that decides from data whether to reject a null hypothesis, built so that the probability of a false rejection is at most a chosen level $\alpha$ . Introduced in: Lesson 11
i.i.d.: Independent and identically distributed: the random variables are independent and all have the same distribution. Introduced in: Lesson 2
identifiable: A model is identifiable if different parameter values give different distributions, so the data can in principle tell them apart. Introduced in: Lesson 8
importance sampling: Estimate $\mathbb{E}_p[f(X)]$ with samples from a proposal $q$ : $\frac1n\sum_i w(X_i)f(X_i)$ with weights $w = p/q$ . It is unbiased when $q > 0$ wherever $pf \ne 0$ . Introduced in: Lesson 14
importance weight: $w(x) = p(x)/q(x)$ , the factor that corrects for sampling from $q$ instead of $p$ . Introduced in: Lesson 14
inclusion–exclusion: The rule $P(A \cup B) = P(A) + P(B) - P(A \cap B)$ . The overlap is counted twice in $P(A) + P(B)$ , so we subtract it once. Introduced in: Lesson 1
independence: Events $A$ and $B$ are independent if $P(A \cap B) = P(A)\,P(B)$ . For $P(B) > 0$ this means $P(A \mid B) = P(A)$ : learning $B$ does not change the probability of $A$ . Introduced in: Lesson 1
indicator: The random variable $\mathbf 1_A$ that is 1 when event $A$ happens and 0 otherwise. Its mean is $P(A)$ . Introduced in: Lesson 2
integrated autocorrelation time: $\tau = 1 + 2\sum_{t \ge 1}\rho_t$ . The variance of the chain mean is about $\sigma^2\tau/n$ . Introduced in: Lesson 15
interleaving: An online evaluation of two rankers that mixes their results in one list for each user and counts which ranker's items get clicks. It often needs far fewer users than an A/B test. Introduced in: Lesson 12
invariance of the MLE: If $\hat\theta$ is the MLE of $\theta$ , then $g(\hat\theta)$ is the MLE of $g(\theta)$ . Introduced in: Lesson 8
inverse-CDF sampling: If $U \sim \text{Uniform}(0, 1)$ , then $X = F^{-1}(U)$ has CDF $F$ (with the generalised inverse when $F$ is not invertible). It turns uniform random numbers into samples of any distribution. Introduced in: Lesson 2
irreducible: Every state can be reached from every other state with positive probability in some number of steps. Introduced in: Lesson 15
Jacobian determinant: For a smooth invertible map $y = g(x)$ , $|\det J_g(x)|$ is the factor by which $g$ scales small volumes near $x$ . Densities change by the inverse factor: $f_Y(y) = f_X(x)/|\det J_g(x)|$ . Introduced in: Lesson 5
Jensen's inequality: For a convex function $\varphi$ , $\varphi(\mathbb{E}[X]) \le \mathbb{E}[\varphi(X)]$ . For a concave function the inequality reverses. Introduced in: Lesson 2
jitter: A small multiple of the identity, $\epsilon I$ , added to a covariance matrix so that a Cholesky factorisation succeeds in floating point. Introduced in: Lesson 5
joint distribution: The distribution of several random variables together, given by a joint PMF $p(x, y) = P(X = x, Y = y)$ or a joint PDF $f(x, y)$ . Introduced in: Lesson 4
jointly Gaussian: A random vector is jointly Gaussian if every linear combination $a^\top X$ is Gaussian. Each coordinate being Gaussian is not enough. Introduced in: Lesson 5
Kullback–Leibler divergence: $\mathrm{KL}(p \,\|\, q) = \mathbb{E}_{p}[\log(p(X)/q(X))] \ge 0$ , with equality only when $p = q$ . Maximising the likelihood minimises the KL divergence from the empirical distribution to the model. Introduced in: Lesson 8
Langevin dynamics: The update $x' = x + \frac{\eta}{2}\nabla\log\pi(x) + \sqrt{\eta}\,z$ with $z \sim \mathcal N(0, I)$ : a gradient step plus Gaussian noise. With a Metropolis–Hastings correction it is MALA; with mini-batch gradients and no correction it is SGLD. Introduced in: Lesson 15
Laplace approximation: Approximate a posterior (or likelihood) by a Gaussian centred at its mode, with covariance equal to the inverse of the negative Hessian of the log-density there. Introduced in: Lesson 9
law of large numbers: For i.i.d. variables with finite mean $\mu$ , the sample mean $\bar X_n$ converges to $\mu$ . The weak law gives convergence in probability; the strong law gives almost sure convergence. Introduced in: Lesson 6
law of total probability: If $B_1, \dots, B_k$ is a partition, then $P(A) = \sum_i P(A \mid B_i)\,P(B_i)$ . Introduced in: Lesson 1
law of total variance: $\operatorname{Var}(X) = \mathbb{E}[\operatorname{Var}(X \mid Y)] + \operatorname{Var}(\mathbb{E}[X \mid Y])$ : total variance = within-group part + between-group part. Introduced in: Lesson 4
Le Cam's inequality: A sum of independent $\text{Bernoulli}(p_i)$ variables is within total variation distance $\sum_i p_i^2$ of $\text{Poisson}(\sum_i p_i)$ . Introduced in: Lesson 3
leapfrog integrator: A numerical integrator for Hamiltonian dynamics: half step in momentum, full step in position, half step in momentum. It is reversible and preserves volume. Introduced in: Lesson 15
length-biased sampling: Picking an interval with probability proportional to its length. If you arrive at a random time, you are more likely to land in a long gap than in a short one. Introduced in: Lesson 3
leverage: The diagonal entry $h_{ii}$ of the hat matrix: how much observation $i$ pulls its own fitted value. Points far from the other $x$ values have high leverage. Introduced in: Lesson 13
likelihood: The probability (or density) of the observed data under a hypothesis or parameter value, $P(E \mid H)$ , seen as a function of the hypothesis. Introduced in: Lesson 1
likelihood equations: The equations $\partial\ell/\partial\theta = 0$ that the MLE satisfies when the maximum is in the interior of the parameter space. Introduced in: Lesson 8
likelihood function: For data $x_1, \dots, x_n$ and model $p(x \mid \theta)$ , $L(\theta) = \prod_i p(x_i \mid \theta)$ , seen as a function of $\theta$ with the data fixed. It is not a probability distribution over $\theta$ . Introduced in: Lesson 8
likelihood ratio: The ratio $P(E \mid H) / P(E \mid H^c)$ . In odds form, Bayes' rule says: posterior odds = likelihood ratio × prior odds. Introduced in: Lesson 1
linear model: $y = X\beta + \varepsilon$ with $\mathbb{E}[\varepsilon] = 0$ and $\operatorname{Cov}(\varepsilon) = \sigma^2 I$ ; for exact tests also $\varepsilon \sim \mathcal N(0, \sigma^2 I)$ . Introduced in: Lesson 13
linearity of expectation: $\mathbb{E}[aX + bY] = a\,\mathbb{E}[X] + b\,\mathbb{E}[Y]$ for any random variables with finite means, with no independence needed. Introduced in: Lesson 2
log-likelihood: $\ell(\theta) = \sum_i \log p(x_i \mid \theta)$ . The log turns the product into a sum, avoids underflow and has the same maximiser. Introduced in: Lesson 8
log-odds: $\operatorname{logit}(p) = \log(p/(1 - p))$ . It is the natural parameter of the Bernoulli distribution; its inverse is the sigmoid. Introduced in: Lesson 8
log-partition function: $A(\eta) = \log\int h(x)\,e^{\eta^\top T(x)}\,dx$ , which normalises an exponential family. It is convex, $\nabla A(\eta) = \mathbb{E}_\eta[T(X)]$ and $\nabla^2 A(\eta) = \operatorname{Cov}_\eta(T(X))$ . Introduced in: Lesson 8
Mahalanobis distance: $\sqrt{(x - \mu)^\top\Sigma^{-1}(x - \mu)}$ : the distance from the mean measured in units of the distribution's spread. The contours of a Gaussian density are the ellipsoids of constant Mahalanobis distance. Introduced in: Lesson 5
marginal distribution: The distribution of one variable alone, obtained from the joint by summing or integrating out the others: $p(x) = \sum_y p(x, y)$ . Introduced in: Lesson 4
Markov chain: A sequence of random states where the next state depends only on the current state (the Markov property), not on earlier states. Introduced in: Lesson 15
Markov chain Monte Carlo (MCMC): Sample from a target $\pi$ by running a Markov chain whose stationary distribution is $\pi$ , and average over the states it visits. Introduced in: Lesson 15
Markov's inequality: For $X \ge 0$ and $a > 0$ , $P(X \ge a) \le \mathbb{E}[X]/a$ . Introduced in: Lesson 6
maximum a posteriori (MAP): The mode of the posterior, $\arg\max_\theta\, [\log p(D \mid \theta) + \log p(\theta)]$ . Under a nonlinear change of parameters the MAP point can move, because the density picks up a Jacobian factor. Introduced in: Lesson 10
maximum likelihood estimate (MLE): $\hat\theta = \arg\max_\theta \ell(\theta)$ : the parameter value under which the observed data have the highest probability (or density). Introduced in: Lesson 8
mean squared error (MSE): $\operatorname{MSE}(\hat\theta) = \mathbb{E}[(\hat\theta - \theta)^2] = \operatorname{bias}^2 + \operatorname{Var}(\hat\theta)$ (the bias–variance decomposition). Introduced in: Lesson 8
memoryless property: $P(X > s + t \mid X > s) = P(X > t)$ : the remaining waiting time does not depend on how long you already waited. Among continuous distributions only the exponential has it. Introduced in: Lesson 3
Metropolis–Hastings: An MCMC algorithm: propose $x' \sim q(\cdot \mid x)$ and accept with probability $\min\big(1, \frac{\pi(x')q(x \mid x')}{\pi(x)q(x' \mid x)}\big)$ , else stay. Only ratios of $\pi$ appear, so the normalising constant is not needed. Introduced in: Lesson 15
minimum detectable effect (MDE): The smallest true difference $\delta$ that an experiment should detect with power $1 - \beta_{\text{II}}$ at level $\alpha$ . It sets the sample size $n = 2\sigma^2(z_{1 - \alpha/2} + z_{1 - \beta_{\text{II}}})^2/\delta^2$ per arm. Introduced in: Lesson 12
misspecification: The true distribution is not in the model family. The MLE then converges to the member closest to the truth in KL divergence. Introduced in: Lesson 8
moment generating function (MGF): $M_X(t) = \mathbb{E}[e^{tX}]$ . Its derivatives at 0 are the moments, the MGF of a sum of independent variables is the product of their MGFs, and if it is finite near 0 it determines the distribution. Introduced in: Lesson 3
moment matching: In an exponential family the MLE solves $\mathbb{E}_{\hat\eta}[T(X)] = \frac1n\sum_i T(x_i)$ : the model's mean of the sufficient statistic equals the data's average. Introduced in: Lesson 8
Monte Carlo estimate: Estimate $\mathbb{E}[f(X)]$ by $\frac1n\sum_i f(X_i)$ with i.i.d. samples $X_i$ . Its standard error is $\sigma_f/\sqrt n$ in any dimension. Introduced in: Lesson 14
moving block bootstrap: A bootstrap for dependent data, such as time series: resample blocks of consecutive observations, so that dependence within a block is kept. Introduced in: Lesson 12
multivariate Gaussian: $\mathcal N(\mu, \Sigma)$ on $\R^d$ : density $(2\pi)^{-d/2}\det(\Sigma)^{-1/2}\exp\big(-\frac12(x - \mu)^\top\Sigma^{-1}(x - \mu)\big)$ . It is the distribution of $\mu + Lz$ with $LL^\top = \Sigma$ and $z \sim \mathcal N(0, I)$ . Introduced in: Lesson 5
mutually independent: Events are mutually independent if, for every subcollection, the probability of the intersection is the product of the probabilities. Introduced in: Lesson 1
natural gradient: The update direction $I(\theta)^{-1}\nabla\ell(\theta)$ : the gradient preconditioned by the inverse Fisher information. To first order, the step does not depend on how the model is parametrised. Introduced in: Lesson 9
natural parameter: The parameter $\eta$ that multiplies the sufficient statistic in an exponential family, for example the log-odds for the Bernoulli. Introduced in: Lesson 8
normal equations: $X^\top X\hat\beta = X^\top y$ : the residual vector is orthogonal to every column of $X$ . Introduced in: Lesson 13
normalising flow: A generative model that maps simple noise through an invertible, differentiable function. The change-of-variables formula with the Jacobian determinant gives the exact density. Introduced in: Lesson 5
null distribution: The distribution of the test statistic when $H_0$ is true. Introduced in: Lesson 11
null hypothesis: $H_0$ : the default claim that a test tries to reject, often 'no effect' or 'no difference'. Introduced in: Lesson 11
observed information: $J(\hat\theta) = -\ell''(\hat\theta)$ , the curvature of the log-likelihood of the actual data at the MLE. The expected information $nI(\theta)$ is the mean of $-\ell''(\theta)$ . Introduced in: Lesson 9
odds: The ratio $P(A) / (1 - P(A))$ . A probability of $0.2$ is odds of $1:4$ , or $0.25$ . Introduced in: Lesson 1
one-sample t-test: Tests $H_0: \mu = \mu_0$ with $t = (\bar x - \mu_0)/(s/\sqrt n)$ and the $t_{n - 1}$ distribution. Paired data use it on the differences. Introduced in: Lesson 11
ordinary least squares (OLS): The estimate $\hat\beta$ that minimises the residual sum of squares $\|y - X\beta\|^2$ . If $X$ has full column rank, $\hat\beta = (X^\top X)^{-1}X^\top y$ . Introduced in: Lesson 13
overdispersion: Count data whose variance is larger than its mean. A Poisson model cannot fit it; a negative binomial or a mixture can. Introduced in: Lesson 3
p-hacking: Trying many analyses, subsets or stopping times and reporting the one with a small p-value. It makes the real type I error rate much larger than $\alpha$ . Introduced in: Lesson 11
p-value: The probability, under $H_0$ , of a test statistic at least as extreme as the observed one. It is not $P(H_0 \mid \text{data})$ and not an effect size. Introduced in: Lesson 11
PAC learning: Probably approximately correct learning: with probability at least $1 - \delta$ , the learned model has true risk within $\varepsilon$ of the best in its class, using a number of samples polynomial in $1/\varepsilon$ and $1/\delta$ . Introduced in: Lesson 7
pairwise independent: Every pair of the events is independent. This does not imply mutual independence (example: two fair coins and their XOR). Introduced in: Lesson 1
partition: Disjoint events $B_1, \dots, B_k$ whose union is the whole sample space $\Omega$ . Introduced in: Lesson 1
Pearson's χ² test: Compares observed counts $O$ with expected counts $E$ under $H_0$ by $\sum (O - E)^2/E$ , which is approximately $\chi^2$ for large counts. Introduced in: Lesson 11
peeking: Looking at the p-value many times during an experiment and stopping when it is small. It inflates the type I error rate; sequential tests fix it. Introduced in: Lesson 12
percentile interval: The bootstrap confidence interval between the $\alpha/2$ and $1 - \alpha/2$ quantiles of the bootstrap distribution. Introduced in: Lesson 12
permutation test: A test of 'no difference between groups': shuffle the group labels many times and compare the observed statistic with its distribution over the shuffles. Introduced in: Lesson 12
plug-in prediction: Prediction with one point estimate, $p(x_{\text{new}} \mid \hat\theta)$ . It ignores the uncertainty in $\theta$ and is too confident for small data. Introduced in: Lesson 10
Poisson distribution: $\text{Poisson}(\lambda)$ : $P(X = k) = e^{-\lambda}\lambda^k / k!$ for $k = 0, 1, \dots$ . Mean and variance are both $\lambda$ . It is the limit of $\text{Bin}(n, \lambda/n)$ and models counts of rare events. Introduced in: Lesson 3
Poisson process: Random event times with independent $\text{Exp}(\lambda)$ gaps. Equivalently, the counts in disjoint intervals are independent and $\text{Poisson}(\lambda \cdot \text{length})$ . Introduced in: Lesson 3
positive semidefinite: A symmetric matrix $\Sigma$ with $a^\top \Sigma a \ge 0$ for all vectors $a$ . Every covariance matrix is positive semidefinite because $a^\top\Sigma a = \operatorname{Var}(a^\top X)$ . Introduced in: Lesson 4
posterior: The probability of a hypothesis (or the distribution of a parameter) after we see the data; it is proportional to likelihood × prior. Introduced in: Lesson 1
posterior mean: $\mathbb{E}[\theta \mid D]$ . It minimises the expected squared error under the posterior. Introduced in: Lesson 10
posterior predictive: The distribution of a new observation given the data, $p(x_{\text{new}} \mid D) = \int p(x_{\text{new}} \mid \theta)\,p(\theta \mid D)\,d\theta$ . It includes the uncertainty about $\theta$ . Introduced in: Lesson 10
power: The probability of rejecting $H_0$ when a given alternative is true, $1 - \beta_{\text{II}}$ . It grows with the effect size, the sample size and $\alpha$ . Introduced in: Lesson 11
precision: The fraction of positives that are true, $P(\text{sick} \mid +)$ ; also called the positive predictive value. It depends on the base rate. Introduced in: Lesson 1
precision matrix: The inverse covariance $\Lambda = \Sigma^{-1}$ . For a Gaussian, $\Lambda_{ij} = 0$ means $x_i$ and $x_j$ are conditionally independent given all other coordinates. Introduced in: Lesson 5
prediction interval: An interval for one new observation $y = x^\top\beta + \varepsilon$ . It adds the noise variance: half-width $t\,\hat\sigma\sqrt{1 + x^\top(X^\top X)^{-1}x}$ , so it does not shrink to 0 as $n$ grows. Introduced in: Lesson 13
principal components: The eigenvectors of the covariance matrix. The first one is the direction of largest variance, and the projections on different principal components are uncorrelated. Introduced in: Lesson 4
prior: The probability of a hypothesis (or the distribution of a parameter) before we see the data. Introduced in: Lesson 1
probability density function (PDF): A function $f \ge 0$ with $P(a \le X \le b) = \int_a^b f(x)\,dx$ . A density value can be larger than 1; only its integrals are probabilities. Introduced in: Lesson 2
probability integral transform (PIT): If $X$ has a continuous CDF $F$ , then $F(X) \sim \text{Uniform}(0, 1)$ . Introduced in: Lesson 2
probability mass function (PMF): For a discrete random variable, $p(x) = P(X = x)$ . The values are in $[0, 1]$ and add up to 1. Introduced in: Lesson 2
probability space: A triple $(\Omega, \mathcal F, P)$ : a sample space, a σ-algebra $\mathcal F$ of events, and a probability $P$ that satisfies the axioms: $P(A) \ge 0$ , $P(\Omega) = 1$ and countable additivity. Introduced in: Lesson 1
proposal: The distribution $q$ that we can sample from, used in place of the target $p$ in importance or rejection sampling, or to suggest moves in MCMC. Introduced in: Lesson 14
Rademacher variable: A random sign: $+1$ or $-1$ with probability $\frac12$ each. It is sub-Gaussian with variance proxy 1. Introduced in: Lesson 7
random variable: A function $X: \Omega \to \R$ that assigns a number to each outcome. Introduced in: Lesson 2
random-walk Metropolis: Metropolis–Hastings with a symmetric proposal $x' = x + s z$ , $z \sim \mathcal N(0, I)$ ; the acceptance probability is then $\min(1, \pi(x')/\pi(x))$ . Introduced in: Lesson 15
regularity conditions: Assumptions that make the Fisher theory work, for example: the support does not depend on $\theta$ , $\theta$ is in the interior of the parameter space, and we can differentiate under the integral. Introduced in: Lesson 9
rejection sampling: Sample from a density $p$ with an envelope $Mq \ge p$ : draw $x \sim q$ and accept it with probability $p(x)/(Mq(x))$ . Accepted points are exact samples from $p$ ; for normalised $p$ and $q$ the acceptance rate is $1/M$ . Introduced in: Lesson 14
relative standard error: The standard error divided by the estimated value. For a rare-event probability $p$ with plain Monte Carlo it is about $1/\sqrt{np}$ , so small $p$ needs very many samples. Introduced in: Lesson 14
reparameterisation trick: Write a random sample as a differentiable function of the parameters and of noise with a fixed distribution, e.g. $x = \mu + \sigma\varepsilon$ with $\varepsilon \sim \mathcal N(0, 1)$ . Gradients then pass through the sampling step. Introduced in: Lesson 2
residual sum of squares (RSS): $\sum_i e_i^2$ . Under the Gaussian linear model, $\text{RSS}/\sigma^2 \sim \chi^2_{n - p}$ , and $\hat\sigma^2 = \text{RSS}/(n - p)$ is unbiased. Introduced in: Lesson 13
residuals: $e = y - \hat y = (I - H)y$ , the differences between observed and fitted values. Introduced in: Lesson 13
sample space: The set $\Omega$ of all possible outcomes of a random experiment. Introduced in: Lesson 1
sample-ratio mismatch (SRM): The observed split of users between arms differs from the planned split by more than chance. It signals a bug in assignment or logging. Introduced in: Lesson 12
sampling distribution: The distribution of a statistic (such as $\bar X_n$ ) over repeated samples of the same size from the same population. Introduced in: Lesson 6
sampling distribution of OLS: Under the Gaussian linear model, $\hat\beta \sim \mathcal N(\beta, \sigma^2(X^\top X)^{-1})$ , independent of the RSS. This gives t-tests and confidence intervals for each coefficient. Introduced in: Lesson 13
Schur complement: For a block matrix with blocks $\Sigma_{11}, \Sigma_{12}, \Sigma_{21}, \Sigma_{22}$ , the matrix $\Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$ . It is the conditional covariance of a Gaussian. Introduced in: Lesson 5
score: The derivative of the log-likelihood, $s(\theta) = \frac{\partial}{\partial\theta}\log p(x \mid \theta)$ . Under regularity conditions its mean at the true $\theta$ is 0. Introduced in: Lesson 9
self-normalised importance sampling: $\sum_i \bar w_i f(X_i)$ with $\bar w_i = w_i/\sum_j w_j$ . It works when $p$ is known only up to a constant; it is slightly biased but consistent. Introduced in: Lesson 14
sensitivity: The probability that a test is positive for a sick person, $P(+ \mid \text{sick})$ : the true positive rate. Introduced in: Lesson 1
shrinkage estimator: An estimator that pulls raw estimates toward a common value (for example the prior mean). The posterior mean under a conjugate prior is one. Introduced in: Lesson 10
significance level: The chosen bound $\alpha$ on the type I error rate, often 0.05. We reject $H_0$ when $p \le \alpha$ . Introduced in: Lesson 11
simplex: The set of probability vectors $\{p \in \R^K : p_k \ge 0, \sum_k p_k = 1\}$ . Introduced in: Lesson 3
Simpson's paradox: A trend that holds in every group reverses (or disappears) when the groups are pooled. It can happen when the group is related to both the choice of option and the outcome. Introduced in: Lesson 1
size: The largest probability of a type I error of a test over all distributions in $H_0$ . A test is conservative when its size is below $\alpha$ . Introduced in: Lesson 11
specificity: The probability that a test is negative for a healthy person, $P(- \mid \text{healthy})$ . The false positive rate is $1 - \text{specificity}$ . Introduced in: Lesson 1
split- $\hat R$: A convergence diagnostic: split each of $m$ chains into halves and compare the variance between the halves with the variance within them. Values near 1 (below 1.01) mean the chains agree. Introduced in: Lesson 15
standard deviation: The square root of the variance, $\sigma = \sqrt{\operatorname{Var}(X)}$ . It has the same unit as $X$ . Introduced in: Lesson 2
standard error: The standard deviation of an estimator's sampling distribution. For the sample mean of i.i.d. data, $\operatorname{SE} = \sigma/\sqrt n$ , estimated by $s/\sqrt n$ . Introduced in: Lesson 6
stationary distribution: A distribution $\pi$ with $\pi P = \pi$ : if the chain starts in $\pi$ , it stays in $\pi$ . Introduced in: Lesson 15
Student's t distribution: The distribution of $(\bar X - \mu)/(S/\sqrt n)$ for Gaussian data, with $\nu = n - 1$ degrees of freedom. It has heavier tails than $\mathcal N(0, 1)$ and approaches it as $\nu \to \infty$ . Introduced in: Lesson 6
Student's t-test (pooled): A two-sample t-test that assumes equal variances and uses the pooled variance with $n_1 + n_2 - 2$ degrees of freedom. Introduced in: Lesson 11
sub-exponential: A random variable whose tails decay at least like $e^{-ct}$ but possibly not like $e^{-ct^2}$ ; its MGF is finite only for small $|\lambda|$ . Examples: exponential and $\chi^2$ variables. Introduced in: Lesson 7
sub-Gaussian: A random variable with $\mathbb{E}[e^{\lambda(X - \mu)}] \le e^{\lambda^2\sigma^2/2}$ for all $\lambda$ ( $\sigma^2$ is the variance proxy). Its tails satisfy $P(|X - \mu| \ge t) \le 2e^{-t^2/(2\sigma^2)}$ . Introduced in: Lesson 7
sufficient statistic: A statistic $T(x)$ that contains all information in the data about $\theta$ : the conditional distribution of the data given $T$ does not depend on $\theta$ . By the Fisher–Neyman factorisation, $T$ is sufficient if and only if $p(x \mid \theta) = g(T(x), \theta)\,h(x)$ . Introduced in: Lesson 8
survival function: $S(x) = P(X > x) = 1 - F(x)$ : the probability that the waiting time is longer than $x$ . Introduced in: Lesson 3
t-interval: The interval $\bar x \pm t_{n - 1, 1 - \alpha/2}\, s/\sqrt n$ for a mean when $\sigma$ is unknown. It is exact for Gaussian data. Introduced in: Lesson 6
test statistic: A number computed from the data that measures how far the data are from what $H_0$ predicts, for example $t = (\bar x - \mu_0)/(s/\sqrt n)$ . Introduced in: Lesson 11
total variation distance: $d_{TV}(P, Q) = \max_A |P(A) - Q(A)|$ , the largest difference in probability of any event. On the integers it equals $\frac12\sum_k |P(k) - Q(k)|$ . Introduced in: Lesson 3
tower rule: $\mathbb{E}[\mathbb{E}[X \mid Y]] = \mathbb{E}[X]$ : the average of the group means, weighted by group probability, is the overall mean. Also called the law of total expectation. Introduced in: Lesson 4
transition matrix: The matrix $P$ with $P_{ij} = P(X_{t + 1} = j \mid X_t = i)$ ; each row sums to 1. A distribution (row vector) $\mu$ moves to $\mu P$ in one step. Introduced in: Lesson 15
true risk: The expected loss of a model on new data from the population, $R(h) = \mathbb{E}[\ell(h(X), Y)]$ . Introduced in: Lesson 7
two-proportion z-test: Tests $H_0: p_1 = p_2$ with $z = (\hat p_1 - \hat p_2)/\sqrt{\hat p(1 - \hat p)(1/n_1 + 1/n_2)}$ , where $\hat p$ is the pooled rate. Introduced in: Lesson 11
type I error: Rejecting $H_0$ when it is true: a false positive. Introduced in: Lesson 11
type II error: Not rejecting $H_0$ when it is false: a false negative. Its probability is $\beta_{\text{II}}$ . Introduced in: Lesson 11
uncorrelated: $\operatorname{Cov}(X, Y) = 0$ . Independent variables are uncorrelated, but uncorrelated variables can be dependent (example: $X$ symmetric around 0 and $Y = X^2$ ). Introduced in: Lesson 4
union bound: $P(\bigcup_i A_i) \le \sum_i P(A_i)$ for any events, with no assumption about dependence. Also called Boole's inequality. Introduced in: Lesson 1
variance: $\operatorname{Var}(X) = \mathbb{E}[(X - \mu)^2] = \mathbb{E}[X^2] - \mu^2$ with $\mu = \mathbb{E}[X]$ : the mean squared distance from the mean. Introduced in: Lesson 2
variance inflation factor (VIF): $\text{VIF}_j = 1/(1 - R_j^2)$ , where $R_j^2$ comes from regressing feature $j$ on the other features. It is the factor by which collinearity inflates $\operatorname{Var}(\hat\beta_j)$ . Introduced in: Lesson 13
Wald interval: The interval $\hat\theta \pm z_{1 - \alpha/2}\widehat{\operatorname{SE}}$ with the standard error estimated at $\hat\theta$ ; for a proportion, $\hat p \pm 1.96\sqrt{\hat p(1 - \hat p)/n}$ . It has poor coverage for small $n$ or $p$ near 0 or 1. Introduced in: Lesson 6
warm-up: The first part of an MCMC run, used to forget the starting point and to tune the step size; its draws are discarded. Also called burn-in. Introduced in: Lesson 15
weight collapse: In high dimension, almost all importance weight goes to one or a few samples, so the effective sample size is close to 1. Introduced in: Lesson 14
Welch's t-test: A two-sample t-test that does not assume equal variances: $t = (\bar x_1 - \bar x_2)/\sqrt{s_1^2/n_1 + s_2^2/n_2}$ with Welch–Satterthwaite degrees of freedom. It is a good default. Introduced in: Lesson 11
Wilson interval: A confidence interval for a proportion obtained by solving $|\hat p - p| \le z\sqrt{p(1 - p)/n}$ for $p$ . Its coverage is much closer to nominal than the Wald interval's for small $n$ . Introduced in: Lesson 6
z-score: The standardised value $z = (x - \mu)/\sigma$ : how many standard deviations $x$ is from the mean. Introduced in: Lesson 2
σ-algebra: A collection $\mathcal F$ of events that contains $\Omega$ and is closed under complements and countable unions. These are the events that get a probability. Introduced in: Lesson 1
χ² distribution: The distribution of $Z_1^2 + \dots + Z_k^2$ for independent standard normal $Z_i$ , with $k$ degrees of freedom and mean $k$ . Introduced in: Lesson 11