Probability and Statistics
Probability is the language of uncertainty, and statistics turns data into claims you can trust. Start from coins, dice and a medical test, then build the tools of machine learning: random variables, the multivariate Gaussian, the central limit theorem and concentration, maximum likelihood and Bayesian inference, tests and confidence intervals, regression, Monte Carlo and MCMC.
For students, engineers and researchers who want to understand the probability and statistics behind machine learning, from a first course to the level of research papers. You need high-school algebra, sums and logarithms (the foundations track), and for the later lessons derivatives, integrals and matrices (the calculus and linear algebra tracks). Every lesson has simulations you can run and change.
Quick recap
The key takeaways of all lessons on one page
All interactive figures
Every figure of the track on one page
Videos and books
The best free courses
Glossary
Every technical term of the track with a short definition
Concept map
How the lessons of this track connect to each other and to other tracks.
Keeps every lesson page and figure of this track in this browser, so you can read it without internet.
Chance, random variables and distributions
Events, conditional probability and Bayes' rule, random variables, expectation, and the common distributions with their relations.
-
01
Probability, conditional probability and Bayes' ruleOutcomes, events and counting; the axioms of probability; conditioning as "zoom in and rescale"; Bayes' rule with natural frequencies and odds; independence and conditional independence; and the classic traps (base rates, Monty Hall, Simpson's paradox).
80 min
5 figures
Beginner -
02
Random variables: PMF, PDF, CDF, expectation and varianceA random variable puts a number on each outcome. Its distribution is described by a PMF (bars), a PDF (a curve whose area is probability) and a CDF. Expectation is the balance point, linearity needs no independence, variance measures spread, and the CDF read backwards turns uniform noise into samples of any distribution.
70 min
4 figures
Beginner -
03
Common distributions, the Poisson process and moment generating functionsThe distributions you meet in every ML paper (Bernoulli, binomial, geometric, Poisson, categorical, exponential, gamma, beta, Gaussian, Dirichlet), what each one models, and how they are related. Memorylessness, moment generating functions, sums of independent variables, and the Poisson process with thinning and superposition.
75 min
5 figures
Beginner
Many random variables
Joint and conditional distributions, covariance, change of variables and the multivariate Gaussian.
-
04
Joint and conditional distributions, conditional expectation and covarianceHow two or more random variables behave together. Marginals and conditionals from a joint table, the tower rule and the law of total variance, covariance and correlation, and the covariance matrix with its link to PCA.
80 min
5 figures
Intermediate -
05
Change of variables and the multivariate GaussianHow a density changes under a map (the Jacobian rule and Box–Muller), and the multivariate Gaussian N(μ, Σ). Its density, sampling and log-density with Cholesky, marginals as shadows, and conditionals as slices with the Schur complement.
85 min
5 figures
Intermediate
From samples to populations
Why averages settle down, why they look Gaussian, how to put an error bar on them, and how fast they concentrate.
-
06
The law of large numbers, the central limit theorem and confidence intervalsWhy a sample mean settles down (the law of large numbers), why its error looks Gaussian (the central limit theorem), and how to turn this into an error bar. Standard errors, confidence intervals with z and Student's t, intervals for a proportion, Monte Carlo error bars and the delta method.
75 min
5 figures
Beginner -
07
Concentration inequalities: Chernoff, Hoeffding and sample sizesGuaranteed error bounds for averages. The Chernoff method, Hoeffding's inequality, sub-Gaussian variables, how to turn a bound into a sample size, and the union bound behind the simplest generalisation bound for machine learning.
60 min
4 figures
Advanced
Estimation
Maximum likelihood, how accurate it can be (Fisher information), and the Bayesian alternative with priors and posteriors.
-
08
Maximum likelihood, the exponential family and properties of estimatorsThe likelihood, the maximum likelihood estimate (MLE) for coins, Gaussians, Poisson and exponential data, and how to judge an estimator by its bias, variance and mean squared error. Why training losses are negative log-likelihoods (the KL view), and why the exponential family makes the MLE a matching of averages.
75 min
4 figures
Intermediate -
09
Fisher information, the Cramér–Rao bound and asymptotic normality of the MLEHow precise can an estimate be? The score and the Fisher information (the curvature of the log-likelihood), the Cramér–Rao lower bound, the asymptotic normality of the MLE, Wald intervals and standard errors, and the cases where this theory fails.
70 min
5 figures
Advanced -
10
Bayesian inference: priors, posteriors, MAP and predictionTreat the unknown parameter as a random variable. Bayes' rule turns a prior and the likelihood into a posterior. The Beta–Bernoulli and Gaussian–Gaussian models, MAP as a regularised MLE, shrinkage, the posterior predictive, credible intervals, and a Bayesian A/B test.
75 min
5 figures
Intermediate
Testing, experiments and regression
Hypothesis tests and p-values, the bootstrap and permutation tests, A/B tests, and linear regression from the statistics view.
-
11
Hypothesis testing: p-values, power, t-tests and multiple testingHow to decide whether an observed difference is larger than noise. The null hypothesis, the p-value and what it is not, type I and type II errors and power, t-tests (Student and Welch), the two-proportion z-test and χ² tests, confidence intervals by test inversion, and multiple testing with Bonferroni and Benjamini–Hochberg.
80 min
4 figures
Intermediate -
12
The bootstrap, permutation tests and A/B testingTests and error bars by computation instead of formulas. Permutation tests shuffle the labels and are exact; the bootstrap resamples the data and is approximate, with known failures. Then A/B tests as a process - sample size, peeking, sample-ratio mismatch, CUPED variance reduction - and how to compare two ML models on one test set.
80 min
5 figures
Intermediate -
13
Linear regression from the statistics viewThe least-squares line is a random estimate. Its sampling distribution, standard errors, t-tests and confidence intervals, the difference between a confidence band and a prediction band, R² and the F-test, the Gauss–Markov theorem, residual diagnostics, and Bayesian linear regression with its link to ridge.
80 min
5 figures
Intermediate
Monte Carlo and MCMC
Compute expectations by sampling, with rejection and importance sampling, and sample hard posteriors with Markov chains.
-
14
Monte Carlo: rejection sampling, importance sampling and variance reductionCompute expectations by sampling and averaging. The Monte Carlo error bar, exact samplers (inverse CDF and rejection), importance sampling and its self-normalised version, the effective sample size and why the proposal needs heavy tails, and two ways to reduce variance (control variates and antithetic variables).
75 min
5 figures
Intermediate -
15
Markov chains and MCMC: Metropolis–Hastings, Gibbs and Hamiltonian Monte CarloHow a Markov chain forgets its start and settles at a stationary distribution, and how MCMC turns this into a sampler for distributions known only up to a constant. Detailed balance, Metropolis–Hastings, Gibbs sampling, Hamiltonian Monte Carlo with the leapfrog integrator, and the diagnostics ESS and split-R̂.
90 min
5 figures
Advanced