Information Theory
Information is measured in bits of surprise. Start from coins and yes/no questions, then build the quantities inside machine learning losses and objectives: entropy, cross-entropy and KL divergence, compression, mutual information, contrastive learning, the ELBO of VAEs, and rate-distortion.
For students, engineers and researchers who train models with cross-entropy, KL terms or contrastive losses and want to know exactly what these numbers mean. You need discrete and continuous random variables, expectations and logarithms, from the first lessons of the probability track; later lessons also use the multivariate Gaussian, determinants and maximum likelihood.
Quick recap
The key takeaways of all lessons on one page
All interactive figures
Every figure of the track on one page
Videos and books
The best free courses
Glossary
Every technical term of the track with a short definition
Concept map
How the lessons of this track connect to each other and to other tracks.
Keeps every lesson page and figure of this track in this browser, so you can read it without internet.
Surprise and entropy
Surprise in bits, entropy as average surprise and as the number of yes/no questions, and the entropy of two variables.
-
01
Surprise, bits and entropyThe surprisal of an outcome is −log p; entropy is the average surprisal and the average number of yes/no questions. Bits and nats, the binary entropy, and the bounds 0 ≤ H ≤ log K.
50 min
4 figures
Beginner -
02
Joint and conditional entropy, the chain ruleThe entropy of a pair, the entropy that remains when one variable is known, and the chain rule H(X, Y) = H(X) + H(Y | X). Conditioning reduces entropy on average, but not for every value.
55 min
3 figures
Beginner
KL divergence, cross-entropy and compression
Jensen's inequality and the KL divergence, cross-entropy as the loss of classifiers and language models, and why compressing is predicting.
-
03
Jensen's inequality and the KL divergenceJensen's inequality for convex and concave functions, and the Kullback–Leibler divergence as the extra surprise of a wrong model. Gibbs' inequality (KL ≥ 0), its consequences for entropy, and why KL is not a distance.
60 min
3 figures
Intermediate -
04
Cross-entropy: the loss of classifiers and language modelsCross-entropy is the average surprise of a model on data from another distribution. It equals entropy plus KL, its minimization is maximum likelihood, its gradient through a softmax is prediction minus target, and perplexity and bits per dimension are the same number in other units.
70 min
5 figures
Intermediate -
05
Source coding: compression is predictionPrefix codes, the Kraft inequality and the source coding theorem H ≤ L* < H + 1. Huffman codes, arithmetic coding, and why a model with low cross-entropy is a good compressor.
70 min
5 figures
Intermediate
Continuous variables and mutual information
Differential entropy, the Gaussian, forward and reverse KL, mutual information and the data processing inequality.
-
06
Continuous variables: differential entropy, Gaussians and forward vs reverse KLDifferential entropy is the entropy of a density. It can be negative and it depends on the units, but the KL divergence does not. The Gaussian has the largest entropy for a given covariance, the KL between Gaussians has a closed form, and fitting a model by forward or reverse KL gives very different answers (mode covering vs mode seeking).
75 min
5 figures
Intermediate -
07
Mutual information and the data processing inequalityMutual information measures how much one variable tells about another. It is an entropy difference and the KL divergence from the joint distribution to the product of the marginals. It sees nonlinear dependence, ignores invertible changes of each variable, can only shrink under processing (the data processing inequality), and its simple plug-in estimate is biased upward.
75 min
5 figures
Intermediate
Information-theoretic objectives in ML
Variational bounds with critics and contrastive learning, the ELBO of VAEs, and the rate-distortion view of lossy compression and representations.
-
08
Critics and variational bounds: f-divergences, InfoNCE and contrastive learningWhen you can only sample from two distributions, you can still bound their divergence. A classifier (a critic) learns the density ratio, tangent lines of a convex function turn any critic into a lower bound, and InfoNCE, NWJ, DV, contrastive learning and GANs are all special cases.
90 min
5 figures
Advanced -
09
The ELBO: variational inference, VAEs and importance-weighted boundsThe evidence lower bound (ELBO) replaces an intractable integral by an optimization. Its gap to log p(x) is exactly a KL divergence to the posterior. The two standard forms of the ELBO, the VAE loss with the reparameterization trick, and the tighter importance-weighted bound (IWAE).
80 min
4 figures
Advanced -
10
Lossy compression: rate–distortion, the information bottleneck and the β-VAEThe rate–distortion function R(D) is the fewest bits per symbol for an average error D. Closed forms for a coin and a Gaussian, the Blahut–Arimoto algorithm, the information bottleneck, and the β-VAE loss as distortion plus β times rate.
75 min
5 figures
Advanced