Videos and books

Every resource on this page is free to watch or read, from the author's or the publisher's own page. Where a classic book is not free, we say so. Each lesson of the track also ends with a Watch and read section that names the exact lecture, chapter or paper.

How to use it. Watch for the picture, read for the proofs, and write code to check both. A good default: one short video before a lesson, then the lesson and its notebook, then one book section from the table at the end of this page. All books below use the same quantities as this track. Most use $\log_2$ (bits) by default, while ML papers and libraries use the natural log (nats).

Video courses #

For the picture #

• 3Blue1Brown, Compression is Intelligence. Part 1 (above) builds entropy; Part 2, "But what is cross-entropy?" (34 min), explains cross-entropy as the cost of coding with the wrong model, and why it is the loss of language models. Watch Part 1 before lesson 01 and Part 2 before lesson 04.
• Artem Kirsanov, "The Key Equation Behind Probability" (26 min). Surprise, entropy, cross-entropy and KL divergence in one animated story. A good review after lesson 04.

University courses #

• David MacKay, Information Theory, Pattern Recognition and Neural Networks (Cambridge, 2012; YouTube playlist, course page). Sixteen lectures that follow his book (below). Lectures 2–5 cover entropy and data compression (lessons 01 and 05), lectures 6–8 cover joint entropy, mutual information and channel capacity (lessons 02 and 07). The best single course for the first half of the track.
• Raymond W. Yeung, Information Theory (CUHK; YouTube playlist). A careful, proof-based course in short sections: entropy and the information inequalities, data compression, channel capacity, differential entropy and the maximum-entropy distributions. Pairs with lessons 01–07 when you want every proof.
• MIT 6.441, Information Theory (Polyanskiy, 2016; course). Graduate lecture notes, no videos. They grew into the Polyanskiy–Wu book below.

• Stanford EE274, Data Compression: Theory and Applications (2023; YouTube playlist, course site, notes). Prefix codes, Huffman, arithmetic coding, ANS, context models, LZ77, rate–distortion, quantization and transform coding. Pairs with lessons 05 and 10.

For machine learning #

• Stanford CS236, Deep Generative Models (Ermon, 2023; YouTube playlist, course site). Maximum likelihood as KL minimization, latent-variable models, the ELBO and VAEs, flows, GANs as divergence minimization, and diffusion models. Pairs with lessons 04, 06, 08 and 09.

Free books #

Main textbooks #

• David MacKay, Information Theory, Inference, and Learning Algorithms (2003; free PDF, book page). The closest book to this track: information content and entropy (ch. 2, 4), symbol codes and arithmetic coding (ch. 5, 6), mutual information and channels (ch. 8–10), and variational methods (ch. 33). Many exercises have full solutions. Read it alongside lessons 01–07.
• Kevin Murphy, Probabilistic Machine Learning: An Introduction (2022; free PDF). Chapter 6 is a compact information theory chapter for ML: entropy, cross-entropy, perplexity, KL divergence, mutual information and the data processing inequality. Good for lessons 01–07.
• Kevin Murphy, Probabilistic Machine Learning: Advanced Topics (2023; free PDF). §2.7 (f-divergences), §5.3.6 (variational bounds on mutual information), §5.4 (rate–distortion), §5.6 (the information bottleneck), chapter 10 (variational inference) and chapter 21 (VAEs). Good for lessons 08–10.
• Yury Polyanskiy and Yihong Wu, Information Theory: From Coding to Learning (free draft). A rigorous modern book: divergences and their variational forms, mutual information, lossless and lossy compression, and statistics. Read it when you want the general proofs behind lessons 03, 06–08 and 10.
• Christopher Bishop, Pattern Recognition and Machine Learning (2006; free PDF). §1.6 (information theory, differential entropy, maximum entropy) and chapter 10 (variational inference, the two KL directions). The newer Deep Learning: Foundations and Concepts by Bishop and Bishop (free online) covers the same ideas and VAEs in a deep-learning setting.
• Cover and Thomas, Elements of Information Theory (2nd edition, 2006). The standard textbook of the field. It is not free; MacKay and Polyanskiy–Wu cover the same material.

Short reads #

• Christopher Olah, "Visual Information Theory" (2015). Entropy, cross-entropy, KL and mutual information as pictures of code lengths. Read it before or after lesson 03.
• Claude Shannon, "A Mathematical Theory of Communication" (1948). The paper that started the field. §6 defines entropy; the source coding theorem and channel capacity follow. Very readable.
• Tim Vieira, "KL-divergence as an objective function". Which KL direction needs samples and which needs the normalizer. Lesson 06.
• Delétang et al., "Language Modeling Is Compression" (ICLR 2024). Arithmetic coding with a language model, and why low cross-entropy means good compression. Lesson 05.
• Poole et al., "On Variational Bounds of Mutual Information" (2019). DV, NWJ, MINE and InfoNCE in one framework, with their bias and variance. Lesson 08.
• Lilian Weng, "Contrastive Representation Learning" (2021). A survey of contrastive losses (InfoNCE, NT-Xent) and methods (CPC, SimCLR, MoCo, CLIP). Lesson 08.
• Kingma and Welling, "An Introduction to Variational Autoencoders" (2019). The ELBO, its gap, the reparameterization trick and the importance-weighted bound, by the authors of the VAE. Lesson 09.
• Lilian Weng, "From Autoencoder to Beta-VAE" (2018). VAEs, β-VAEs and their variants with clear figures. Lessons 09 and 10.
• Alemi et al., "Fixing a Broken ELBO" (ICML 2018). The VAE loss as rate plus distortion, and why the ELBO alone cannot tell good and bad models apart. Lesson 10.

Which lesson pairs with what #

The first choice for each lesson. The Watch and read section at the end of each lesson gives more, with exact section numbers.