Information Theory · 44 figures

Interactive figures

Every figure in the track. Each one is a small experiment: drag, slide, and watch the numbers. The link under each figure takes you to the lesson that explains it.

Lesson 1 · Surprise, bits and entropy

In context: Lesson 1

Press Play once to draw a hidden outcome and follow its path in the question tree (yellow). Then press Play 100 games a few times: the running average settles at the expected number of questions of the tree (grey line). For $(\frac12, \frac14, \frac18, \frac18)$ this equals the entropy $H = 1.75$ bits (blue line). Try the strategy split the list in half: it needs 2 questions. Then try the distribution $(0.7, 0.2, 0.1)$ : even the best tree needs more than $H$ , because surprisals such as $0.515$ bits are not whole numbers.

In context: Lesson 1

Drag the tops of the blue bars (the probabilities); the other bars rescale so the total stays 1. The yellow bars below are the contributions $-p\log_2 p$ of each outcome; their sum is the entropy $H$ . Use one-hot and uniform to reach the two extremes $H = 0$ and $H = \log_2 K$ (the dashed line in the meter). Notice that a tiny probability contributes almost nothing, and so does a probability close to 1.

In context: Lesson 1

Lesson 2 · Joint and conditional entropy

Choose an example and press Play once. The blue questions find the first variable; the yellow questions then find the second variable given the first. The readout compares the average number of questions of each stage with $H(X)$ and $H(Y \mid X)$ . Try a coin and its copy: stage 2 needs no question. Then try weather: rain first: the entropies still add up, but whole questions cannot match surprisals that are not whole numbers.

In context: Lesson 2

Edit the weights or choose an example. Each row of the table is divided by its sum to give the bars $p(y \mid x)$ and the row entropy $H(Y \mid X = x)$ (blue). The grey bottom row is the marginal $p(y)$ with $H(Y)$ , also drawn as the dashed line. The yellow bar is $H(Y \mid X)$ , the average of the row entropies weighted by $p(x)$ . Notice: one row can reach past the dashed line, but the yellow bar never does.

In context: Lesson 2

Move the flip probability $\varepsilon$ . $X$ is a coin with $P(X = 1) = a$ , and $Y$ is $X$ with probability $1 - \varepsilon$ and flipped with probability $\varepsilon$ . The second bar is $H(X)$ , split into $H(X \mid Y)$ and the shared part $I(X;Y)$ ; the third bar is $H(Y)$ , split into $I(X;Y)$ and $H(Y \mid X)$ . At $\varepsilon = 0$ the bars overlap completely; at $\varepsilon = 0.5$ they do not overlap at all. Also try the weather table.

In context: Lesson 2

Lesson 3 · Jensen and KL divergence

Drag the blue points along the curve and move the weight sliders. The yellow dot is the centre of mass $(\mathbb{E}[X], \mathbb{E}[f(X)])$ ; the small dot on the curve is $(\mathbb{E}[X], f(\mathbb{E}[X]))$ . The aqua segment is the Jensen gap. For a convex $f$ the yellow dot is always above the curve; switch to $\log x$ or $\sqrt{x}$ and it is always below. Put all the weight on one point: the gap closes.

In context: Lesson 3

Move $a$ (the true coin) and $b$ (the model coin). The aqua curve is $\mathrm{KL}(\mathrm{Ber}(a)\,\|\,\mathrm{Ber}(b))$ as a function of $b$ ; the grey dashed curve is the reverse, $\mathrm{KL}(\mathrm{Ber}(b)\,\|\,\mathrm{Ber}(a))$ . Both touch 0 only at $b = a$ . Near $b = 0$ or $b = 1$ the aqua curve shoots up to infinity, but the dashed one stays finite. Try the preset $a = 0$ .

In context: Lesson 3

Lesson 4 · Cross-entropy as a loss

In context: Lesson 4

Choose a target: one-hot, label smoothing (slider $\varepsilon$ ) or a teacher at temperature $T$ . The blue bars are the target $t$ and the orange bars the student $q = \operatorname{softmax}(s \cdot (2, 1, 0))$ . The stacked bar splits the loss into $H(t)$ and $\mathrm{KL}(t\,\|\,q)$ . With a one-hot target, a larger $s$ always lowers the loss. With label smoothing, the loss is smallest at a moderate $s$ , and a very confident student (large $s$ ) has a larger loss.

In context: Lesson 4

Lesson 5 · Compression is prediction

Pick a symbol, then click a node of the tree to give that symbol this codeword. The bar below the tree shows the interval of each codeword in $[0, 1)$ ; its length $2^{-l}$ is the codeword's share of the budget (orange, the code's model). The blue bar shows the probabilities $p$ for comparison. Watch the Kraft sum and the average length $L$ against the entropy $H$ . Try "not prefix-free": two intervals overlap (yellow) and the code cannot be decoded instantly.

In context: Lesson 5

Move the slider to change the coin's probability of heads. The dots are the bits per flip of the best (Huffman) code for blocks of $n$ flips. They stay between the entropy $H$ (blue line) and the bound $H + 1/n$ (magenta). Notice that the rate does not always go down when $n$ grows (for $p = 0.9$ , $n = 7$ is a little worse than $n = 6$ ); only the bound does. At $p = 0.5$ there is nothing to gain: $H = 1$ bit already.

In context: Lesson 5

Add symbols with the buttons and watch the interval shrink. Each row zooms into the piece kept in the row above (yellow); the piece sizes are the model's probabilities $q(x_t \mid x_{<t})$ . The readout gives the final width $w = q(x_{1:n})$ , the ideal length $-\log_2 w$ , and the real codeword. Switch the model: a model that predicts the sequence well keeps wide intervals and needs few bits. "Learns from counts" is an adaptive model that updates its probabilities after every symbol.

In context: Lesson 5

Lesson 6 · Differential entropy and KL fits

Make the bins smaller with the $\Delta$ slider. Top: the density $p(x)$ (blue) and the histogram of $X_\Delta$ (yellow, bin probability divided by $\Delta$ ). Bottom: $H(X_\Delta)$ grows by about 1 bit per halving of $\Delta$ , while $H(X_\Delta) - \log(1/\Delta)$ (grey, dashed) settles at $h(X)$ ; the aqua gap is $\log(1/\Delta)$ . Then press "σ = 0.2" and notice that the density goes above the dashed line at 1 and $h(X)$ becomes negative.

In context: Lesson 6

Press "Fit forward KL", then "Fit reverse KL". The blue target is $p = 0.25\,\mathcal{N}(-3, 1) + 0.75\,\mathcal{N}(3, 1)$ ; the orange curve is the Gaussian $q$ . Drag its peak to choose a start (left and right move $\mu$ , up and down change $\sigma$ ), or use the "Start" buttons. Notice: the forward fit always ends at the same wide Gaussian. The reverse fit ends on the bump nearest to the start, and the two KL values show that each fit is good only for its own objective.

In context: Lesson 6

Change $\sigma_1$ , $\sigma_2$ and the correlation $\rho$ . The blue ellipse is the $1\sigma$ ellipse of $X$ ; its area is $\pi\sqrt{\det\Sigma}$ and the readout gives $h(X) = \frac12\log((2\pi e)^2\det\Sigma)$ . Push $\rho$ towards $\pm 1$ : the ellipse flattens and $h \to -\infty$ . Then move $a_1$ or $a_2$ : the yellow dashed ellipse is $Y = AX$ with $A = \operatorname{diag}(a_1, a_2)$ , a change of units, and $h$ shifts by $\log|\det A|$ .

In context: Lesson 6

Lesson 7 · Mutual information

Move the flip probability $\varepsilon$ . Left: the channel; the arrow widths are the transition probabilities. Middle: the bar $H(X)$ is split into $H(X \mid Y)$ and the shared part $I(X;Y)$ ; the bar $H(Y)$ is split into $I$ and $H(Y \mid X) = H_b(\varepsilon)$ . Right: the violet curve $1 - H_b(\varepsilon)$ and the current point. Notice that $\varepsilon = 0.9$ is as good as $\varepsilon = 0.1$ . Then move the input bias $a = P(X = 1)$ : the information drops below the violet curve.

In context: Lesson 7

Edit the weights of the joint table, or try the presets. Left: the joint $p(x, y)$ (blue). Middle: the product of its marginals $p(x)\,p(y)$ (orange), the "independent version". Right: each cell's contribution $p(x, y)\operatorname{pmi}(x, y)$ in bits, violet when positive and grey when negative. Their sum is $I(X;Y)$ . Make all rows proportional and watch $I$ become 0. Try " $Y = X^2$ ": the correlation of this pair is 0, but $I$ is not.

In context: Lesson 7

Apply $x \mapsto x^3$ or $y \mapsto e^y$ to the samples. The true mutual information $-\frac12\log(1 - \rho^2)$ does not change: the maps are invertible. The sample correlation $r$ drops, and the Gaussian formula with $r$ (orange) gives a wrong value, because the transformed data are not jointly Gaussian. The entropy of the horizontal coordinate changes too. Move $\rho$ to see how $I$ grows as $|\rho| \to 1$ .

In context: Lesson 7

Change $K$ and $N$ , then switch to "weak dependence". The magenta histogram shows 400 plug-in estimates; its mean is the magenta line. The blue line is the true $I$ , and the aqua dashed line is the true $I$ plus the predicted bias $(K - 1)^2/(2N)$ . The yellow line is one data set, with its permutation p-value. Notice: with $K = 10$ and $N = 100$ the bias is large; with large $N$ the estimates concentrate at the truth.

In context: Lesson 7

Lesson 8 · InfoNCE and contrastive learning

Press "Draw samples and fit the classifier". The blue density is $P$ (label 1) and the orange one is $Q$ (label 0). The lower chart shows the true log ratio $\log p(x)/q(x)$ (dark line) and the logit of a logistic classifier fitted on the features $1, x, x^2$ (dashed). Notice that the two curves agree between the dashed grey lines, where both classes have samples. Try "different widths": the true log ratio becomes a parabola. With 30 samples per class, look at how the fit goes wrong far from the data.

In context: Lesson 8

Move the slope $t$ . The dark curve is the generator $f$ (here $f(u) = u\log u$ of KL). The magenta line with slope $t$ stays below the curve; it touches it at the yellow dot and crosses the vertical axis at $-f^*(t)$ . The aqua segment at the ratio $u$ is the gap: it is never negative. Press "Use the best slope at u" and the gap becomes 0. Press "Show many tangent lines" to see $f$ rebuilt as the envelope of its lines. Switch to TV and notice the kink at $u = 1$ .

In context: Lesson 8

Change the true MI with $\rho$ and $d$ . Each magenta dot is the estimate from one batch of $N$ correlated Gaussian pairs, with the optimal critic. The violet line is the true MI, $I = -\frac{d}{2}\log(1 - \rho^2)$ , and the dashed line is $\log N$ . Use "Try": at low MI the three estimators agree. At high MI, notice that InfoNCE has a small spread but stops below $\log N$ , while NWJ and DV are not capped but spread widely, and their mean is often above the truth.

In context: Lesson 8

Each row is a classification over the batch. Cell $(i, j)$ is darker when the softmax of row $i$ gives more probability to candidate $y_j$ ; the framed diagonal holds the true partners. The bars compare the true MI (violet) with the InfoNCE bound $\log N - \mathcal{L}$ (magenta) and the ceiling $\log N$ (dashed). Raise $\rho$ to 0.99 and $d$ to 10: the bound stops at $\log N$ . Then increase $N$ . Choose the constant critic: the rows become uniform and the bound is 0.

In context: Lesson 8

Move the generator. The data density $p$ is blue, the generator density $q$ is orange, and the dark curve is the optimal discriminator $D^* = p/(p+q)$ . The lower chart shows the GAN value at $D^*$ as a function of the generator mean $\mu$ (magenta), with the current $\mu$ as a yellow dot. Notice: the integral $V(D^*, G)$ and $2\,\mathrm{JS} - \log 4$ agree to 4 digits. Move $\mu$ to 6: the curve is flat and the slope is almost 0.

In context: Lesson 8

Lesson 9 · The ELBO and VAEs

In context: Lesson 9

Move the mean and the variance of $q$ and compare it with the posterior. The posterior lies between the prior (dashed grey, centred at 0) and the likelihood $p(x{=}2 \mid z)$ (thin dark line, centred at 2). The two number lines show the same ELBO in two ways: $\log p(x)$ minus the gap, and the reconstruction term minus the KL to the prior. Notice that $\mathcal{N}(2, \frac12)$ has the best reconstruction but pays a large KL to the prior.

In context: Lesson 9

Increase $K$ and watch two things. Top: the orange histogram of 1000 estimates $\log \hat p_K$ becomes narrow, and its average (≈ $\mathcal{L}_K$ ) moves up toward the dashed ceiling $\log p(x)$ . Some single estimates land above the ceiling. Bottom: the gap against $K$ on log-log axes approaches the dashed $1/K$ line. Change the proposal $q$ : the gap is 0 for every $K$ when $q$ is the posterior $\mathcal{N}(1, \frac12)$ .

In context: Lesson 9

Lesson 10 · Rate–distortion and β-VAE

In context: Lesson 10

Move the encoder sliders $a$ and $s$ . The orange point is (MSE, rate) and the violet point is (MSE, $I(X;Z)$ ); the aqua bar between them is $\mathrm{KL}(q(z)\,\|\,p(z))$ . The violet point always lies on the curve. Then move $\beta$ : the yellow point is the β-VAE optimum, where the dashed line of equal loss $D + \beta R$ touches the curve. Press Go to the β-optimum, and try $\beta \ge 2$ to see posterior collapse.

In context: Lesson 10