Reinforcement Learning · 41 figures

Interactive figures

Every figure in the track. Each one is a small experiment: drag, slide, and watch the numbers. The link under each figure takes you to the lesson that explains it.

Lesson 1 · MDPs and Bellman equations

In context: Lesson 1

Move the sliders to change the policy. The orange edges are the actions with probabilities $\pi(a \mid s)$ ; the aqua labels are the transition probabilities and rewards; the blue numbers are the values. Each $Q$ is "reward now plus $\gamma$ times the average next value", and $V^\pi$ is the $\pi$ -average of the $Q$ values. Then switch to max to see the optimality equation: the average over actions becomes a max (the arc).

In context: Lesson 1

Click a cell to change its action (random, then ↑, →, ↓, ←, then random again). The values are recomputed by one linear solve; darker blue means a higher value. For the random policy, the cell next to a corner has $V^\pi = -14$ : on average a random walk needs 14 steps. The side panel shows the backup of the selected cell: $V^\pi(s)$ is the $\pi$ -average of the $Q$ values. Now make two neighbouring cells point at each other: those two states never end, so their value is $-\infty$ , and so is the value of every state from which the random policy can walk into them.

In context: Lesson 1

Lesson 2 · Policy and value iteration

Press "Evaluate: one sweep" a few times, then "Improve (greedy)". The blue values are $V_k$ ; the orange arrows are the policy (several arrows = a tie, the agent picks one of them at random). After three sweeps the greedy arrows are already optimal. Then try "Run policy iteration" and "Run value iteration", and the other worlds: in the slippery world, raise the slip and watch the policy move away from the pit; in the cliff world, the safe path moves up when the slip is above 0.

In context: Lesson 2

Drag the two points. The robot vacuum has two states, so a value function is a point $(V(H), V(L))$ . The blue $V$ and the violet $U$ are two guesses; the arrows show where $T$ sends them. The dashed square around $V$ has half-width $\|V - U\|_\infty$ , so $U$ lies on its edge. The shaded square around $TV$ is $\gamma$ times smaller, and $TU$ always lands inside it. Press "Apply T" a few times: both points move to the same fixed point $V^*$ . The small chart is the max lemma in state H: the gap between the two maxima is never larger than the largest vertical gap.

In context: Lesson 2

Move $\gamma$ and predict the slope. On a log scale, the true error $\|V_k - V^*\|_\infty$ (blue) follows the theorem's line $\gamma^k \|V_0 - V^*\|_\infty$ (violet, dashed); the slope is $\log \gamma$ . The grey dashed line is the stopping-rule bound, which you can compute without knowing $V^*$ . The orange line marks the iteration from which the greedy policy is optimal: on the 5×5 grid with $\gamma = 0.9$ this is iteration 29, while the value error is still about 1.15.

In context: Lesson 2

Lesson 3 · Bandits and exploration

Pull the arms by hand first (the buttons 1–4), about 20 times, and guess which arm is best; then press Reveal. Then press Reset and let an algorithm play. The blue dot is the sample mean $\hat\mu_a$ ; in UCB1 mode the yellow bar above it is the exploration bonus, and its top is the index that UCB1 maximizes. In Thompson mode the yellow diamond is the posterior sample $\theta_a$ . The last pulled arm is orange. After Reveal, the aqua dashes are the true means and the chart shows the cumulative regret. Notice how the algorithms stop pulling the bad arms, but never completely.

In context: Lesson 3

Move $n$ and $t$ . The yellow curve is the bonus $\sqrt{2 \ln t / n}$ for the chosen $t$ ; grey curves are $t = 10, 100, 1000, 10000$ . Doubling $n$ divides the bonus by $\sqrt 2$ ; multiplying $t$ by 10 changes it only a little. The number line shows the interval $\hat\mu \pm c$ and the index. Switch to Hoeffding vs exact to compare the bound with the exact tail for fair coin flips: for $n = 20$ , $c = 0.2$ the bound is 0.2019 and the true probability is 0.0577. The bound is loose, but it holds for every reward distribution on $[0, 1]$ .

In context: Lesson 3

Press "Sample once" a few times, then "Sample 100 times". The curves are the posteriors Beta(6, 2) (arm 1: 5 successes, 1 failure) and Beta(3, 3) (arm 2: 2 and 2). Each draw marks one $\theta$ per arm; the orange line is the largest, and that arm is pulled. The frequency of arm 1 approaches the exact $P(\theta_1 > \theta_2) = 28/33 \approx 0.848$ . Try more data (same means, about ten times the pulls): the posteriors narrow, and arm 2 almost never wins.

In context: Lesson 3

Before you press Run, predict which curves keep growing like a straight line and which bend. The mean regret over 100 runs is shown for four Bernoulli arms with means 0.2, 0.4, 0.6, 0.8 and $T = 5000$ . Greedy and fixed ε-greedy grow linearly (greedy leaves the chart); UCB1, decaying ε and Thompson sampling bend. Switch the $t$ axis to logarithmic: a curve that grows like $\log t$ becomes a straight line there. Thompson sampling is almost straight already; UCB1 still bends upward at this horizon, because its $\log T$ behaviour starts late (its constant is large). The table divides the final regret by $\ln T$ .

In context: Lesson 3

Lesson 4 · Monte Carlo and TD prediction

Choose a method and press Step. The blue bars are the current values (all start at 0.5), the dashed lines the true values, and the orange dot the agent. The default episode is C → B → C → D → E → right end. With TD(0), every TD error before the last step is 0, so only E moves. With MC, nothing moves until the end; then every visited state moves toward the return 1. With TD(λ), the orange trace bars show which states share the final TD error. Then switch to learning curves (mean of 100 runs) and look for the best α of each method: the best TD(0) curve ends lower than the best MC curve.

In context: Lesson 4

Step through the episode C → B → C → D → E → +1. The orange bars are the traces; C was visited twice, so its bar has two pieces (the older visit is lighter). All TD errors are 0 until the reward arrives; then each state moves by $\alpha\,\delta\,z(s)$ (blue numbers). Predict first: at the last step, is $z(C)$ larger or smaller than $z(D)$ ? Then move λ: at λ = 0 only E moves; at λ = 1 every visit counts fully.

In context: Lesson 4

Lesson 5 · SARSA and Q-learning

Switch the next action $a'$ and move the sliders. The blue bars are $Q(s', a_1)$ and $Q(s', a_2)$ , with their ε-greedy probabilities in orange. The three targets are SARSA, expected SARSA and Q-learning. Only the SARSA target changes when you switch $a'$ . Press "Sample a′ 200 times": the dashed line, the mean of many SARSA targets, lands on the expected SARSA target. Set $\varepsilon = 0$ and expected SARSA becomes Q-learning.

In context: Lesson 5

Predict the two paths first, then press "Train 500 episodes". The grid shows run 1 of the selected method: the shading is $\max_a Q(s, a)$ (blue, darker is higher), the small arrow is the greedy action, and the orange line is the greedy path from S. The chart shows the online return, the total reward per episode while the agent explores (10 runs, smoothed). Compare Q-learning, SARSA and expected SARSA: which one has the shortest greedy path, and which one collects the most reward? Then set $\varepsilon = 0$ and train again.

In context: Lesson 5

Predict first, then press "Repeat 1000 times". Ten actions all have true value 0. Each estimate is the mean of $n = 10$ samples of $N(0, 1)$ . The orange histogram is the single estimator $\max_i \hat X_i$ : its mean is about $+0.49$ , not 0. The violet histogram is the double estimator: choose $i^*$ with the blue set, read the value from the independent violet set. Its mean is 0. Make $K = 1$ (no choice, no bias) and make $n$ large (less noise, less bias).

In context: Lesson 5

Lesson 6 · Function approximation and DQN

Predict which staircase fits better, then press "Train 1000 episodes" a few times. The dashed black line is the true value $V^\pi(s)$ . The aqua staircase is gradient Monte Carlo, the violet one semi-gradient TD(0), both trained on the same episodes. The grey curve at the bottom is the on-policy distribution $\mu$ (the walk spends most time in the middle). "Show the limits" draws where each method converges. Notice where TD is worse: at the ends, where $\mu$ is small.

In context: Lesson 6

Move $\gamma$ across $0.5$ , then switch the updates to "both states". The chart shows $|w|$ on a log scale over 100 episodes, starting from $w = 1$ . With only the first state updated (magenta), $w$ grows for every $\gamma > 1/2$ , even though $w = 0$ is exactly right. With on-policy updates (blue), the second state is also updated, towards its own target 0, with gradient $x = 2$ . This pulls $w$ back, and it shrinks to 0.

In context: Lesson 6

Press "Next step" six times to walk through one iteration of DQN: act, store, sample, compute targets with the target network, take a gradient step on the online network, and copy. Then switch to "moving target". There, a linear Q-network with two weights must learn the values of two states, $(0.9, 1)$ (aqua), from off-policy data, and the second state's feature doubles the first one's weight. The blue path shows the values over 1000 updates; the violet dots are the regression targets after each copy. Predict first: with $C = 1$ (no target network), do the values reach the aqua point? Then try larger $C$ , and on-policy data.

In context: Lesson 6

Press "50 steps" with $\alpha = 0$ , then set $\alpha = 1$ and press again. One state, three actions, a one-step decision (the target is the reward). The dashed lines are the true values $(1.0, 0.5, 0.0)$ . The data contain only $a_1$ (grey coverage bars); the magenta actions never appear in the data. The values start at $(1.0, 0.5, 3.0)$ : the network's guess for $a_3$ is an extrapolation error, and the greedy action (orange outline) is $a_3$ . Plain offline Q-learning ( $\alpha = 0$ ) never changes it. The CQL penalty pushes $a_3$ down and $a_1$ up until $a_1$ is greedy.

In context: Lesson 6

Lesson 7 · Policy gradients and REINFORCE

Drag the logits $\theta_1, \theta_2, \theta_3$ , then press Sample. The bars are the policy π and the rewards $r = (1, 0, 2)$ . The plane shows gradient vectors (all of them have entries that add up to 0, so they lie in a plane). Each orange point $g_a = (r_a - b)(e_a - \pi)$ is one possible REINFORCE sample; its area is its probability $\pi_a$ . The black arrow is the exact gradient $\nabla J$ , and the dashed orange arrow is the mean of your samples. Move the baseline $b$ : the points move, the black arrow does not, and the variance in the readout changes.

In context: Lesson 7

Press "Sample 20 actions". The aqua curve is the reward $r(a)$ , and the orange area is the policy $N(\mu, \sigma^2)$ . Each orange dot is a sampled action on the reward curve. Under the axis, each violet arrow is that sample's push on $\mu$ , $\frac{a - \mu}{\sigma^2}(r(a) - b)$ . Then press "Gradient step" a few times and watch $\mu$ climb to the peak. Turn on the baseline $b = J(\mu, \sigma)$ : the arrows change a lot, their mean does not.

In context: Lesson 7

Click a step $t$ . The top row shows the actions $a_t$ of one episode, the second row the rewards. The bracket shows which rewards multiply the score of the selected step. In the "full return" mode all rewards count; in the "reward-to-go" mode the rewards before $t$ are dropped (step 4 of the proof: their term has mean 0). In this toy problem every step is an independent coin with $\pi(a = 1) = p$ and reward $r_t = a_t$ . The chart shows the exact variance of the three estimators. Their means are all equal.

In context: Lesson 7

Predict the best $p$ , then press "Run 100 episodes" a few times. The cells show the corridor; X has switched actions. The first chart is $J(p)$ ; the aqua line marks $p^* \approx 0.586$ . Each dot is the current policy of one REINFORCE run, and the second chart shows $p$ over episodes: orange runs use no baseline, blue runs use a learned constant baseline. All runs start at $p = 0.1$ . Raise the step size to see the runs without a baseline become unstable first.

In context: Lesson 7

Lesson 8 · Actor-critic, GAE and SAC

Press "1 step (slow)" a few times, then "100 steps". The robot vacuum of lesson 01: the boxes show the actor's policy (orange bars) and the critic's values (blue bars) in the states H and L. The slow step shows the five phases of one update. The charts show $V(H)$ and $V(L)$ against the optimal values $V^*$ (dashed), and the probabilities of the optimal actions. Watch $\pi(\text{charge} \mid L)$ in the first 100 steps: the critic starts at $V = 0$ , so "clean" (reward 1 now) looks better than "charge" (reward 0 now) until the critic learns that $V(H)$ is large.

In context: Lesson 8

Move the critic error $c$ and the reward noise $\sigma$ , then find the best $\lambda$ . A chain of 20 steps, each with a reward $r \sim N(1, \sigma^2)$ , $\gamma = 0.95$ . The critic is wrong by $c$ in every state. The top chart shows the error of 40 sampled estimates (the orange dots are at the current setting) and their mean, the bias. The bottom chart shows bias², variance and their sum, the mean squared error, for every $\lambda$ . Switch to "k-step" to see the same trade-off for $k$ . A perfect critic favours small $\lambda$ ; a bad critic and quiet rewards favour $\lambda$ near 1.

In context: Lesson 8

Press "Step back" until all steps are computed. The data is the worked example below: rewards, values, done flags, and the bootstrap value $V(s_T)$ on the right. Each step shows $\delta_t$ and $\hat A_t$ in violet; the violet arrow carries $\gamma\lambda \hat A_{t+1}$ back, and it is cut at a dashed line where an episode ends. Click a done flag to toggle it. Make the last step terminal: $V(s_T)$ is no longer used.

In context: Lesson 8

Move the temperature $\alpha$ . The blue bars are the action values $Q$ , the orange bars the soft-greedy policy $\pi \propto \exp(Q/\alpha)$ . The violet line is the soft value $V$ , the dashed line is $\max_a Q$ . The chart shows the entropy of $\pi$ (it rises to $\log K$ ) and how far $V$ is above $\max_a Q$ . The readout checks $V = \pi^\top Q + \alpha \mathcal H(\pi)$ . Edit the $Q$ values to see that only their differences, divided by $\alpha$ , shape $\pi$ .

In context: Lesson 8

Lesson 9 · PPO and RL for LLMs

Predict, then slide. Move the step size $t$ along the policy gradient. The blue curve is the true improvement $J(\pi_{\text{new}}) - J(\pi_{\text{old}})$ , computed exactly. The violet curve is the surrogate $L$ , the improvement predicted from old data. Near $t = 0$ they agree (same value, same slope). Far away the surrogate keeps rising while the true improvement falls below zero. The lower chart shows the KL from the old policy (solid) and its Fisher approximation $\tfrac12 t^2 u^\top F u$ (dashed). The shaded band is the trust region $\mathrm{KL} \le \delta$ .

In context: Lesson 9

Drag the orange point along the curve (or use the $\rho$ slider), then switch the sign of $\hat A$ . The solid orange curve is PPO's per-sample objective $\min(\rho \hat A, \operatorname{clip}(\rho, 1-\varepsilon, 1+\varepsilon)\hat A)$ ; the dashed line is the unclipped $\rho \hat A$ . In the magenta region the curve is flat, so the gradient is zero. Notice that the flat part is only on one side: a sample that moved the wrong way keeps its full gradient.

In context: Lesson 9

Pick a response and move the sliders. The tree shows a toy model that writes two tokens from $\{0, 1\}$ . Each edge shows the policy probability $\pi_\theta$ and the reference probability $\pi_{\text{ref}}$ . For the chosen response, the panel lists the per-token log-ratios, the KL rewards $-\beta \log(\pi_\theta/\pi_{\text{ref}})$ and the reward-model score. Check that the per-token rewards add up to $r - \beta \log(\pi_\theta(y)/\pi_{\text{ref}}(y))$ , and that the sequence KL equals the sum of the per-token KLs.

In context: Lesson 9

Move $\beta$ and drag the reward dots. Grey bars are $\pi_{\text{ref}}$ ; orange bars are the optimum $\pi^* \propto \pi_{\text{ref}} \exp(r/\beta)$ ; the aqua dots are the rewards $r(y)$ . The lower chart shows, for every $\beta$ , the expected reward $\mathbb E_{\pi^*}[r]$ (aqua) and $\mathrm{KL}(\pi^* \| \pi_{\text{ref}})$ (magenta). Predict first: what happens to $\pi^*$ when $\beta \to 0$ ? Can $\pi^*$ put mass on a response with $\pi_{\text{ref}} = 0$ ?

In context: Lesson 9

Lesson 10 · Model-based RL and AlphaZero

In context: Lesson 10

Predict, then step. Click the move (A, B or C) that you think the next Select will choose, then press Next phase four times: Select, Expand, Evaluate, Backup. The orange path is the selection; edge width grows with the visit count $N$ ; Q is the mean result for the player who made the move. Under each leaf, p is your win probability there. Then press Run 100 a few times. Move A has the best average (0.633), but the opponent can answer it with A3 ( $p = 0.2$ ). Watch the visits move to B, the best move against a good opponent. Set $c = 0$ and press Reset a few times: without exploration the search can lock onto a wrong move.

In context: Lesson 10

Predict first: if you double the horizon $T$ , how much does the expected cost of BC grow? Move $T$ . BC cars leave the road and never come back, because their data has only on-road states (the dots on the left show where the data has expert labels). Switch to the DAgger rounds: each round adds the positions the learner visited, labelled by the expert, and the cars return to the road. The chart compares the costs with $\varepsilon T^2$ and $\varepsilon T$ .

In context: Lesson 10