Reinforcement Learning
An agent learns to act by trying actions and seeing rewards. Start from Markov decision processes and the Bellman equations on a small gridworld, then learn from samples with bandits, temporal-difference learning and Q-learning, scale up with DQN, policy gradients, actor-critic methods and PPO, and finish with RL for language models and planning with MCTS and AlphaZero.
For students, engineers and researchers who want to understand the algorithms behind deep RL, RLHF and AlphaZero well enough to derive, implement and debug them. You need expectations and conditional probability, matrix inverses, gradients and the chain rule, and SGD; the later lessons also use neural networks, softmax and KL divergence. Each lesson links the exact prerequisite lessons.
Quick recap
The key takeaways of all lessons on one page
All interactive figures
Every figure of the track on one page
Videos and books
The best free courses
Glossary
Every technical term of the track with a short definition
Concept map
How the lessons of this track connect to each other and to other tracks.
Keeps every lesson page and figure of this track in this browser, so you can read it without internet.
MDPs and dynamic programming
States, actions, rewards and returns, the Bellman equations, and how policy and value iteration solve a known MDP.
-
01
Markov decision processes, returns and the Bellman equationsAn agent acts, gets rewards and moves between states. The return adds up discounted rewards, the value functions V and Q average it, and the Bellman equations write them as one-step recursions that a linear solve can answer.
60 min
4 figures
Beginner -
02
Dynamic programming: policy evaluation, policy iteration and value iterationWhen the model is known, the Bellman equations can be solved exactly. Iterative policy evaluation, policy iteration and value iteration all work because the Bellman operators are contractions; the policy improvement theorem says why each greedy step helps.
70 min
4 figures
Intermediate
Learning from experience
Exploration in bandits, Monte Carlo and temporal-difference prediction, and SARSA and Q-learning for control.
-
03
Exploration and multi-armed bandits: ε-greedy, UCB and Thompson samplingTo find the best action you must try actions that look worse. Regret measures the cost; ε-greedy has linear regret, while UCB1 and Thompson sampling reach the best possible order, log T.
70 min
4 figures
Intermediate -
04
Learning values from experience: Monte Carlo and temporal-difference predictionEstimate the value of a policy from episodes, without a model. Monte Carlo averages real returns; TD bootstraps from its own estimates. n-step returns, the λ-return and eligibility traces connect the two, and importance sampling learns about one policy from another's data.
70 min
4 figures
Intermediate -
05
TD control: SARSA, Q-learning and the maximization biasLearn to act without a model by improving a policy while you estimate its action values. SARSA, expected SARSA and Q-learning differ only in their one-step target; cliff walking shows what on-policy and off-policy mean, and the maximization bias shows why the max in Q-learning overestimates and how double Q-learning fixes it.
70 min
3 figures
Intermediate
Function approximation and policy optimization
Value functions with neural networks (DQN) and learning from fixed data, the policy gradient theorem, and actor-critic methods from A2C to SAC.
-
06
Function approximation, DQN and learning from fixed dataReplace the value table by a function with weights. Gradient Monte Carlo is true SGD, semi-gradient TD is not; the deadly triad explains when values diverge. DQN stabilises Q-learning with a replay buffer and a target network, double DQN removes overestimation, and offline RL adds a conservative penalty (CQL) because the max picks actions that the data never tried.
85 min
5 figures
Advanced -
07
Policy gradients: the policy gradient theorem, REINFORCE and baselinesChange the policy parameters directly in the direction that raises the expected return. The log-derivative trick turns this gradient into an average over sampled episodes, the unknown dynamics drop out, and a baseline cuts the variance without adding bias.
75 min
4 figures
Advanced -
08
Actor-critic methods: advantages, GAE, A2C and soft actor-criticAn actor-critic method learns a policy (the actor) and a value function (the critic) together. The critic turns noisy returns into advantage estimates; GAE mixes them with one parameter λ. A2C is the on-policy version; TD3 and SAC are off-policy methods for continuous actions, and SAC adds an entropy bonus to the objective.
85 min
4 figures
Advanced
Trust regions, language models and planning
PPO and the RL used to train language models, then models of the world, imitation, MCTS and AlphaZero.
-
09
Trust regions, PPO and RL for language modelsWhy a policy-gradient step must stay close to the old policy (the performance difference lemma, TRPO's surrogate and KL trust region), how PPO's clipped objective does this cheaply, and how the same ideas train language models with RLHF, a KL penalty, RLOO and GRPO.
95 min
5 figures
Advanced -
10
Models, planning and imitation: Dyna, MCTS and AlphaZeroLearn a model of the world and plan in it (Dyna-Q, MPC), search a game tree with Monte Carlo tree search, copy an expert (behaviour cloning, DAgger), and see how AlphaZero joins search and imitation in one loop.
90 min
4 figures
Advanced