Videos and books

Every resource on this page is free to watch or read, from the author's, the university's or the publisher's own page. Each lesson of the track also ends with a Watch and read section that names the exact lecture, chapter or paper.

How to use it. Watch a lecture for the picture, read Sutton and Barto for the details, and run the notebook to check both. A good default: one short video before a lesson, then the lesson and its notebook, then the book section from the table at the end of this page.

A note on notation. This track writes the reward that follows $(s_t, a_t)$ as $r_t$. Sutton and Barto, the Silver course and the DeepMind 2021 course write the same reward as $R_{t+1}$. The formulas are the same after this shift of the index (see lesson 01).

Video courses #

Start here #

• Mutual Information, Reinforcement Learning By the Book (YouTube playlist). Six animated videos of about 20–30 minutes: MDPs, dynamic programming, Monte Carlo, TD learning, function approximation and policy gradients. Watch each one before the matching lesson 01–07.
• Steve Brunton, Reinforcement Learning (YouTube playlist). Board lectures on Q-learning, policy gradients and deep RL, with a view from control engineering. Pairs with lessons 05–07.

University courses #

• David Silver, UCL course on RL (2015) (course page with slides, YouTube playlist). Ten lectures, the classic first course. It follows Sutton and Barto closely: MDPs, dynamic programming, model-free prediction and control, function approximation, policy gradients, Dyna and MCTS, bandits, games. Pairs with lessons 01–07 and 10.
• DeepMind x UCL, Reinforcement Learning Lecture Series (2021) (Hado van Hasselt, Diana Borsa, Matteo Hessel; on the Google DeepMind YouTube channel). Thirteen lectures, newer and more careful than the 2015 course, with more on function approximation, off-policy learning and deep RL. The lessons link the matching lectures: lecture 3 in lessons 01–02, lecture 2 in lesson 03, lectures 5, 6, 7 and 9 in lessons 04, 05, 06 and 08, and lecture 8 in lesson 10.
• Stanford CS234, Reinforcement Learning (Emma Brunskill, 2024) (course page, YouTube playlist). A graduate course from tabular RL to policy search, offline RL and RLHF, and exploration. Pairs with lessons 01–09.

Deep RL #

• Pieter Abbeel, Foundations of Deep RL (YouTube playlist). Six lectures, about four hours in total: MDPs and exact methods, deep Q-learning, policy gradients and advantage estimation, TRPO and PPO, DDPG and SAC, model-based RL. The shortest complete tour of lessons 02 and 06–10.
• Sergey Levine, UC Berkeley CS 285, Deep Reinforcement Learning (course page, YouTube playlist, 2023). The most complete deep RL course: imitation learning, policy gradients, actor-critic, value-based methods, model-based RL, offline RL, exploration and RL for language models. Pairs with lessons 06–10.
• Hugging Face, Deep Reinforcement Learning Course (online). A free hands-on course with notebooks: Q-learning, DQN, policy gradients, A2C and PPO on standard environments. Use it when you want to train agents on real benchmarks after the notebooks of this track.

Free books #

Main textbooks #

• Richard Sutton and Andrew Barto, Reinforcement Learning: An Introduction, 2nd edition (book page, free PDF). The standard textbook. Most running examples of this track come from it: the 4×4 and 5×5 gridworlds, the random walk, cliff walking and the short corridor. Read it next to every lesson from 01 to 07 and lesson 10. Remember the $R_{t+1}$ convention.
• Alekh Agarwal, Nan Jiang, Sham Kakade and Wen Sun, Reinforcement Learning: Theory and Algorithms (free PDF). Short and rigorous: contractions, sample complexity, policy gradients, the performance difference lemma and natural policy gradient, with complete proofs. For the proofs behind lessons 01–02, 07 and 09.
• Kevin Murphy, Reinforcement Learning: An Overview (arXiv). A long survey, updated often, of the whole field: value-based, policy-based and model-based methods, offline RL, and RL for language models. Use it as a map of what exists after you finish the track.
• Csaba Szepesvári, Algorithms for Reinforcement Learning (book page, free PDF). A compact, mathematical book (about 100 pages) on value prediction, control and policy search. Lessons 02 and 04–06.

Special topics #

• Tor Lattimore and Csaba Szepesvári, Bandit Algorithms (free PDF). The reference on bandits: concentration, UCB, lower bounds, Thompson sampling. Lesson 03.
• Nathan Lambert, Reinforcement Learning from Human Feedback (free online). Reward models, PPO, RLOO and GRPO for language models, and direct alignment (DPO). Lesson 09.
• Mykel Kochenderfer, Tim Wheeler and Kyle Wray, Algorithms for Decision Making (free PDF). MDPs, online planning (including MCTS), model-based and model-free RL, imitation learning and POMDPs, with code. Lessons 02 and 10.
• Dimitri Bertsekas, Reinforcement Learning and Optimal Control (book page). The view from control and dynamic programming: rollout, approximation in value space, MPC. The book is not free, but the page has free chapters, slides and videos. Lessons 02 and 10.

Short reads and code #

• OpenAI, Spinning Up in Deep RL (online). A short, precise introduction (Key Concepts, Intro to Policy Optimization), then one page per algorithm (VPG, TRPO, PPO, DDPG, TD3, SAC) with pseudocode. Lessons 01 and 07–09.
• Lilian Weng, "Policy Gradient Algorithms" (online). The policy gradient theorem and a long list of methods (A3C, DDPG, TRPO, PPO, SAC, TD3) in one notation. Lessons 07–09.
• Lilian Weng, "The Multi-Armed Bandit Problem and Its Solutions" (online). ε-greedy, UCB and Thompson sampling with code. Lesson 03.
• Andrej Karpathy, "Deep Reinforcement Learning: Pong from Pixels" (online). REINFORCE on Atari Pong in about 130 lines of NumPy. Lesson 07.
• Huang et al., "The 37 Implementation Details of Proximal Policy Optimization" (ICLR blog track). What a working PPO really does: advantage normalization, value clipping, learning-rate annealing and more. Lessons 08–09.
• John Schulman, "Approximating KL Divergence" (online). The $k_1$, $k_2$ and $k_3$ estimators used in RLHF code. Lesson 09.
• Andy Jones, "Debugging Reinforcement Learning Systems" (online) and Alex Irpan, "Deep Reinforcement Learning Doesn't Work Yet" (online). Practical advice on why deep RL runs fail and how to find the bug. Read them before you train your first DQN or PPO agent.
• Shangtong Zhang, Python code for Sutton and Barto (GitHub). Code that reproduces almost every figure of the book. Compare it with the notebooks of lessons 01–07.
• CleanRL (GitHub). Single-file implementations of DQN, PPO, DDPG, TD3 and SAC, each a few hundred lines. Read one after lessons 06, 08 and 09.
• Gymnasium (documentation). The standard Python interface for RL environments (CartPole, MountainCar, Atari, MuJoCo). Use it to try the algorithms of this track on real tasks.

Which lesson pairs with what #

The first choice for each lesson. The Watch and read section at the end of each lesson gives more, with exact section numbers.