Function approximation, DQN and learning from fixed data

Before this lesson: lesson 05 (Q-learning, the maximization bias, double Q-learning); lesson 04 (MC and TD targets); neural networks and the training loop (deep learning 01, deep learning 02); SGD (optimization 05); gradients (calculus 02). Time: about 85 minutes, plus the notebook.

---

Why this matters #

Every deep RL method uses function approximation. DQN (Mnih et al., 2015) learned 49 Atari games from pixels with one architecture and one set of hyperparameters, and reached a level comparable to a professional human tester. Its two stabilisers, experience replay and target networks, are now in almost every off-policy method: DDPG, TD3 and SAC (lesson 08) use them too. The deadly triad explains why these methods can be unstable and how to debug them. Offline RL matters wherever new experiments are costly or unsafe and only logs exist: recommendation, robotics from recorded demonstrations, decisions in operations and medicine.

1. Predict first #

2. The picture #

From a table to a function #

A parametrized value function $\hat v(s, w) \approx V^\pi(s)$ has a weight vector $w \in \R^d$ with $d$ much smaller than the number of states. The simplest case is linear:

The vector $x(s) \in \R^d$ is the feature vector of state $s$. Three common choices:

• One-hot features, one per state: this is the table again ($d$ = number of states).
• State aggregation: group the states; $x(s)$ is the one-hot vector of the group of $s$. All states in a group share one value, so the value function is a staircase.
• Tile coding: several overlapping grids (tilings) over a continuous state; $x(s)$ has one 1 per tiling. Nearby states share many tiles, so they get similar values.

A neural network $\hat v(s, w)$ is the nonlinear case: the last layer is linear in features that the network learns itself.

Which weights are best? We can only be good where we are often, so the error is weighted by the on-policy distribution $\mu(s)$, the fraction of time the policy spends in $s$:

With fewer weights than states, $\overline{VE}$ can usually not be 0: making the value right in one state makes it wrong in another, and $\mu$ decides which states matter more.

Two ways to fit: Monte Carlo targets and TD targets #

As in lesson 04, each update moves the value of the current state towards a target $U_t$, now through the weights:

• Gradient Monte Carlo uses the return, $U_t = G_t$.
• Semi-gradient TD(0) uses the bootstrapped target, $U_t = r_t + \gamma\,\hat v(s_{t+1}, w)$ (and $U_t = r_t$ when $s_{t+1}$ is terminal).

The 1000-state random walk (Sutton and Barto, Example 9.1) compares them. States 1 to 1000, start in state 500. Each step jumps 1 to 100 states to the left or to the right, all 200 moves equally likely. A jump past state 1 ends the episode with reward $-1$, past state 1000 with reward $+1$; $\gamma = 1$. We use state aggregation with 10 groups of 100 states.

Interactive figure: loading…

What you see. Gradient MC converges to the best staircase: in each group, the $\mu$-weighted average of the true values ($\overline{VE} = 0.0030$). Semi-gradient TD converges to a different staircase: too flat at the ends ($\overline{VE} = 0.0136$, more than four times larger). TD's error at one group leaks into its neighbours through the bootstrapped targets. TD is still often preferred in practice: it learns online, before the episode ends, and with lower variance.

When the target moves: the $w \to 2w$ example #

Bootstrapping with a function has a new danger. Take two states that share one weight $w$ (Sutton and Barto §11.2). The first has feature $x = 1$, so its value is $w$; the second has $x = 2$, value $2w$. From the first state the agent moves to the second with reward 0; from the second the episode ends with reward 0. All true values are 0, and $w = 0$ represents them exactly.

Suppose the data contain only the first transition (an off-policy distribution: the second state is never updated). The TD error and the update are

For $\gamma > 1/2$ the factor is larger than 1, and $w$ grows geometrically. Each update raises $w$ to move the first value towards $\gamma \cdot 2w$; but that also raises the target $2w$, and by more.

Interactive figure: loading…

3. Definitions and main results #

Gradient Monte Carlo is SGD; semi-gradient TD is not #

Gradient Monte Carlo is stochastic gradient descent on $\overline{VE}$. The gradient of $\frac12 \overline{VE}(w)$ is $-\sum_s \mu(s)\,[V^\pi(s) - \hat v(s, w)]\,\nabla \hat v(s, w)$. The states $s_t$ of an episode are visited with frequencies $\mu$, and $\mathbb E[G_t \mid s_t] = V^\pi(s_t)$. So the update $\alpha\,[G_t - \hat v(s_t, w)]\,\nabla \hat v(s_t, w)$ is, on average, a step along $-\nabla \frac12\overline{VE}$: SGD (see optimization 05). With Robbins–Monro step sizes it converges to a local minimum of $\overline{VE}$, the global one in the linear case.

Semi-gradient TD is not. Its target $r_t + \gamma \hat v(s_{t+1}, w)$ depends on $w$, but the update treats it as a constant: it differentiates only the prediction $\hat v(s_t, w)$. That is why it is called semi-gradient. The full gradient of $\frac12 \delta_t^2$ would also contain the term $-\gamma \nabla \hat v(s_{t+1}, w)$ ("residual gradient" methods use it; they are slower and need two independent samples of $s_{t+1}$ to be unbiased).

For linear features we can say more. Write $x_t = x(s_t)$. The expected semi-gradient TD update is

with $x_{t+1} = 0$ after a terminal state. A vector field $b - Aw$ is the negative gradient of some function only if its Jacobian $-A$ is symmetric. Here $A$ is not symmetric in general, so semi-gradient TD does not minimize any fixed loss. The guarantees of SGD do not apply; convergence must be proved separately.

Theorem (linear TD on-policy; Tsitsiklis and Van Roy, 1997). With linear features, data from the on-policy distribution $\mu$, and Robbins–Monro step sizes, semi-gradient TD(0) converges with probability 1 to the TD fixed point

The key step of the proof shows that $A$ is positive definite under the on-policy distribution, so the expected update $w \leftarrow w + \alpha(b - Aw)$ is stable.

The TD fixed point: what it solves and how good it is

Projected Bellman equation. Let $X$ be the matrix with rows $x(s)^\top$, $D = \operatorname{diag}(\mu)$, and $\Pi$ the projection onto the span of the features in the $\mu$-weighted norm $\|v\|_\mu^2 = \sum_s \mu(s) v(s)^2$. The condition $b - A w = 0$ is $X^\top D\,(r_\pi + \gamma P_\pi X w - X w) = 0$: the Bellman error is orthogonal to the features. Equivalently, $Xw_{TD} = \Pi\,T^\pi(X w_{TD})$: the TD solution is a fixed point of "apply the Bellman operator, then project". Gradient MC instead finds $\Pi\,V^\pi$, the projection of the true values.

Error bound (continuing tasks with $\gamma < 1$; Sutton and Barto eq. 9.14):

$$\overline{VE}(w_{TD}) \le \frac{1}{1 - \gamma}\,\min_w \overline{VE}(w).$$

The proof uses that $\Pi T^\pi$ is a $\gamma$-contraction in $\|\cdot\|_\mu$ when $\mu$ is the on-policy distribution (this is where on-policy matters). For $\gamma$ close to 1 the factor can be large, but in practice TD is often close to the best fit. In the random walk ($\gamma = 1$, episodic) the bound does not apply, and TD's error is 4.6 times the best.

Off-policy. With a distribution $d \ne \mu$, $A = X^\top D_d (I - \gamma P_\pi) X$ can have eigenvalues with negative real part, and then the expected update diverges. The $w \to 2w$ example is the smallest case: $d = (1, 0)$ gives $A = 1 \cdot (1 - 2\gamma) < 0$ for $\gamma > 1/2$. Gradient-TD methods (GTD2, TDC) minimize the projected Bellman error with true gradients and are stable off-policy, at the cost of a second set of weights.

The deadly triad #

The deadly triad (Sutton and Barto §11.3) is the combination of

1. function approximation: values share parameters (linear or nonlinear), so an update in one state changes others;
2. bootstrapping: targets use current estimates, as in TD and Q-learning, not complete returns;
3. off-policy training: the data come from a distribution other than the one of the policy being evaluated.

With all three, the values can diverge, even with exact expected updates and a function class that can represent the true values. With any two of the three, instability can be avoided: tabular Q-learning (no approximation) converges; gradient Monte Carlo (no bootstrapping) is SGD; linear TD on-policy converges to the TD fixed point. Q-learning with a neural network has all three: approximation, a bootstrapped target, and a greedy target policy that differs from the ε-greedy behaviour. DQN's tricks do not remove the triad, but they make divergence much less likely in practice.

DQN #

Deep Q-network (DQN) (Mnih et al., 2015) is Q-learning with a neural network $Q_\theta(s, a)$ that outputs one value per action. For a minibatch of transitions $(s, a, r, s', d)$, where $d = 1$ if the episode ended at $s'$, it minimizes

where $\ell$ is the squared loss or the Huber loss, and no gradient flows through $y$. The factor $(1 - d)$ makes the target $y = r$ at the end of an episode. With the Huber loss $\ell(u) = \frac12 u^2$ for $|u| \le 1$ and $|u| - \frac12$ otherwise, the gradient is $\operatorname{clip}(u, -1, 1)$: a large TD error cannot produce a huge step. (The Nature DQN clipped the error term to $[-1, 1]$, which is the same.)

Two tools make the regression stable:

• Replay buffer. Store the last $N$ transitions (one million in the Atari DQN) and train on random minibatches (32). Consecutive steps are strongly correlated, and SGD assumes roughly independent samples; random samples from many episodes are much less correlated. Each transition is also used many times, which saves data. Replay needs an off-policy method: the stored actions came from older policies. Q-learning is off-policy.
• Target network $\theta^-$, a copy of $\theta$ that is updated only every $C$ steps ($C$ = 10 000 parameter updates in the Atari DQN). Between copies the targets $y$ do not depend on $\theta$, so each phase is an ordinary regression problem with fixed labels. Without it, a step that raises $Q_\theta(s, a)$ also raises $Q_\theta(s', \cdot)$ for similar states, and the target runs away from the prediction. A smooth alternative is Polyak averaging: $\theta^- \leftarrow \tau\theta + (1 - \tau)\theta^-$ after every step, with small $\tau$ such as $0.005$ (used by DDPG, TD3, SAC).

Interactive figure: loading…

The moving-target view shows a real effect, not only an intuition. With $C = 1$ the expected updates spiral away (the error grows by a factor 1.024 per step); with $C = 50$ each period of 50 steps shrinks the error by a factor 0.805, and the values converge. With on-policy data even $C = 1$ converges, as the theorem above promises. But a target network is not a guarantee: in the $w \to 2w$ example any $C$ still diverges, because there the regression itself asks for $w \leftarrow 2\gamma w$ at each copy.

Other details of the Atari DQN. The input is the last 4 frames (84 × 84 grey images) stacked, so that the network can see motion; the agent repeats each action for 4 frames; rewards are clipped to $[-1, 1]$; ε falls linearly from 1.0 to 0.1 over the first million frames; $\gamma = 0.99$; the optimizer is RMSProp.

Double DQN #

DQN inherits the maximization bias of lesson 05: $\max_{a'} Q_{\theta^-}(s', a')$ is a max over noisy estimates, and bootstrapping carries the overestimation backwards. Double DQN (van Hasselt, Guez and Silver, 2016) applies the double estimator with the two networks that DQN already has: the online network selects, the target network evaluates:

The two networks are not independent, so the bias is reduced, not removed. The change is one line of code, and on Atari it gave much less overestimation and better scores.

Interactive figure: loading…

Later extensions, combined in Rainbow (Hessel et al., 2018); each one line:

extension	idea
double DQN	the online network selects, the target network evaluates (above)
prioritized replay	sample transitions with large TD error more often; correct with importance weights
dueling network	two heads, $Q(s, a) = V(s) + A(s, a) - \frac{1}{|\mathcal A|}\sum_b A(s, b)$; learns $V$ even for actions not taken
multi-step targets	$n$-step returns $r_t + \dots + \gamma^{n-1} r_{t+n-1} + \gamma^n \max_{a'} Q_{\theta^-}(s_{t+n}, a')$: faster propagation
distributional RL (C51)	learn the distribution of the return, not only its mean
noisy nets	learned noise in the weights replaces ε-greedy exploration

For continuous actions the max over $a'$ is not available; DDPG, TD3 and SAC learn an actor for it, and TD3 and SAC use clipped double Q (the smaller of two critics) against overestimation (lesson 08).

Learning from a fixed dataset (offline RL) #

In offline RL (also called batch RL), the agent learns from a fixed dataset $\mathcal D$ of transitions collected by some behaviour policy $\pi_b$, and it cannot collect new data. Off-policy methods such as DQN can, in principle, learn from any data. In practice they often fail offline:

1. The target $r + \gamma \max_{a'} Q(s', a')$ asks for the values of all actions in $s'$, including actions that $\pi_b$ rarely or never took there: out-of-distribution (OOD) actions.
2. No data point ever corrects the value of an OOD action, so its value is whatever the network extrapolates: this is extrapolation error.
3. The max picks the largest of these values, which are often too large, and bootstrapping carries the error to earlier states. The learned policy prefers exactly the actions that the data do not support.

Online, the agent would try such an action, see the real reward and correct the value. Offline it cannot. This is the maximization bias of lesson 05 in its strongest form: the noise is not small sampling noise but unconstrained guesses.

Conservative Q-learning (CQL) (Kumar et al., 2020) adds a penalty to the TD loss. In its basic form, for each state $s$ of the data, with $a_{\mathcal D}$ the action taken in the data,

(Here $\alpha$ is the weight of the penalty, not a step size.) The log-sum-exp is a soft maximum over all actions. The gradient of the penalty with respect to the values in state $s$ is

So gradient descent pushes every action down in proportion to its softmax weight, which is largest for the most over-valued actions, and pushes the data action up. Unsupported high values fall the most. Kumar et al. prove that, for $\alpha$ large enough, the learned value of the policy, $\mathbb E_{a \sim \pi}[\hat Q(s, a)]$, is a lower bound on its true value; single values of data actions can still be pushed above their true values.

Interactive figure: loading…

Two other families avoid OOD actions in other ways:

• Policy constraints keep the learned policy close to the behaviour policy. TD3+BC (Fujimoto and Gu, 2021) adds a behaviour-cloning term to the actor loss: maximize $\lambda Q(s, \pi(s)) - (\pi(s) - a_{\mathcal D})^2$.
• Implicit Q-learning (IQL) (Kostrikov et al., 2022) never queries actions outside the data: it learns values only from dataset actions (with expectile regression), then extracts a policy by weighted behaviour cloning.

Behaviour cloning itself, learning the policy by supervised learning on the data actions, is in lesson 10.

Answers to section 1. (1) No. On-policy linear TD converges to the TD fixed point $A^{-1}b$, which can have a larger $\overline{VE}$ than the best weights (on the random walk, 0.0136 vs 0.0030); gradient MC finds the best weights. (2) Yes: $w \leftarrow (1 + \alpha(2\gamma - 1))\,w$ grows for $\gamma > 1/2$. Function approximation, bootstrapping and the off-policy distribution are all present. (3) Nothing corrects it: no data point contains $a_3$, so $Q(s, a_3)$ stays at 3 (or drifts with the network), the max picks it, and bootstrapping spreads the overestimate to earlier states.

4. Worked example #

(a) DQN and double DQN targets. One transition with $r = 0$, $\gamma = 0.99$, not done ($d = 0$). Online network: $Q_\theta(s', \cdot) = (1.0, 5.0, 2.0)$. Target network: $Q_{\theta^-}(s', \cdot) = (1.5, 3.0, 4.0)$.

1. DQN: $\max_{a'} Q_{\theta^-}(s', a') = 4.0$ (action 3), so $y = 0 + 0.99 \cdot 4.0 = 3.96$.
2. Double DQN: the online network selects $\arg\max_{a'} Q_\theta(s', a') = $ action 2 (value 5.0).
3. The target network evaluates action 2: $Q_{\theta^-}(s', 2) = 3.0$, so $y = 0.99 \cdot 3.0 = 2.97$.
4. If the episode had ended at $s'$ ($d = 1$), both targets would be $y = r = 0$.

The two networks disagree about the best action. The DQN target takes a max over the noise of one network, and so it is the higher, more optimistic one.

(b) Divergence in numbers. The $w \to 2w$ example with $\alpha = 0.1$, $\gamma = 0.9$, $w_0 = 1$, only the first state updated:

1. Factor per update: $1 + \alpha(2\gamma - 1) = 1 + 0.1 \cdot 0.8 = 1.08$.
2. After 50 updates: $w_{50} = 1.08^{50} \approx 46.9$. The true value is 0.
3. On-policy, each episode also updates the second state: $\delta = 0 - 2w$, gradient $x = 2$, so $w \leftarrow w + 0.1 \cdot (-2w) \cdot 2 = 0.6\,w$. One episode multiplies $w$ by $1.08 \cdot 0.6 = 0.648$, and $w \to 0$.

(c) The CQL penalty for one state. Three actions with $Q(s, \cdot) = (1.0, 0.5, 3.0)$; the data contain only action 1.

1. Log-sum-exp: $\log(e^{1} + e^{0.5} + e^{3}) = \log(2.718 + 1.649 + 20.086) = \log 24.453 = 3.197$.
2. Penalty: $3.197 - Q(s, a_1) = 3.197 - 1 = 2.197$.
3. Softmax: $(2.718, 1.649, 20.086)/24.453 = (0.111, 0.067, 0.821)$.
4. Gradient: $\operatorname{softmax}(Q) - e_1 = (-0.889,\ 0.067,\ 0.821)$.

Gradient descent raises $Q(s, a_1)$ and lowers $Q(s, a_3)$ the most: the unsupported value 3.0 falls fastest.

5. In code #

The cells below check the worked example with NumPy. In a real DQN the same target code runs on batches; the notebook writes it with done flags and adds fitted Q-iteration with a scikit-learn regressor, a DQN-like method without a deep-learning library.

import numpy as np
from scipy.special import logsumexp, softmax

# (a) DQN vs double DQN, vectorized over a batch with done flags
r = np.array([0.0, 1.0]); done = np.array([0.0, 1.0]); gamma = 0.99
q_online_next = np.array([[1.0, 5.0, 2.0], [0.0, 0.0, 0.0]])
q_target_next = np.array([[1.5, 3.0, 4.0], [9.0, 9.0, 9.0]])
y_dqn = r + gamma * (1 - done) * q_target_next.max(axis=1)
a_star = q_online_next.argmax(axis=1)
y_double = r + gamma * (1 - done) * q_target_next[np.arange(2), a_star]
print(y_dqn, y_double)                      # [3.96 1.  ] [2.97 1.  ]

# (b) the w -> 2w example
alpha, gamma = 0.1, 0.9
w = 1.0
for _ in range(50):
    w += alpha * (gamma * 2 * w - w) * 1.0  # only the first state, feature 1
print(round(w, 1), round(1.08 ** 50, 1))    # 46.9 46.9

# (c) CQL penalty and its gradient for one state, data action a_1
Q = np.array([1.0, 0.5, 3.0])
print(round(logsumexp(Q) - Q[0], 3))        # 2.197
print((softmax(Q) - np.eye(3)[0]).round(3)) # [-0.889  0.067  0.821]

The done flag matters in the second transition: its target is the reward 1.0, not $1 + 0.99 \cdot 9$.

6. Where it shows up #

• Games and control from pixels. DQN and Rainbow on Atari; the replay buffer and target network (or Polyak averaging) are in DDPG, TD3, SAC and most off-policy deep RL code (CleanRL, Stable-Baselines3).
• Recommendation and operations. Q-learning with function approximation is trained on logs of user sessions or system states; this is offline or nearly offline RL, so extrapolation error and conservative methods matter.
• Robotics from recorded data. Large offline datasets of robot interaction are trained with conservative or constrained Q-learning before any real deployment.
• Language models. Value-based offline RL ideas (implicit Q-learning, conservative values) appear in methods that fine-tune language models from logged conversations with rewards.
• ML for science. Control of physical processes from logged sensor data (for example fusion or chemical plant operation logs) is offline RL: the logged actions cover only safe operating regions, and a plain Q-learner would trust its guesses outside them. Fitted Q-iteration with tree ensembles is a common, stable baseline on such tabular, low-dimensional data.

7. Common mistakes #

• Gradient through the target. In DQN the target $y$ is a constant (torch.no_grad() or .detach()). Backpropagating into it gives a different method (residual gradient), usually slower.
• Forgetting the done flag. At a terminal transition $y = r$. Without $(1 - d)$ the agent adds the value of a state that does not exist; with time limits, a transition cut by the time limit is not terminal.
• "Semi-gradient TD minimizes the squared TD error." It does not minimize any fixed loss; on-policy linear TD converges to the TD fixed point, which is not the minimizer of $\overline{VE}$.
• "The target network removes the deadly triad." It makes each phase a regression with fixed labels and often helps a lot, but divergence is still possible (the $w \to 2w$ example diverges for every $C$).
• "Double DQN needs two new networks." It reuses DQN's online and target networks; only the target formula changes.
• Trusting offline Q-values. A high value for an action that is rare in the data is a warning sign, not a discovery. Check the data coverage and evaluate the policy with off-policy evaluation before deployment.
• Mixing up the two $\alpha$. In CQL, $\alpha$ is the penalty weight; the learning rate of the network is $\eta$.

8. Check yourself #

1. Why is gradient MC true SGD on $\overline{VE}$, but semi-gradient TD is not SGD on any loss?
   Answer

   MC's target $G_t$ does not depend on $w$ and has mean $V^\pi(s_t)$, so the update is an unbiased sample of $-\nabla \frac12 \overline{VE}$. TD's target depends on $w$ but is treated as a constant; the expected update $b - Aw$ has the non-symmetric Jacobian $-A$, so it is not the gradient of any function.
2. Name the three parts of the deadly triad and one method that avoids each.
   Answer

   Function approximation (tabular Q-learning), bootstrapping (gradient Monte Carlo), off-policy data (on-policy linear TD).
3. With $\alpha = 0.05$ and $\gamma = 0.99$, what is the factor per update in the $w \to 2w$ example, and per episode with on-policy updates?
   Answer

   Off-policy: $1 + 0.05 \cdot 0.98 = 1.049$ (diverges). On-policy: $1.049 \cdot (1 - 4 \cdot 0.05) = 1.049 \cdot 0.8 = 0.839$ (converges).
4. What does the replay buffer fix, and what does the target network fix?
   Answer

   Replay: strongly correlated consecutive samples (and it reuses data). Target network: the regression target that moves with every gradient step; with $\theta^-$ fixed, each phase is a regression with fixed labels.
5. Compute the DQN and double DQN targets for $r = 1$, $\gamma = 0.9$, not done, $Q_\theta(s', \cdot) = (2, 0)$, $Q_{\theta^-}(s', \cdot) = (1, 3)$.
   Answer

   DQN: $1 + 0.9 \cdot 3 = 3.7$. Double DQN: the online network picks action 1, the target network gives 1, so $y = 1 + 0.9 \cdot 1 = 1.9$.
6. Why does offline Q-learning overestimate more than online Q-learning, and how does CQL respond?
   Answer

   The max in the target queries actions that the data never contain; their values are never corrected, and the max picks the largest guesses. Online, the agent would try them and correct them. CQL adds $\alpha(\log\sum_a e^{Q(s,a)} - Q(s, a_{\mathcal D}))$, which pushes all values down by their softmax weight and the data action up.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Write the semi-gradient TD(0) update for a parametrized value function.

   Show answer

2. Why is it called "semi-gradient"?

   Show answer

3. Name the three parts of the deadly triad.

   Show answer

4. In the $w \to 2w$ example, what is the factor per update of the first state?

   Show answer

5. Write the DQN target.

   Show answer

6. Write the double DQN target.

   Show answer

7. What is the purpose of the replay buffer?

   Show answer

8. What is the purpose of the target network?

   Show answer

9. What is extrapolation error in offline RL?

   Show answer

10. Compute the CQL penalty gradient for $Q(s, \cdot) = (1, 0.5, 3)$ with data action $a_1$.

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Why does DQN need a target network?" The target depends on the same weights; a step that raises $Q(s, a)$ also raises $Q(s', \cdot)$ and the target runs away. A frozen copy makes each phase a regression with fixed labels.
• "Why a replay buffer? What does it require?" It decorrelates the samples and reuses data; it requires an off-policy learner, since the stored actions came from older policies.
• "Why can Q-learning with neural networks diverge?" The deadly triad: approximation, bootstrapping, off-policy data. Give the $w \to 2w$ example: factor $1 + \alpha(2\gamma - 1)$.
• "Where does overestimation come from, and how does double DQN fix it?" The max over noisy estimates; the online network selects, the target network evaluates. TD3 and SAC take the min of two critics.
• "Why is offline RL hard, and what does CQL do?" Out-of-distribution actions and extrapolation error; CQL penalizes the soft max of $Q$ and rewards data actions, which gives conservative values. Alternatives: policy constraints (TD3+BC), IQL.
• "Does TD with linear features converge to the best fit?" On-policy it converges, but to the TD fixed point, not to the $\overline{VE}$ minimizer; gradient MC finds the minimizer.

10. Practice #

The notebook for this lesson runs in Colab. It has five exercises, each with a check cell and a folded solution:

1. Gradient MC and semi-gradient TD(0) with state aggregation on the 1000-state random walk; compare with the exact fixed points computed by linear algebra.
2. The $w \to 2w$ example: divergence with off-policy updates and convergence with on-policy updates.
3. Vectorized dqn_targets and double_dqn_targets with done flags.
4. Fitted Q-iteration with a scikit-learn regressor on a small task with a continuous state: a batch, DQN-like method where each iteration is one regression with a frozen target.
5. Offline Q-learning on a gridworld with partial data coverage, then with a CQL penalty: measure the overestimation.

11. Watch and read #

RL Course by David Silver - Lecture 6: Value Function ApproximationGoogle DeepMind · David Silver, UCL RL course · 97 min

Features, gradient MC and TD with approximation, convergence tables, and DQN with experience replay.

L2 Deep Q-Learning (Foundations of Deep RL Series)Pieter Abbeel · 34 min

From Q-learning to DQN, replay and target networks, double DQN and the Rainbow extensions, in one compact lecture.

DeepMind x UCL RL Lecture Series - Function Approximation [7/13]Google DeepMind · Hado van Hasselt · 150 min

The full theory: linear TD, the TD fixed point, the deadly triad and its examples.

Function Approximation | Reinforcement Learning Part 5Mutual Information · 21 min

A visual introduction to features, semi-gradient methods and why bootstrapping with a function is different.

CS 285: Lecture 15, Part 2: Offline Reinforcement LearningSergey Levine · UC Berkeley CS 285 · 26 min

Why offline RL fails with standard off-policy methods: distribution shift and out-of-distribution actions.

• Richard Sutton and Andrew Barto, Reinforcement Learning: An Introduction, 2nd ed., free PDF: chapter 9 (§9.1–9.5: $\overline{VE}$, gradient MC, semi-gradient TD, linear methods and the TD fixed point; Example 9.1 is the 1000-state random walk) and §11.1–11.3 (off-policy divergence, the $w \to 2w$ example, the deadly triad).
• Volodymyr Mnih et al., Human-level control through deep reinforcement learning, Nature 2015: the DQN paper; its Methods section lists all hyperparameters.
• Hado van Hasselt, Arthur Guez, David Silver, Deep Reinforcement Learning with Double Q-learning.
• Matteo Hessel et al., Rainbow: Combining Improvements in Deep Reinforcement Learning.
• Sergey Levine, Aviral Kumar, George Tucker, Justin Fu, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, §1–4.
• Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine, Conservative Q-Learning for Offline Reinforcement Learning.
• Costa Huang et al., CleanRL single-file DQN: about 200 lines that contain everything in section 3.