Actor-critic methods: advantages, GAE, A2C and soft actor-critic

Before this lesson: the policy gradient theorem, baselines and the advantage (lesson 07); TD errors, $n$-step returns and the λ-return (lesson 04); DQN, target networks and replay buffers (lesson 06); the maximization bias (lesson 05); entropy (information theory 01); the reparameterization trick (calculus 12). Time: about 85 minutes, plus the notebook.

---

Why this matters #

Almost every deep RL algorithm used today is an actor-critic method.

• PPO (lesson 09) is an actor-critic method with GAE (usually $\lambda = 0.95$). It trains robots in simulation, game agents such as OpenAI Five, and language models in RLHF, where the critic is a "value head" on the language model.
• SAC and TD3 are standard for continuous control from raw experience: robot locomotion and manipulation in MuJoCo and on real robots, where data is expensive and reuse matters.
• AlphaGo and AlphaZero (lesson 10) have a policy head and a value head, the same two parts.
• The maximum-entropy view of SAC returns in RL for language models: the KL-regularized objective of RLHF has the same soft-greedy optimum, $\pi \propto \pi_{\text{ref}} \exp(r/\beta)$ (lesson 09).

1. Predict first #

2. The picture #

The actor and the critic #

From lesson 07, the policy gradient is $\mathbb E[\sum_t \gamma^t \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, A^\pi(s_t, a_t)]$, and REINFORCE estimates the advantage by $G_t - b(s_t)$. An actor-critic method learns a value function $V_\phi(s)$ with parameters $\phi$ (the critic) and uses it in two ways: as the baseline, and to replace most of the sampled return by a prediction.

The simplest estimate uses one real reward and then the critic. This is the TD error of lesson 04:

If the critic were exact ($V_\phi = V^\pi$), then $\mathbb E[\delta_t \mid s_t, a_t] = r + \gamma \mathbb E[V^\pi(s_{t+1})] - V^\pi(s_t) = Q^\pi(s_t, a_t) - V^\pi(s_t) = A^\pi(s_t, a_t)$. So $\delta_t$ is an advantage estimate that uses only one random reward and one random next state.

The two parts help each other. The critic gives the actor a low-variance signal after every step, without waiting for the episode to end. The actor changes the policy, and the critic follows it. But a wrong critic gives the actor a wrong signal: this is the bias of actor-critic methods.

Interactive figure: loading…

Short or long: k-step advantage estimators #

Between $\delta_t$ (one real reward) and $G_t - V(s_t)$ (all real rewards) there is a family. The $k$-step advantage estimator uses $k$ real rewards and then the critic:

• $k = 1$: $\hat A_t^{(1)} = \delta_t$. Low variance (one random reward), but biased when $V \ne V^\pi$.
• $k = \infty$ (to the end of the episode): $\hat A_t^{(\infty)} = G_t - V(s_t)$. This is REINFORCE with the baseline $V$: no bias in the policy gradient for any $V$, but high variance.

Where does the bias come from? Write the critic as $V = V^\pi + e$, with an error $e(s)$. Then

The last term, $-e(s_t)$, depends only on the state, so it acts like a baseline and does not bias the policy gradient (lesson 07). The middle term is the real bias: the critic's error at the state where we bootstrap, discounted by $\gamma^k$. More real rewards push the bootstrap point further away and shrink this term, but each real reward adds noise.

Interactive figure: loading…

The figure measures the error of $\hat A_t + V(s_t)$ as an estimate of $Q^\pi(s_t, a_t)$. This removes the harmless baseline term $-e(s_t)$, and it is the same quantity as the error of the value target that trains the critic.

3. Definitions and main results #

Generalized advantage estimation #

The λ-return of lesson 04 averages $n$-step returns with weights $(1-\lambda)\lambda^{n-1}$. GAE (Schulman et al., 2016) does the same with the $k$-step advantage estimators.

Theorem (GAE). For $0 \le \lambda \le 1$ and a critic $V$,

In particular, $\lambda = 0$ gives $\delta_t$, and $\lambda = 1$ gives $G_t - V(s_t)$.

Proof.

1. Telescoping. For every $k \ge 1$,

In the second sum, each $\gamma^{l+1} V(s_{t+l+1})$ cancels the next term's $-\gamma^{l+1} V(s_{t+l+1})$; only $\gamma^k V(s_{t+k}) - V(s_t)$ is left.

2. Insert step 1 into the definition: $\hat A_t^{\text{GAE}} = (1 - \lambda) \sum_{k \ge 1} \lambda^{k-1} \sum_{l=0}^{k-1} \gamma^l \delta_{t+l}$.

3. Swap the two sums. The term $\gamma^l \delta_{t+l}$ appears for every $k \ge l + 1$. Its total weight is

for $\lambda < 1$. So $\hat A_t^{\text{GAE}} = \sum_{l \ge 0} \lambda^l \gamma^l \delta_{t+l} = \sum_{l \ge 0} (\gamma\lambda)^l \delta_{t+l}$.

4. $\lambda = 0$: only $l = 0$ is left (with $0^0 = 1$), so $\hat A_t = \delta_t$.

5. $\lambda = 1$ (defined by the right-hand side, the limit of the left): $\sum_l \gamma^l \delta_{t+l}$ is the telescoping sum of step 1 with $k \to \infty$. The boundary term $\gamma^k V(s_{t+k})$ is 0 after the episode ends (and tends to 0 when $\gamma < 1$ and $V$ is bounded), so the sum is $G_t - V(s_t)$. $\blacksquare$

If the episode ends $N$ steps after $t$, every $\hat A_t^{(k)}$ with $k \ge N$ is the same full estimate $G_t - V(s_t)$, so all the remaining weight $\lambda^{N-1}$ lands on it, exactly as for the λ-return. And $\hat A_t^{\text{GAE}} + V(s_t)$ is the λ-return with the critic $V$ as the bootstrap: the usual value target for the critic.

With an exact critic every $\lambda$ is unbiased. If $V = V^\pi$, then $\mathbb E[\delta_t \mid s_t, a_t] = A^\pi(s_t, a_t)$, and $\mathbb E[\delta_{t+l} \mid s_{t+l}] = 0$ for $l \ge 1$, because later actions also follow $\pi$. So $\mathbb E[\hat A_t \mid s_t, a_t] = A^\pi(s_t, a_t)$ for every $\lambda$. With an imperfect critic, a smaller $\lambda$ means more bias and less variance.

Computing GAE: the backward recursion #

The sum $\sum_l (\gamma\lambda)^l \delta_{t+l}$ satisfies $\hat A_t = \delta_t + \gamma\lambda\, \hat A_{t+1}$. So one backward pass over a rollout computes all advantages. A rollout of $T$ steps (it may contain several episodes) has:

• rewards $r_0, \dots, r_{T-1}$ and critic values $V(s_0), \dots, V(s_{T-1})$;
• done flags $d_t \in \{0, 1\}$: $d_t = 1$ means the episode ended with step $t$ ($s_{t+1}$ is terminal);
• the bootstrap value $V(s_T)$, the critic's value of the state after the last step.

for $t = T-1, \dots, 0$. The value targets are $\hat A_t + V(s_t)$. The factor $(1 - d_t)$ does two jobs: no bootstrapping from a terminal state, and no advantage flows back across the end of an episode into the previous one.

Termination or truncation. An episode can stop for two different reasons.

• Termination: the task really ended (the robot fell, the game was won). The value after it is 0. Set $d_t = 1$.
• Truncation: we stopped collecting data (a time limit, or the end of the rollout buffer). The next state is a normal state with a nonzero value. Do not cut the return to 0: bootstrap with $V(s_{t+1})$ of the real next state.

The end of every rollout buffer is a truncation; that is what the bootstrap value $V(s_T)$ is for. Gymnasium reports the two cases separately (terminated and truncated); only terminated should become $d_t = 1$.

Interactive figure: loading…

$\gamma$ and $\lambda$ have different jobs. $\gamma$ is part of the objective: it decides how much the agent cares about later rewards (an effective horizon of about $1/(1-\gamma)$ steps). Even with a perfect critic, a smaller $\gamma$ changes which policy is best. $\lambda$ is only part of the estimator: with a perfect critic, every $\lambda$ gives an unbiased advantage for the same objective. With an imperfect critic, $\lambda$ trades its bias against the variance of the rewards. Common choices are $\gamma = 0.99$ and $\lambda = 0.95$.

A2C and A3C #

A3C (asynchronous advantage actor-critic, Mnih et al., 2016) runs many actor-learners in parallel threads. Each one collects a few steps, computes $n$-step advantages and sends gradients to shared parameters. A2C is the synchronous version: $N$ copies of the environment step together, the rollout has $T \times N$ transitions, and one batched update follows. A2C uses the GPU better and works as well as A3C.

The policy and the critic often share a network body with two heads. A2C minimizes one loss over the batch:

where $\hat A_i$ is an $n$-step or GAE advantage, treated as a constant, and $\hat G_i = \hat A_i + V(s_i)$ is the value target. Typical weights are $c_v = 0.5$ and $c_e = 0.01$. The entropy bonus keeps the policy from becoming deterministic too early. Many implementations also normalize the advantages in each batch (subtract the mean, divide by the standard deviation); a constant shift is a baseline, and the scaling only changes the step size.

A2C is on-policy: the policy gradient needs data from the current policy, so each batch is used once and then thrown away. PPO (lesson 09) reuses each batch for a few epochs, carefully.

Continuous actions and off-policy learning: DDPG and TD3 #

A replay buffer (lesson 06) reuses every transition many times. For continuous actions the max over actions of Q-learning is a hard optimization problem, so DDPG (Lillicrap et al., 2016) learns a deterministic actor $a = \mu_\theta(s)$ that does the max: it moves the action uphill on the critic. By the deterministic policy gradient theorem (Silver et al., 2014),

the chain rule through the critic. The critic learns with the target $y = r + \gamma (1 - d)\, Q_{\phi^-}(s', \mu_{\theta^-}(s'))$ and target networks; exploration comes from noise added to the action.

DDPG overestimates $Q$, for the same reason as Q-learning (the actor finds the errors of the critic, a maximization bias, lesson 05). TD3 (Fujimoto et al., 2018) adds three fixes:

1. Clipped double Q-learning. Learn two critics and use the smaller target: $y = r + \gamma (1 - d) \min_{j=1,2} Q_{\phi_j^-}(s', \tilde a')$. The min fights the overestimation.
2. Delayed policy updates. Update the actor (and the target networks) once every 2 critic updates, so the actor follows a critic that has settled.
3. Target policy smoothing. $\tilde a' = \mu_{\theta^-}(s') + \operatorname{clip}(\epsilon, -c, c)$ with $\epsilon \sim N(0, \tilde\sigma^2)$: the target averages $Q$ over a small region, so the actor cannot exploit a narrow, wrong peak of the critic.

Soft actor-critic: maximum-entropy RL #

SAC (Haarnoja et al., 2018) changes the objective. Besides the reward, the agent is paid for the entropy of its policy in every state it visits:

Here $\alpha > 0$ is the temperature (in this section $\alpha$ is not a step size). With $\alpha = 0$ this is the usual objective. The entropy bonus makes the policy explore, makes it robust (it learns several good ways to act, not one), and makes training more stable.

Soft value functions. With the bonus, the Bellman equations of lesson 01 become

The second equation says: the soft value is the expected $Q$ plus $\alpha$ times the entropy.

The soft-greedy policy. In a state with action values $Q(s, \cdot)$, which policy maximizes $\mathbb E_{a \sim \pi}[Q(s, a)] + \alpha \mathcal H(\pi)$? For discrete actions:

Theorem (Gibbs policy). Let $Z = \sum_a e^{Q(s, a)/\alpha}$ and $\pi^*(a \mid s) = e^{Q(s, a)/\alpha} / Z$. For every distribution $\pi$ over the actions,

with equality exactly when $\pi = \pi^*$. So the best policy is $\pi^* \propto \exp(Q/\alpha)$, and the soft value is $V(s) = \alpha \log \sum_a e^{Q(s, a)/\alpha}$.

Proof.

1. From the definition of $\pi^*$: $\log \pi^*(a) = Q(s, a)/\alpha - \log Z$, so $Q(s, a) = \alpha \log \pi^*(a) + \alpha \log Z$.
2. Insert this into the left side: $\sum_a \pi(a)\big(\alpha \log \pi^*(a) + \alpha \log Z\big) - \alpha \sum_a \pi(a) \log \pi(a)$.
3. Since $\sum_a \pi(a) = 1$, this is $\alpha \log Z - \alpha \sum_a \pi(a) \log \frac{\pi(a)}{\pi^*(a)} = \alpha \log Z - \alpha\, \mathrm{KL}(\pi \,\|\, \pi^*)$.
4. $\mathrm{KL} \ge 0$, with equality only for $\pi = \pi^*$ (information theory 03). $\blacksquare$

The soft value $\alpha \log \sum_a e^{Q/\alpha}$ is a smooth maximum ("log-sum-exp"). As $\alpha \to 0$ it tends to $\max_a Q(s, a)$ and $\pi^*$ becomes greedy; as $\alpha \to \infty$, $\pi^*$ becomes uniform.

Interactive figure: loading…

SAC in practice (continuous actions). SAC is an off-policy actor-critic method with a replay buffer.

• Critics: two soft Q-networks with target networks (Polyak averaging) and the clipped double-Q target $y = r + \gamma (1 - d)\big(\min_{j} Q_{\phi_j^-}(s', \tilde a') - \alpha \log \pi_\theta(\tilde a' \mid s')\big)$ with a fresh $\tilde a' \sim \pi_\theta(\cdot \mid s')$.
• Actor: a Gaussian whose sample is squashed by $\tanh$ into the action range. It minimizes $\mathbb E_{s}\big[\mathbb E_{a \sim \pi_\theta}[\alpha \log \pi_\theta(a \mid s) - \min_j Q_{\phi_j}(s, a)]\big]$, a KL projection of $\pi$ onto $\exp(Q/\alpha)$. The gradient uses the reparameterization trick (calculus 12): $a = \tanh(\mu_\theta(s) + \sigma_\theta(s) \odot \varepsilon)$ with $\varepsilon \sim N(0, I)$, so the gradient flows through the critic into $\theta$, with much lower variance than the score function.
• Temperature: a good $\alpha$ depends on the scale of the rewards. The second SAC paper (Haarnoja et al., 2018b) tunes it automatically: keep the entropy near a target $\bar{\mathcal H}$ (often $-\dim \mathcal A$) by minimizing $\mathbb E_{a \sim \pi}[-\alpha(\log \pi(a \mid s) + \bar{\mathcal H})]$. When the entropy is below the target, this raises $\alpha$; when it is above, it lowers $\alpha$.

The tanh correction in the log-probability

The actor samples $u = \mu + \sigma \odot \varepsilon$ from a Gaussian and outputs $a = \tanh(u)$. By the change-of-variables formula (calculus 10), the density of $a$ is the density of $u$ divided by $|\det \partial a / \partial u| = \prod_i (1 - \tanh^2 u_i)$:

$$\log \pi(a \mid s) = \log N(u; \mu, \operatorname{diag}\sigma^2) - \sum_i \log\big(1 - \tanh^2(u_i)\big).$$

Forgetting the second term makes the entropy estimate wrong, and the temperature tuning then fails. Code uses the stable form $\log(1 - \tanh^2 u) = 2(\log 2 - u - \operatorname{softplus}(-2u))$.

On-policy or off-policy?

	A2C, PPO (on-policy)	DDPG, TD3, SAC (off-policy)
Data	fresh rollouts of the current policy	a replay buffer of old transitions
Policy	stochastic (softmax or Gaussian)	deterministic (DDPG, TD3) or stochastic (SAC)
Critic	$V_\phi(s)$, targets from GAE	$Q_\phi(s, a)$ (two in TD3 and SAC), TD targets
Actor gradient	score function × advantage	through the critic: $\nabla_a Q$ (TD3) or reparameterization (SAC)
Actions	discrete or continuous	mainly continuous (SAC also has a discrete version)
Sample efficiency	lower	higher (each transition is reused)
Typical use	parallel simulators, RLHF for language models	robots and expensive simulators

Answers to section 1. (1) Bootstrap with $V$ of the next state: the episode did not end, only the data did. Using 0 treats a truncation as a termination and biases the targets. (2) No. $\lambda$ only changes how we estimate the advantage; with an exact critic every $\lambda$ is unbiased. $\gamma$ is part of the objective. (3) $\alpha \to 0$ gives the usual objective and a greedy policy $\pi \to \arg\max_a Q$. A very large $\alpha$ makes the entropy dominate: the policy becomes almost uniform, whatever the rewards.

4. Worked example #

(a) GAE with an episode end. Three steps, $\gamma = 0.9$, $\lambda = 0.8$, so $\gamma\lambda = 0.72$.

$t$	$r_t$	$d_t$	$V(s_t)$
0	1	0	1.0
1	1	1	0.5
2	0	0	1.0

The episode ends with step 1; step 2 is the first step of a new episode. The rollout stops after step 2 while that episode is still running, so we bootstrap with $V(s_3) = 2.0$. Go backward:

• $t = 2$: $d_2 = 0$, so bootstrap. $\delta_2 = 0 + 0.9 \cdot 2.0 - 1.0 = 0.8$. There is nothing after it in the rollout ($\hat A_3 = 0$), so $\hat A_2 = 0.8$.
• $t = 1$: $d_1 = 1$, so no bootstrap and no carry. $\delta_1 = 1 + 0.9 \cdot 0 \cdot V(s_2) - 0.5 = 0.5$, and $\hat A_1 = 0.5 + 0.72 \cdot 0 \cdot \hat A_2 = 0.5$.
• $t = 0$: $d_0 = 0$. $\delta_0 = 1 + 0.9 \cdot 0.5 - 1.0 = 0.45$, and $\hat A_0 = 0.45 + 0.72 \cdot 0.5 = 0.81$.

Advantages $(0.81, 0.5, 0.8)$; value targets $\hat A_t + V(s_t) = (1.81, 1.0, 1.8)$.

Check $\hat A_0$ with the theorem. The episode of step 0 ends after two steps, so $\hat A_0^{(1)} = \delta_0 = 0.45$ and every longer estimator equals $\hat A_0^{(2)} = 1 + 0.9 \cdot 1 - 1.0 = 0.9$ (that is, $G_0 - V(s_0)$). The weights are $1 - \lambda = 0.2$ on $\hat A_0^{(1)}$ and the rest, $\lambda = 0.8$, on the full estimate: $0.2 \cdot 0.45 + 0.8 \cdot 0.9 = 0.09 + 0.72 = 0.81$ ✓.

What if step 1 were not an episode end? Then $\delta_1 = 1 + 0.9 \cdot 1.0 - 0.5 = 1.4$ and $\hat A_1 = 1.4 + 0.72 \cdot 0.8 = 1.976$: the advantage of step 2 now flows back.

(b) The soft-greedy policy. One state, three actions with $Q = (1, 2, 0)$.

• $\alpha = 1$: $e^{Q} = (2.718, 7.389, 1)$, sum $Z = 11.107$. So $\pi = (0.245, 0.665, 0.090)$ and $V = 1 \cdot \ln 11.107 = 2.4076$. Check: $\pi^\top Q = 0.245 + 1.330 = 1.575$ and $\mathcal H(\pi) = 0.832$, so $\pi^\top Q + \alpha \mathcal H = 1.575 + 0.832 = 2.408$ ✓. The soft value is above $\max Q = 2$ because of the entropy bonus.
• $\alpha = 0.1$: $Q/\alpha = (10, 20, 0)$; the policy is almost greedy, $\pi \approx (0.00005, 0.99995, 0)$, and $V = 2 + 0.1 \ln(1 + e^{-10} + e^{-20}) \approx 2.000005$.
• $\alpha = 10$: $Q/\alpha = (0.1, 0.2, 0)$; $\pi = (0.332, 0.367, 0.301)$, almost uniform, and $V = 10 \ln(e^{0.1} + e^{0.2} + 1) = 10 \ln 3.327 = 12.019$. Now $\alpha \mathcal H = 10 \cdot 1.095 = 10.95$ is most of the value.

5. In code #

NumPy checks both parts of the worked example. The GAE function works for one environment; the notebook extends it to many parallel environments.

import numpy as np

def gae(rewards, values, dones, last_value, gamma, lam):
    T = len(rewards)
    adv = np.zeros(T)
    next_adv, next_value = 0.0, last_value
    for t in reversed(range(T)):
        nonterminal = 1.0 - dones[t]
        delta = rewards[t] + gamma * nonterminal * next_value - values[t]
        next_adv = delta + gamma * lam * nonterminal * next_adv
        adv[t] = next_adv
        next_value = values[t]
    return adv, adv + values

r = np.array([1.0, 1.0, 0.0]); d = np.array([0, 1, 0]); v = np.array([1.0, 0.5, 1.0])
adv, targets = gae(r, v, d, last_value=2.0, gamma=0.9, lam=0.8)
print(adv.round(4), targets.round(4))        # [0.81 0.5  0.8 ] [1.81 1.   1.8 ]
print(gae(r, v, d, 2.0, 0.9, 0.0)[0].round(4))   # lambda = 0: the TD errors [0.45 0.5  0.8 ]
print(gae(r, v, d, 2.0, 0.9, 1.0)[0].round(4))   # lambda = 1: G_t - V(s_t) [0.9 0.5 0.8]

# soft-greedy policy and soft value for Q = (1, 2, 0)
Q = np.array([1.0, 2.0, 0.0])
for alpha in (0.1, 1.0, 10.0):
    z = Q / alpha
    pi = np.exp(z - z.max()); pi /= pi.sum()
    V = alpha * (z.max() + np.log(np.exp(z - z.max()).sum()))
    H = -(pi * np.log(pi + 1e-300)).sum()
    print(alpha, pi.round(3), round(V, 4), np.isclose(V, pi @ Q + alpha * H))
# 0.1 [0. 1. 0.] 2.0 True
# 1.0 [0.245 0.665 0.09 ] 2.4076 True
# 10.0 [0.332 0.367 0.301] 12.0194 True

CleanRL's ppo.py contains the same GAE loop (note that its dones array is shifted by one step: there, dones[t] says whether $s_t$ is the first state of a new episode). The log-sum-exp form of $V$ subtracts the maximum first, so it does not overflow for small $\alpha$.

6. Where it shows up #

• PPO for robots, games and language models. GAE with $\lambda \approx 0.95$, the critic fitted to $\hat A + V$, advantage normalization and an entropy bonus are the defaults of PPO (lesson 09). In RLHF the critic is a value head on a copy of the language model; critic-free methods such as GRPO replace it with the mean score of several answers to the same prompt.
• Continuous control. SAC and TD3 are standard baselines on MuJoCo and DeepMind Control tasks, and SAC has learned locomotion and grasping directly on real robots, where its sample efficiency matters.
• Game agents at scale. A3C showed that strong Atari agents can be trained on one multi-core CPU; IMPALA (Espeholt et al., 2018) scales the actor-critic idea to many machines with an off-policy correction (V-trace).
• ML for science and engineering. An actor-critic agent controls the shape of the plasma in a tokamak fusion reactor (Degrave et al., 2022; trained with the actor-critic method MPO in a simulator). Similar agents are studied for chemical process control and particle accelerator tuning, where actions are continuous and simulators are expensive.

7. Common mistakes #

• Truncation treated as termination. A time limit is not an end of the task. Set $d_t = 1$ only for real terminations; at a time limit, bootstrap with $V$ of the real next state. With auto-resetting vector environments, the returned observation is already the first state of the next episode; use the final observation that the environment reports separately.
• No reset at episode ends. Without the factor $(1 - d_t)$, GAE leaks advantages from one episode into the previous one.
• Gradient through the advantage or the target. $\hat A_t$ and $\hat A_t + V(s_t)$ are constants in the losses. Detach them; otherwise the policy loss also changes the critic, and the value loss chases a moving target.
• "λ is a second discount." $\gamma$ changes the objective; $\lambda$ only changes the estimator.
• One critic, no target network, off-policy. Off-policy actor-critic methods overestimate without clipped double Q and target networks; this is the deadly triad of lesson 06 with an actor that exploits it.
• Forgetting the entropy term in SAC's targets. The soft target contains $-\alpha \log \pi(\tilde a' \mid s')$; without it the critic learns ordinary $Q$ values, and the actor's objective no longer matches.
• A fixed temperature across tasks. The right $\alpha$ depends on the reward scale. Tune it automatically or scale the rewards.

8. Check yourself #

1. Write the actor and critic updates of one-step actor-critic.
   Answer

   $\delta = r + \gamma(1 - d) V_\phi(s') - V_\phi(s)$; critic $\phi \leftarrow \phi + \eta_\phi \delta \nabla_\phi V_\phi(s)$; actor $\theta \leftarrow \theta + \eta_\theta \delta \nabla_\theta \log \pi_\theta(a \mid s)$.
2. The critic is wrong by $e(s)$. Which part of the error of $\hat A_t^{(k)}$ biases the policy gradient, and which part does not?
   Answer

   The error $\gamma^k e(s_{t+k})$ at the bootstrap state biases it. The term $-e(s_t)$ depends only on the current state, so it acts like a baseline and adds no bias.
3. Show in one line that $\hat A_t^{(2)} = \delta_t + \gamma \delta_{t+1}$.
   Answer

   $\delta_t + \gamma\delta_{t+1} = r_t + \gamma V(s_{t+1}) - V(s_t) + \gamma r_{t+1} + \gamma^2 V(s_{t+2}) - \gamma V(s_{t+1}) = r_t + \gamma r_{t+1} + \gamma^2 V(s_{t+2}) - V(s_t)$.
4. Compute GAE for one step with $r = 2$, $V(s_0) = 1$, $\gamma = 0.9$, $\lambda = 0.95$, when (a) the episode terminates after the step, (b) the rollout is truncated and $V(s_1) = 3$.
   Answer

   (a) $\hat A_0 = \delta_0 = 2 - 1 = 1$. (b) $\hat A_0 = \delta_0 = 2 + 0.9 \cdot 3 - 1 = 3.7$. With one step, $\lambda$ plays no role.
5. Two actions with $Q = (0, \ln 3)$ and $\alpha = 1$. Give the soft-greedy policy and the soft value.
   Answer

   $e^Q = (1, 3)$, so $\pi = (0.25, 0.75)$ and $V = \ln 4 \approx 1.386$. Check: $\pi^\top Q = 0.75 \ln 3 = 0.824$ and $\mathcal H = 0.562$, sum $1.386$.
6. Name TD3's three changes to DDPG and the problem each one solves.
   Answer

   Clipped double Q (the min of two critics) against overestimation; delayed actor updates so the actor follows a settled critic; target policy smoothing (clipped noise on the target action) so the actor cannot exploit narrow errors of the critic.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Write the TD error used by the critic and the actor.

   Show answer

2. Write the one-step actor-critic updates.

   Show answer

3. Write GAE as a sum of TD errors.

   Show answer

4. Write the backward recursion of GAE with done flags.

   Show answer

5. What do $\lambda = 0$ and $\lambda = 1$ give in GAE?

   Show answer

6. What is the difference between the roles of $\gamma$ and $\lambda$?

   Show answer

7. A rollout stops at a time limit. What value follows the last step in GAE?

   Show answer

8. Name the three terms of the A2C loss.

   Show answer

9. Write SAC's maximum-entropy objective.

   Show answer

10. Give the soft-greedy policy and the soft value for discrete actions.

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Explain GAE and $\lambda$." — An exponentially weighted average of $k$-step advantage estimators, equal to $\sum_l (\gamma\lambda)^l \delta_{t+l}$; $\lambda$ trades the critic's bias against Monte Carlo variance. Give the two ends.
• "How do you handle time limits?" — Bootstrap at truncation, not at termination; keep terminated and truncated apart, and use the final observation, not the reset one.
• "Why two critics in TD3 and SAC?" — The actor exploits overestimation; the min of two critics is a pessimistic target (clipped double Q), like double Q-learning.
• "SAC or PPO?" — SAC: off-policy, sample efficient, continuous actions, needs a replay buffer and careful critics. PPO: on-policy, simple and stable, works for discrete and continuous actions and with many parallel simulators, the standard for RLHF.
• "Why the entropy bonus?" — It keeps exploration, avoids early collapse to a deterministic policy, and in SAC gives a smooth (soft) Bellman equation with a Boltzmann policy.
• "Which should learn faster, the actor or the critic?" — Usually the critic. It must track the current policy (the two-timescale analysis of actor-critic methods); a fast actor follows the critic's errors.

10. Practice #

The notebook for this lesson runs in Colab. It has five exercises, each with a check cell and a folded solution:

1. gae(rewards, values, dones, last_value, gamma, lam) for one or many parallel environments, with done flags and a bootstrap value; checked against the worked example, the two ends $\lambda = 0, 1$, and a direct sum.
2. normalize_advantages: zero mean and unit standard deviation over the whole batch.
3. Bias and variance of GAE with an imperfect critic on a chain, by simulation, for several $\lambda$; compare with the exact formulas.
4. One-step actor-critic (tabular softmax actor, tabular critic) on the 4×4 gridworld; the learned greedy policy takes the shortest path from every cell.
5. Soft value iteration on the 4×4 gridworld for several temperatures $\alpha$: the soft values approach $V^*$ as $\alpha \to 0$ and grow with $\alpha$, and the policy's entropy grows.

11. Watch and read #

L3 Policy Gradients and Advantage Estimation (Foundations of Deep RL Series)Pieter Abbeel · 41 min

Advantage estimation: from baselines to k-step estimates and GAE.

L5 DDPG and SAC (Foundations of Deep RL Series)Pieter Abbeel · 12 min

The deterministic policy gradient, DDPG and the maximum-entropy idea behind SAC.

DeepMind x UCL RL Lecture Series - Policy-Gradient and Actor-Critic methods [9/13]Google DeepMind · Hado van Hasselt · 1 h 39 min

A careful lecture from the policy gradient theorem to actor-critic methods and their variants.

• Sutton and Barto, Reinforcement Learning: An Introduction, 2nd edition, free PDF: §13.5 (actor-critic) and §12.1 (the λ-return).
• Schulman, Moritz, Levine, Jordan and Abbeel, High-dimensional continuous control using generalized advantage estimation (ICLR 2016): §2–3.
• Mnih et al., Asynchronous methods for deep reinforcement learning (ICML 2016): A3C.
• Fujimoto, van Hoof and Meger, Addressing function approximation error in actor-critic methods (ICML 2018): TD3.
• Haarnoja, Zhou, Abbeel and Levine, Soft actor-critic (ICML 2018), and Haarnoja et al., Soft actor-critic algorithms and applications (2018): automatic temperature.
• OpenAI Spinning Up: Soft Actor-Critic and TD3, with pseudocode.
• Pardo, Tavakoli, Levdik and Kormushev, Time limits in reinforcement learning (ICML 2018): termination versus truncation.