Reinforcement Learning · 137

Glossary

The technical terms of this track, with short definitions (and the Persian term where available). Each term links to the lesson where it is introduced.

A2C: Advantage actor-critic: the synchronous version of A3C. Many copies of the environment step together, and one update uses $n$ -step advantages from all of them. Introduced in: Lesson 8
action: A choice the agent makes in a state, written $a_t$ ; the set of actions is $\mathcal A$ . Introduced in: Lesson 1
action-value function: $Q^\pi(s, a) = \mathbb E_\pi[G_t \mid s_t = s, a_t = a]$ , the expected return after taking action $a$ in state $s$ and then following $\pi$ . Introduced in: Lesson 1
actor-critic: A method that learns a policy (the actor) and a value function (the critic) together. The critic supplies advantage estimates for the actor's policy gradient. Introduced in: Lesson 8
advantage: $A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)$ : how much better action $a$ is than the policy's average action in state $s$ . Introduced in: Lesson 1
agent: The learner and decision maker. In each step it observes a state, picks an action and receives a reward. Introduced in: Lesson 1
AlphaZero: Self-play with MCTS guided by a policy–value network. The network is trained to predict the search's visit distribution and the game result, so search and imitation improve each other. Introduced in: Lesson 10
backup diagram: A small tree that shows how a value is computed from the values of the next states or actions. Introduced in: Lesson 1
baseline: A function $b(s)$ subtracted from the return in the policy gradient. It does not depend on the action, so it adds no bias, and a good choice such as $V^\pi(s)$ lowers the variance. Introduced in: Lesson 7
behaviour cloning: Imitation learning by supervised learning: fit $\pi(a \mid s)$ to the expert's state–action pairs, for example with cross-entropy. Introduced in: Lesson 10
behaviour policy: The policy that generated the data, written $\pi_b$ . It must give positive probability to every action that $\pi$ can take (coverage). Introduced in: Lesson 4
Bellman expectation equation: $V^\pi(s) = \sum_a \pi(a \mid s)\big[R(s, a) + \gamma \sum_{s'} P(s' \mid s, a) V^\pi(s')\big]$ : the value now is the expected reward plus $\gamma$ times the expected value of the next state. In matrix form, $V^\pi = (I - \gamma P_\pi)^{-1} r_\pi$ . Introduced in: Lesson 1
Bellman expectation operator: $T^\pi V = r_\pi + \gamma P_\pi V$ . Its unique fixed point is $V^\pi$ . Introduced in: Lesson 2
Bellman optimality equation: $V^*(s) = \max_a \big[R(s, a) + \gamma \sum_{s'} P(s' \mid s, a) V^*(s')\big]$ , and the same with $Q^*$ . The max sits inside, so the system is nonlinear. Introduced in: Lesson 1
Bellman optimality operator: $(TV)(s) = \max_a \big[R(s, a) + \gamma \sum_{s'} P(s' \mid s, a) V(s')\big]$ . Its unique fixed point is $V^*$ . Introduced in: Lesson 2
bootstrapping: Using a current estimate inside the target of an update, as in $r_t + \gamma V(s_{t+1})$ . Introduced in: Lesson 4
Bradley–Terry model: A preference model: $\Pr(y_w \succ y_l \mid x) = \sigma\big(r(x, y_w) - r(x, y_l)\big)$ . A reward model is trained by maximizing this likelihood. Introduced in: Lesson 9
certainty-equivalence estimate: The value computed as if the maximum-likelihood model fitted to the data were the true MDP. Batch TD(0) converges to it. Introduced in: Lesson 4
clip range: The number $\varepsilon$ (often 0.2) in PPO's clip. It is not the exploration rate of ε-greedy. Introduced in: Lesson 9
compounding error: Small one-step errors grow over many steps because each step starts from the previous wrong state. For a learned model with Lipschitz constant $L > 1$ the error bound grows like $L^H$ over $H$ steps; for behaviour cloning the extra cost can grow like $T^2 \varepsilon$ . Introduced in: Lesson 10
conservative Q-learning (CQL): Offline RL that adds to the TD loss the penalty $\alpha\big[\log \sum_a e^{Q(s, a)} - Q(s, a_{\mathcal D})\big]$ , which pushes down the values of actions not in the data. Introduced in: Lesson 6
contextual bandit: A one-step decision problem with a context. For an LLM the context is the prompt $x$ , the action is the whole response $y$ , and the reward is $r(x, y)$ . Introduced in: Lesson 9
contraction: A map $F$ with $\|F(x) - F(y)\| \le c\,\|x - y\|$ for some $c < 1$ . The Bellman operators are $\gamma$ -contractions in the $\infty$ -norm, so repeated application converges to one fixed point. Introduced in: Lesson 2
critic: The learned value function $V_\phi$ or $Q_\phi$ in an actor-critic method. Introduced in: Lesson 8
DAgger: Dataset aggregation: let the learner drive, ask the expert for the right action in the states the learner visits, add these labels to the data and retrain. Its extra cost grows only linearly in $T$ . Introduced in: Lesson 10
DDPG: Deep deterministic policy gradient: an off-policy actor-critic for continuous actions. A deterministic actor $a = \mu_\theta(s)$ moves the action uphill on a learned critic $Q_\phi$ . Introduced in: Lesson 8
deadly triad: Function approximation, bootstrapping and off-policy training together. With all three, values can diverge. Introduced in: Lesson 6
deep Q-network (DQN): Q-learning with a neural network $Q_\theta$ , a replay buffer and a target network. Target: $y = r + \gamma (1 - d) \max_{a'} Q_{\theta^-}(s', a')$ . Introduced in: Lesson 6
discount factor: The number $\gamma \in [0, 1)$ that weights a reward $k$ steps ahead by $\gamma^k$ . $\gamma = 1$ is allowed only in episodic tasks. Introduced in: Lesson 1
discounted state visitation: $d^\pi(s) = \sum_{t \ge 0} \gamma^t \Pr(s_t = s)$ . It is not normalized; $(1-\gamma) d^\pi$ is a distribution. Introduced in: Lesson 7
distribution shift: The states seen at test time differ from the training states. In imitation, the learner's own mistakes lead it to states the expert never visited. Introduced in: Lesson 10
double DQN: DQN with target $r + \gamma\, Q_{\theta^-}(s', \arg\max_{a'} Q_\theta(s', a'))$ : the online network selects the action and the target network evaluates it. Introduced in: Lesson 6
double Q-learning: Keep two estimates; one selects the best action and the other evaluates it. This removes the upward bias of the max. Introduced in: Lesson 5
DPO: Direct preference optimization: use the closed-form optimum of the KL-regularized objective to fit a policy directly to preference pairs with a logistic loss, with no sampling and no RL loop. Introduced in: Lesson 9
Dyna-Q: After each real step, update Q from the real transition, update the model, and then make $n$ extra Q-learning updates from transitions replayed from the model. Introduced in: Lesson 10
dynamic programming: Methods that compute value functions and optimal policies from a known model by repeated Bellman updates. Introduced in: Lesson 2
effective horizon: $1/(1-\gamma)$ , roughly how many steps ahead the discounted return looks. Introduced in: Lesson 1
eligibility trace: A per-state memory $z(s)$ that is increased on a visit and decays by $\gamma\lambda$ each step. TD( $\lambda$ ) updates every state by $\alpha\, \delta_t\, z(s)$ (the backward view). Introduced in: Lesson 4
entropy bonus: A term $c_e\, \mathcal H(\pi)$ added to the policy objective. It keeps the policy from becoming deterministic too early. Introduced in: Lesson 8
environment: Everything outside the agent. It takes the agent's action and returns the next state and a reward. Introduced in: Lesson 1
episode: One run of the agent from a start state until a terminal state. Tasks that end are episodic; tasks that never end are continuing. Introduced in: Lesson 1
expected reward: $R(s, a)$ , the mean of the reward that follows action $a$ in state $s$ . Introduced in: Lesson 1
expected SARSA: TD control with target $r_t + \gamma \sum_{a'} \pi(a' \mid s_{t+1}) Q(s_{t+1}, a')$ . It removes the noise of sampling $a_{t+1}$ . Introduced in: Lesson 5
exploration bonus: An extra term added to an action's estimated value that is large when the action has been tried few times. Introduced in: Lesson 3
exploration–exploitation trade-off: To earn reward the agent should exploit the best-looking action, but to find the best action it must explore others that look worse. Introduced in: Lesson 3
exponential recency-weighted average: The estimate produced by a constant step size, $Q \leftarrow Q + \alpha (r - Q)$ . Old samples get weights that shrink geometrically, so it can follow a nonstationary mean. Introduced in: Lesson 3
extrapolation error: In offline RL, wrong values for out-of-distribution actions that the data never tried. The max in the target picks them, and no data corrects them. Introduced in: Lesson 6
feature vector: A vector $x(s)$ that describes a state. A linear value function is $\hat v(s, w) = w^\top x(s)$ . Introduced in: Lesson 6
Fisher information matrix: $F = \mathbb E\big[\nabla_\theta \log \pi_\theta\, \nabla_\theta \log \pi_\theta^\top\big]$ . For a small step, $\mathrm{KL} \approx \tfrac12 \Delta\theta^\top F \Delta\theta$ . Introduced in: Lesson 9
fixed point: A point $x$ with $F(x) = x$ . Introduced in: Lesson 2
function approximation: Representing a value function or policy by a function with weights instead of a table, so learning in one state generalizes to others. Introduced in: Lesson 6
gap: $\Delta_a = \mu^* - \mu_a$ , how much worse arm $a$ is than the best arm on average. Introduced in: Lesson 3
generalized advantage estimation (GAE): $\hat A_t = \sum_{l \ge 0} (\gamma\lambda)^l \delta_{t+l}$ , a mix of $k$ -step advantage estimates. $\lambda = 0$ gives the one-step TD error (low variance, biased), $\lambda = 1$ the Monte Carlo advantage (unbiased, high variance). Introduced in: Lesson 8
generalized policy iteration: The general pattern of letting evaluation and improvement work together, each one possibly incomplete. Almost every RL method fits it. Introduced in: Lesson 2
GLIE: Greedy in the limit with infinite exploration: every state–action pair is tried infinitely often, and the policy becomes greedy in the limit (for example $\varepsilon_k = 1/k$ ). Introduced in: Lesson 5
gradient Monte Carlo: SGD on $\overline{VE}$ with the return $G_t$ as target. The target does not depend on $w$ , so it is a true stochastic gradient method. Introduced in: Lesson 6
greedy policy: The policy that picks, in each state, an action with the largest $Q(s, a)$ (or the largest one-step lookahead with $V$ ). Introduced in: Lesson 2
GRPO: Group relative policy optimization: a PPO-style clipped objective where the advantage of each of $K$ responses to one prompt is its reward minus the group mean, divided by the group standard deviation. It needs no critic. Introduced in: Lesson 9
imitation learning: Learning a policy from an expert's demonstrations instead of from rewards. Introduced in: Lesson 10
importance-sampling ratio: $\prod_k \pi(a_k \mid s_k) / \pi_b(a_k \mid s_k)$ , the factor that corrects returns from $\pi_b$ so their mean is the value under $\pi$ . Introduced in: Lesson 4
iterative policy evaluation: Compute $V^\pi$ by applying $T^\pi$ again and again, $V_{k+1} = T^\pi V_k$ , from any start. Introduced in: Lesson 2
KL penalty: The term $-\beta\, \mathrm{KL}(\pi_\theta \,\|\, \pi_{\text{ref}})$ in the RLHF objective. It keeps the policy near the reference model; $\beta$ is the KL coefficient. Introduced in: Lesson 9
log-derivative trick: $\nabla_\theta p_\theta(x) = p_\theta(x)\, \nabla_\theta \log p_\theta(x)$ . It turns the gradient of an expectation into an expectation that can be sampled. Introduced in: Lesson 7
Markov chain: A random sequence of states where the next state depends only on the current one. A fixed policy in an MDP gives a Markov chain with transition matrix $P_\pi$ . Introduced in: Lesson 1
Markov decision process (MDP): A model of sequential decisions given by states $\mathcal S$ , actions $\mathcal A$ , transition probabilities $P(s' \mid s, a)$ , expected rewards $R(s, a)$ and a discount factor $\gamma$ . It is finite when $\mathcal S$ and $\mathcal A$ are finite. Introduced in: Lesson 1
Markov property: The next state and reward depend only on the current state and action, not on the earlier history. Introduced in: Lesson 1
maximization bias: The max of noisy estimates is biased upward: $\mathbb E[\max_a \hat Q(a)] \ge \max_a \mathbb E[\hat Q(a)]$ . This makes Q-learning overestimate values. Introduced in: Lesson 5
maximum-entropy RL: RL with the objective $\mathbb E\big[\sum_t \gamma^t \big(r_t + \alpha\, \mathcal H(\pi(\cdot \mid s_t))\big)\big]$ . Its optimal policy is soft-greedy, $\pi(a \mid s) \propto \exp(Q(s, a)/\alpha)$ . Introduced in: Lesson 8
model: Anything the agent can use to predict the environment's response. A distribution model gives $P(s' \mid s, a)$ and $R(s, a)$ ; a sample model returns one possible $(r, s')$ , like a simulator. Introduced in: Lesson 10
model-based RL: RL that learns or uses a model of the environment and plans with it. Model-free RL learns values or policies directly from experience. Introduced in: Lesson 10
model-predictive control (MPC): At each step, plan an action sequence over a short horizon in the model, carry out only the first action, and plan again from the next state. Introduced in: Lesson 10
Monte Carlo (MC) prediction: Estimate $V^\pi(s)$ as the average of the returns observed after visits to $s$ . It needs complete episodes and no model. First-visit MC uses only the first visit per episode; every-visit MC uses all visits. Introduced in: Lesson 4
Monte Carlo tree search (MCTS): Build a search tree from the current state with repeated simulations of four phases: select, expand, evaluate, back up. Then act by the visit counts at the root. Introduced in: Lesson 10
multi-armed bandit: A problem with $K$ actions (arms) and no states. Each arm gives a random reward with an unknown mean $\mu_a$ , and the goal is a large total reward. Introduced in: Lesson 3
MuZero: AlphaZero with a learned model: a representation, a dynamics and a prediction function in a latent space, trained to predict rewards, values and search policies. Introduced in: Lesson 10
n-step return: $G_t^{(n)} = r_t + \gamma r_{t+1} + \dots + \gamma^{n-1} r_{t+n-1} + \gamma^n V(s_{t+n})$ : $n$ real rewards, then bootstrap. Introduced in: Lesson 4
natural gradient: The steepest ascent direction when step length is measured by KL divergence: $F^{-1} \nabla_\theta J$ . Introduced in: Lesson 9
off-policy: A method that learns about a target policy from data collected by a different behaviour policy. Introduced in: Lesson 5
offline RL: Learning a policy from a fixed dataset, with no new interaction with the environment. Introduced in: Lesson 6
on-policy: A method that learns about the same policy that it uses to collect data. Introduced in: Lesson 5
on-policy distribution: $\mu(s)$ , the fraction of time the policy spends in state $s$ . It weights the value error $\overline{VE}(w) = \sum_s \mu(s)[V^\pi(s) - \hat v(s, w)]^2$ . Introduced in: Lesson 6
optimal value function: $V^*(s) = \max_\pi V^\pi(s)$ and $Q^*(s, a) = \max_\pi Q^\pi(s, a)$ . A policy that is greedy with respect to $Q^*$ is optimal. Introduced in: Lesson 1
optimism in the face of uncertainty: Act as if each action is as good as is still plausible. Actions with few samples get a bonus, so they are tried. Introduced in: Lesson 3
ordinary importance sampling: Average of ratio-weighted returns. It is unbiased but can have very large variance. Introduced in: Lesson 4
parameterized policy: A policy $\pi_\theta(a \mid s)$ given by weights $\theta$ , for example a softmax over action logits or a Gaussian for continuous actions. Introduced in: Lesson 7
performance difference lemma: $J(\pi') - J(\pi) = \sum_s d^{\pi'}(s) \sum_a \pi'(a \mid s) A^\pi(s, a)$ : a new policy's gain equals its expected advantage under the old policy, along the new policy's states. Introduced in: Lesson 9
planning: Any computation that uses a model to improve a policy or a value function, without new real experience. Introduced in: Lesson 10
policy: The rule the agent uses to act. A stochastic policy $\pi(a \mid s)$ gives a probability for each action; a deterministic policy $\pi(s)$ gives one action. Introduced in: Lesson 1
policy gradient theorem: $\nabla_\theta J = \mathbb E_{\tau \sim \pi_\theta}\big[\sum_t \gamma^t \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, Q^{\pi}(s_t, a_t)\big]$ . The gradient needs no derivative of the dynamics. Introduced in: Lesson 7
policy improvement theorem: If $Q^\pi(s, \pi'(s)) \ge V^\pi(s)$ in every state, then $V^{\pi'} \ge V^\pi$ in every state. So the greedy policy with respect to $Q^\pi$ is at least as good as $\pi$ . Introduced in: Lesson 2
policy iteration: Alternate full policy evaluation and greedy improvement. On a finite MDP it reaches an optimal policy in finitely many iterations. Introduced in: Lesson 2
Polyak averaging: A smooth target-network update $\theta^- \leftarrow \tau\theta + (1 - \tau)\theta^-$ with small $\tau$ . Introduced in: Lesson 6
PPO: Proximal policy optimization: maximize $\hat{\mathbb E}_t\big[\min\big(\rho_t \hat A_t, \operatorname{clip}(\rho_t, 1 - \varepsilon, 1 + \varepsilon) \hat A_t\big)\big]$ for several epochs on one batch. The clip removes the incentive to move $\rho_t$ far from 1. Introduced in: Lesson 9
prediction: Estimating $V^\pi$ or $Q^\pi$ for a fixed policy (also called policy evaluation). Control means finding a good policy. Introduced in: Lesson 4
probability ratio: $\rho_t = \pi_\theta(a_t \mid s_t) / \pi_{\text{old}}(a_t \mid s_t)$ , how much more likely the new policy makes the sampled action. Introduced in: Lesson 9
PUCT: The selection rule of AlphaZero: $\arg\max_a Q(s, a) + c_{\text{puct}}\, P(s, a) \sqrt{N(s)} / (1 + N(s, a))$ , where $P$ is the policy network's prior. Introduced in: Lesson 10
Q-learning: Off-policy TD control with target $r_t + \gamma \max_{a'} Q(s_{t+1}, a')$ . It is a sampled version of value iteration and learns $Q^*$ while it explores. Introduced in: Lesson 5
regret: The reward lost compared with always playing the best arm. The pseudo-regret after $T$ rounds is $\sum_a \Delta_a\, \mathbb E[n_a(T)]$ . Introduced in: Lesson 3
REINFORCE: The Monte Carlo policy gradient method: sample episodes and step along $\sum_t \gamma^t \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, G_t$ . Introduced in: Lesson 7
reparameterization trick: Write a sampled action as a differentiable function of the parameters and fixed noise, $a = f_\theta(s, \varepsilon)$ , so gradients flow through the sample. Introduced in: Lesson 8
replay buffer: A memory of past transitions $(s, a, r, s', d)$ . Training on random minibatches from it breaks the correlation of consecutive samples and reuses data. Introduced in: Lesson 6
return: The discounted sum of future rewards, $G_t = \sum_{k \ge 0} \gamma^k r_{t+k}$ . It is also called the reward-to-go. It satisfies $G_t = r_t + \gamma G_{t+1}$ . Introduced in: Lesson 1
reward: A number $r_t$ that the environment gives after the action $a_t$ in state $s_t$ . Sutton and Barto write the same reward as $R_{t+1}$ . Introduced in: Lesson 1
reward hacking: The policy finds outputs that get a high score from a proxy reward but are not good. It is an example of Goodhart's law. Introduced in: Lesson 9
RL with verifiable rewards (RLVR): RL where a program checks the answer (for example math or code) and gives reward 1 if it is correct and 0 if not, instead of a learned reward model. Introduced in: Lesson 9
RLHF: Reinforcement learning from human feedback: fit a reward model to human preferences, then fine-tune the language model with RL (usually PPO) on that reward minus a KL penalty to a reference model. Introduced in: Lesson 9
RLOO: REINFORCE leave-one-out: sample $K$ responses to one prompt and use the mean reward of the other $K-1$ as the baseline of each. It needs no critic. Introduced in: Lesson 9
Robbins–Monro conditions: Step sizes with $\sum_k \alpha_k = \infty$ and $\sum_k \alpha_k^2 < \infty$ , such as $\alpha_k = 1/k$ . They are needed for convergence of stochastic approximation. Introduced in: Lesson 5
SARSA: On-policy TD control with target $r_t + \gamma Q(s_{t+1}, a_{t+1})$ , where $a_{t+1}$ is the action actually taken next. Introduced in: Lesson 5
score function: $\nabla_\theta \log \pi_\theta(a \mid s)$ . Its expectation under $\pi_\theta$ is zero. Introduced in: Lesson 7
semi-gradient TD: TD with function approximation that takes the gradient of the prediction only and treats the bootstrapped target as a constant. It is not the gradient of any fixed loss. Introduced in: Lesson 6
soft actor-critic (SAC): An off-policy maximum-entropy actor-critic for continuous actions, with two soft Q-critics, target networks and a reparameterized stochastic actor. Introduced in: Lesson 8
state: The information that describes the situation at time $t$ , written $s_t$ . In an MDP it is all the agent needs to predict what comes next. Introduced in: Lesson 1
state aggregation: Features that group states: $x(s)$ is the one-hot vector of the group of $s$ , so all states in a group share one value. Introduced in: Lesson 6
state-value function: $V^\pi(s) = \mathbb E_\pi[G_t \mid s_t = s]$ , the expected return from state $s$ when the agent follows $\pi$ . Introduced in: Lesson 1
surrogate objective: $L_\pi(\pi') = \sum_s d^\pi(s) \sum_a \pi'(a \mid s) A^\pi(s, a)$ , the performance difference with the old visitation $d^\pi$ . It is accurate only near $\pi$ . Introduced in: Lesson 9
sweep: One pass of updates over all states. Introduced in: Lesson 2
target network: A copy $\theta^-$ of the network weights that is used to compute targets and is updated only slowly (every $C$ steps or by Polyak averaging). Introduced in: Lesson 6
target policy: The policy whose values we want to learn, written $\pi$ . Introduced in: Lesson 4
TD error: $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ , with $V = 0$ after a terminal state. Introduced in: Lesson 4
TD fixed point: Where linear semi-gradient TD converges on-policy: the solution of the projected Bellman equation $\hat v = \Pi T^\pi \hat v$ . Introduced in: Lesson 6
TD(0): The one-step TD update $V(s_t) \leftarrow V(s_t) + \alpha\, \delta_t$ . Introduced in: Lesson 4
TD3: DDPG with clipped double Q-learning (the minimum of two critics in the target), delayed actor updates and target policy smoothing. Introduced in: Lesson 8
temperature: The weight $\alpha$ of the entropy term in SAC. Large $\alpha$ gives a more random policy; $\alpha \to 0$ gives the greedy policy. Introduced in: Lesson 8
temporal-difference (TD) learning: Update a value estimate toward a target built from one real reward and the current estimate of the next state, before the episode ends. Introduced in: Lesson 4
Thompson sampling: Keep a posterior over each arm's mean, draw one sample from each posterior, and pull the arm with the largest sample. For Bernoulli arms the posteriors are Beta distributions. Introduced in: Lesson 3
tile coding: Features from several overlapping grids (tilings) over a continuous state; $x(s)$ has one 1 per tiling. Introduced in: Lesson 6
trajectory: The sequence $\tau = (s_0, a_0, r_0, s_1, a_1, r_1, \dots)$ of one episode. Introduced in: Lesson 7
transition probability: $P(s' \mid s, a)$ , the probability that the next state is $s'$ after action $a$ in state $s$ . Introduced in: Lesson 1
TRPO: Trust region policy optimization: maximize the surrogate subject to an average KL divergence from the old policy of at most $\delta$ , using the Fisher matrix, conjugate gradient and a line search. Introduced in: Lesson 9
UCB1: Pull $\arg\max_a \big(\hat\mu_a + \sqrt{2 \ln t / n_a}\big)$ . The bonus comes from Hoeffding's inequality, and the regret is $O(\sum_a \ln T / \Delta_a)$ . Introduced in: Lesson 3
UCT: MCTS that uses UCB1 as the tree policy at every node. Introduced in: Lesson 10
value iteration: Apply the Bellman optimality operator, $V_{k+1} = T V_k$ , then act greedily. The error shrinks at least by $\gamma$ each sweep. Introduced in: Lesson 2
weighted importance sampling: Ratio-weighted returns divided by the sum of the ratios. It is biased (the bias goes to zero) but has much lower variance. Introduced in: Lesson 4
ε-greedy: Pick a random action with probability $\varepsilon$ and the greedy action otherwise. With a fixed $\varepsilon$ the regret grows linearly in $T$ . Introduced in: Lesson 3
ε-soft policy: A policy that gives every action probability at least $\varepsilon / |\mathcal A|$ in every state. Introduced in: Lesson 5
λ-return: $G_t^\lambda = (1-\lambda) \sum_{n \ge 1} \lambda^{n-1} G_t^{(n)}$ , a weighted average of all $n$ -step returns. $\lambda = 0$ gives TD(0) and $\lambda = 1$ gives Monte Carlo. Introduced in: Lesson 4