Recurrent networks, LSTMs and vanishing gradients

Before this lesson: lesson 03 (backpropagation, the adjoint $G_l$), lesson 04 ($s' \le 1/4$, $\tanh' \le 1$, saturation), and eigenvalues and powers of a matrix in linear-algebra lesson 05. Norms of matrices are in linear-algebra lesson 11. Time: about 70 minutes, plus the notebook.

---

Why this matters #

An RNN is the simplest network that carries a state through time. Its training problem is the clearest example of a general fact: depth multiplies Jacobians, and a product of many matrices grows or shrinks exponentially. The same fact explains careful initialization (lesson 04), residual connections (lesson 05) and why state-space models control the eigenvalues of their state matrix (lesson 16).

LSTMs were the standard model for translation and speech recognition until about 2017. Attention was first invented to help an RNN translate long sentences. Today, linear recurrences are back in Mamba, RWKV and other state-space models. Next-token prediction, tokens and embeddings, which this lesson introduces with RNNs, are exactly the setup of every large language model. In ML for science, a learned time stepper $u_{t+1} = F_\theta(u_t)$ (a weather emulator, for example) is an RNN, and training it on multi-step rollouts is BPTT. Interviews ask: "why do RNN gradients vanish?", "write the LSTM equations" and "why did transformers replace RNNs?"

1. Predict first #

2. The picture #

From text to vectors: tokens and embeddings #

A network works with numbers, so we first cut a text into tokens. A token is a small unit of text: a character, a word, or a piece of a word. The list of all possible tokens is the vocabulary $\mathcal V$; each token gets an integer token id in $\{0, 1, \dots, |\mathcal V| - 1\}$. Real language models use subword tokenizers such as byte-pair encoding; the LLMs track covers them. In this lesson we use characters, because they are easy to see.

Example. With the vocabulary $\{\texttt{h}, \texttt{e}, \texttt{l}, \texttt{o}\}$ and the ids $\texttt{h} = 0$, $\texttt{e} = 1$, $\texttt{l} = 2$, $\texttt{o} = 3$, the word "hello" becomes the sequence of ids $s = (0, 1, 2, 2, 3)$.

An id is a label, not a quantity: token 3 is not "three times" token 1. So we give each token a learned vector. An embedding matrix $E \in \mathbb{R}^{|\mathcal V| \times D}$ has one row per token, and the embedding of token $s$ is row $s$ of $E$. In matrix form it is a one-hot row vector times $E$:

The product picks one row (linear-algebra lesson 01: a one-hot vector selects a column or a row). Libraries never form the one-hot vector; they read the row directly. The rows are trained by backpropagation like any other weights.

Next-token prediction. A language model reads tokens $s_1, \dots, s_t$ and outputs a probability distribution $p_t = \operatorname{softmax}(z_t)$ over the vocabulary for the next token $s_{t+1}$. The training loss at step $t$ is the cross-entropy $\ell_t = -\log p_t[s_{t+1}]$: the negative log-probability that the model gave to the true next token. For "hello" the inputs are h e l l and the targets are the same text shifted by one: e l l o.

Teacher forcing. During training, the input at step $t + 1$ is the true token $s_{t+1}$, not the token the model would have sampled. So all targets are known in advance, and one pass over the text gives $T$ training signals. At generation time there is no true text: the model samples a token and feeds it back as the next input (this is called autoregressive generation).

The recurrent cell, unrolled #

An RNN keeps a hidden state $h_t \in \mathbb{R}^{1 \times d_h}$ (a row vector, as in PyTorch). At every step it mixes the old state with the new input, using the same weights at every step:

Draw one box per time step and connect $h_{t-1}$ to $h_t$: this is the unrolled network. It is a deep network with $T$ layers, one per step, and every layer uses the same $W_{hh}$, $W_{xh}$ and $b_h$. Training it is ordinary backpropagation on this graph. The only new point is that a shared weight collects one gradient term from each step, and these terms are added.

Interactive figure: loading…

Memory and gradient are the same number #

Start with the simplest case: one number, no nonlinearity, $h_t = w\,h_{t-1} + u\,x_t$. Unroll it:

The input of step $k$ reaches step $t$ with the weight $w^{t-k}$. The derivative is the same number: $\partial h_t / \partial h_k = w^{t-k}$. So memory (how much of the past is still in the state) and gradient flow (how much a loss at step $t$ can change step $k$) are one quantity:

• $|w| < 1$: old information and its gradient decay exponentially (vanishing);
• $|w| > 1$: both grow exponentially (exploding);
• $|w| = 1$: the boundary case; they neither decay nor grow.

The matrix case: eigenvalues and singular values #

With a state vector, $h_t = h_{t-1}W$ gives $h_t = h_0 W^t$. If $W$ is diagonalizable, $W = V\Lambda V^{-1}$ and $W^n = V\Lambda^n V^{-1}$: each eigen-direction is scaled by $\lambda_i^n$. In the long run the largest $|\lambda_i|$ wins (this is power iteration, linear-algebra lesson 05). This number is the spectral radius $\rho(W) = \max_i |\lambda_i|$.

The spectral radius decides the long run, but not the first steps. If the eigenvectors of $W$ are far from orthogonal (a non-normal matrix), $\|W^n\|$ can grow for many steps before it decays, even with $\rho(W) < 1$. Each single step is bounded by the largest singular value $\|W\|_2 \ge \rho(W)$.

With the tanh, every step also multiplies by $\tanh'(a_t) = 1 - h_t^2 \le 1$. A saturated unit (one with $|h_t|$ near 1) has $\tanh'$ near 0 and blocks the gradient.

Interactive figure: loading…

3. Definitions and main results #

Embeddings and the next-token loss #

Let $s_1, \dots, s_{T+1}$ be a sequence of token ids. The model reads $s_1, \dots, s_T$ (teacher forcing) and predicts $s_2, \dots, s_{T+1}$.

• Embedding: $x_t = \operatorname{onehot}(s_t)\,E$, the row $s_t$ of $E \in \mathbb{R}^{|\mathcal V|\times D}$. It has $|\mathcal V| \cdot D$ parameters.
• Logits and probabilities: $z_t = h_t W_{hy} + b_y \in \mathbb{R}^{1\times|\mathcal V|}$, $p_t = \operatorname{softmax}(z_t)$.
• Loss: $\ell_t = -\log p_t[s_{t+1}]$ and $L = \sum_{t=1}^{T} \ell_t$. Libraries usually report the mean $L/T$; the factor $1/T$ does not change anything below.
• Gradient at the logits: $\partial \ell_t / \partial z_t = p_t - \operatorname{onehot}(s_{t+1})$, as for any softmax with cross-entropy (lesson 03).
• Gradient of the embedding: $\partial L / \partial E = \sum_t \operatorname{onehot}(s_t)^\top\, \partial L/\partial x_t$. Only the rows of tokens that appear in the sequence receive a gradient, and a token that appears several times collects one term per appearance.

The vanilla RNN and BPTT #

Inputs $x_t \in \mathbb{R}^{1\times D}$, hidden states $h_t \in \mathbb{R}^{1\times d_h}$, $h_0$ given (usually $0$):

with $W_{hh} \in \mathbb{R}^{d_h\times d_h}$, $W_{xh} \in \mathbb{R}^{D\times d_h}$ and $W_{hy} \in \mathbb{R}^{d_h\times|\mathcal V|}$. Many papers write column vectors, $h_t = \tanh(W h_{t-1} + U x_t + b)$; then $W = W_{hh}^\top$. Norms and eigenvalues are the same for $W$ and $W^\top$, so all results below hold in both conventions.

BPTT. Let $g_t = \partial L / \partial h_t$ be the total gradient (a row vector, like $h_t$) and $\delta_t = \partial L / \partial a_t$. Go from $t = T$ down to $1$, with $\delta_{T+1} = 0$:

The first term is the loss at step $t$; the second is everything that happens after step $t$, passed back through $h_{t+1}$. The weights are used at every step, so their gradients sum over time:

Each term has the shape of a linear-layer gradient, input transposed times adjoint, exactly as $\partial\mathcal L/\partial W_l = A_{l-1}^\top G_l$ in lesson 03.

Theorem (BPTT is backpropagation with tied weights). Give each step its own copy $W^{(1)}, \dots, W^{(T)}$ of $W_{hh}$, and evaluate at $W^{(t)} = W_{hh}$ for all $t$. Then $\partial L / \partial W_{hh} = \sum_t \partial L / \partial W^{(t)}$.

Proof.

1. $L$ depends on $W_{hh}$ only through the $T$ copies: $L(W_{hh}) = \tilde L(W^{(1)}, \dots, W^{(T)})$ with every $W^{(t)} = W_{hh}$.
2. The map $W_{hh} \mapsto (W_{hh}, \dots, W_{hh})$ has the derivative "copy". By the multivariable chain rule, the derivative of $L$ is the sum of the partial derivatives with respect to each copy.
3. Each copy $W^{(t)}$ is used once, in $a_t$. Its gradient is the usual linear-layer gradient $h_{t-1}^\top \delta_t$. $\blacksquare$

Cost. BPTT costs about as much as the forward pass, $O(T\,d_h^2)$, but it must store all states $h_1, \dots, h_T$: memory $O(T\,d_h)$. Truncated BPTT cuts the graph every $k$ steps (the state is passed forward but treated as a constant). Memory drops to $O(k\,d_h)$, but the gradient no longer sees dependencies longer than $k$ steps.

Where the trouble is. Unroll the recursion for one loss term $\ell_t$ and an earlier state $h_k$ ($k < t$). With the one-step Jacobian

the chain rule gives

This is a product of $t - k$ matrices. Pascanu et al. call the terms with large $t - k$ the long-term contributions. The total gradient is a sum of short-term and long-term contributions.

Vanishing and exploding gradients #

Theorem (Pascanu, Mikolov, Bengio 2013). Let $|\sigma'| \le \gamma$ ($\gamma = 1$ for tanh, $\gamma = 1/4$ for the logistic sigmoid $s$). Write $W = W_{hh}$ and $\eta = \gamma\, \|W\|_2$. If $\eta < 1$, then for all $t > k$

so the long-term contributions vanish exponentially, whatever the inputs and states are.

Proof.

1. The 2-norm is submultiplicative: $\|J_i\|_2 \le \|\operatorname{diag}(\sigma'(a_i))\|_2\, \|W^\top\|_2$, and $\|W^\top\|_2 = \|W\|_2$.
2. The 2-norm of a diagonal matrix is its largest absolute entry, so $\|\operatorname{diag}(\sigma'(a_i))\|_2 \le \gamma$.
3. Hence $\|J_i\|_2 \le \eta < 1$ for every $i$.
4. Submultiplicativity again: $\|J_t \cdots J_{k+1}\|_2 \le \|J_t\|_2 \cdots \|J_{k+1}\|_2 \le \eta^{\,t-k} \to 0$. $\blacksquare$

Turning the proof around: the long-term contributions can explode only if $\gamma\,\|W\|_2 > 1$. This is a necessary condition for exploding, not a sufficient one: saturation can still make them vanish. In the linear case ($\sigma$ = identity, $\gamma = 1$) the exact condition is the spectral radius: $W^n \to 0$ if and only if $\rho(W) < 1$, and $\|W^n\|$ grows exponentially if $\rho(W) > 1$.

Gradient clipping #

Global-norm clipping (Pascanu et al., Algorithm 1): let $g$ be the gradient of all parameters, stacked into one vector, and $\tau > 0$ a threshold. Replace

It keeps the direction and caps the step length at $\tau$ times the learning rate. Why this helps: when gradients explode, the loss surface has a steep "wall". A normal gradient step at the wall throws the parameters far away, and training diverges. A clipped step stays short.

Clipping does not fix vanishing gradients. A small gradient is left unchanged. And when the gradient is large, the whole sum is rescaled by one number: the long-term contributions stay tiny next to the short-term ones. Clipping by value (each entry separately, to $[-\tau, \tau]$) is a different rule: it changes the direction of the gradient.

Interactive figure: loading…

The LSTM #

The long short-term memory (LSTM; Hochreiter and Schmidhuber 1997, with the forget gate of Gers et al. 2000) keeps two states: the hidden state $h_t$ and the cell state $c_t$. Three gates, numbers between 0 and 1 for each unit, control the cell. With $s$ the logistic sigmoid and $\odot$ the elementwise product:

• The forget gate $f_t$ decides what to keep of the old cell state.
• The input gate $i_t$ decides how much of the candidate $g_t$ (often written $\tilde c_t$) to write.
• The output gate $o_t$ decides what part of the cell to show as $h_t$.

There are four blocks of weights, each acting on $[x_t, h_{t-1}]$, so one layer has $4\,d_h(D + d_h + 1)$ parameters. PyTorch keeps two bias vectors ($b_{ih}$ and $b_{hh}$), so it reports $4\,d_h(D + d_h + 2)$. PyTorch stores the four blocks in the order $i, f, g, o$.

Why the cell state helps.

1. The cell update is additive: $c_t = f_t \odot c_{t-1} + (\text{new content})$. Hold the gates fixed. Then the direct path from $c_{t-1}$ to $c_t$ has the Jacobian $\partial c_t / \partial c_{t-1} = \operatorname{diag}(f_t)$.
2. Over many steps this path gives $\prod_i \operatorname{diag}(f_i)$: no weight matrix and no $\tanh'$ factor. (The other paths go through $h_{t-1}$ and the gates; they behave like a vanilla RNN.)
3. The network sets $f$ from the data. With $f$ near 1, the memory and its gradient last about $1/(1 - f)$ steps; with $f$ near 0 the cell erases its content on purpose. With $f \equiv 1$ and $i \equiv 0$, the cell copies its value forever: the "constant error carousel" of the original LSTM.
4. It is a residual connection in time, with a gate (compare lesson 05).

Forget-gate bias. A common choice is $b_f = 1$ at initialization (Jozefowicz, Zaremba, Sutskever 2015), so the cell starts by keeping its content. Note that $s(1) \approx 0.73$: a memory of only about 4 steps. Training must push $f$ toward 1 when long memory is needed. PyTorch initializes all biases uniformly at random, so you set $b_f$ by hand (the slice [d_h:2*d_h] of the bias).

The GRU (Cho et al. 2014) uses the same idea with fewer parts: one update gate $u_t$ mixes the old state and a candidate, $h_t = u_t \odot h_{t-1} + (1 - u_t) \odot \tilde h_t$. It has no separate cell state and 3 weight blocks instead of 4. (Papers and libraries differ on which term gets $u_t$.)

Interactive figure: loading…

Why attention and SSMs replaced RNNs for long sequences #

	RNN / LSTM	Self-attention	Linear RNN / SSM
Training over the sequence	sequential: $T$ dependent steps	parallel	parallel (FFT convolution or scan)
Path between tokens $t$ and $t - k$	$k$ nonlinear steps	1 step	$k$ linear steps, eigenvalues controlled
Cost per generated token	constant (fixed-size state)	grows with $T$ (KV cache)	constant (fixed-size state)

• Training. $h_t$ needs $h_{t-1}$, so an RNN cannot spread the time steps over the GPU. Attention (lesson 09) computes all positions at once with large matrix products.
• Path length. Between distant tokens, an RNN gradient passes $k$ Jacobians. Attention connects them directly, so there is no long product to vanish.
• The price of attention: $O(T^2 D)$ training cost and a cache of keys and values that grows with $T$ during generation.
• Linear RNNs and state-space models (S4, Mamba, LRU) keep the update linear in the state: $x_k = \bar A x_{k-1} + \bar B u_k$ (the S4 paper names the state $x$ and the input $u$). This unrolls to a convolution or an associative scan, so training is parallel; generation keeps a constant-size state. They also control the spectrum: $\bar A = \exp(\Delta A)$ with $\Delta > 0$ and $\operatorname{Re}\lambda(A) < 0$ puts every eigenvalue of $\bar A$ inside the unit circle. The price: a fixed-size state must compress the whole past, so exact recall is weaker than with attention. Lesson 16 develops this.

Answers to section 1. (1) $0.9^{100} \approx 2.7 \times 10^{-5}$: closer to $0.00003$. (2) No: clipping only shrinks large gradients. (3) $50{,}000 \times 512 = 25.6$ million parameters, and zero multiplications: the lookup reads one row.

4. Worked example #

(a) BPTT by hand. A scalar tanh RNN $h_t = \tanh(w\,h_{t-1} + u\,x_t)$ with $w = 0.5$, $u = 1$, $h_0 = 0$, inputs $x = (1, 0, 0)$ and loss $L = h_3$.

Forward.

• $h_1 = \tanh(0.5 \cdot 0 + 1) = \tanh(1) = 0.7616$;
• $h_2 = \tanh(0.5 \cdot 0.7616 + 0) = \tanh(0.3808) = 0.3634$;
• $h_3 = \tanh(0.5 \cdot 0.3634) = \tanh(0.1817) = 0.1797$.

Backward, with $g_t = \partial L / \partial h_t$ and $\delta_t = g_t (1 - h_t^2)$. There is no loss at steps 1 and 2, so $g_t = w\,\delta_{t+1}$ there:

• $g_3 = 1$, $\delta_3 = 1 \cdot (1 - 0.1797^2) = 0.9677$;
• $g_2 = 0.5 \cdot 0.9677 = 0.4838$, $\delta_2 = 0.4838 \cdot (1 - 0.3634^2) = 0.4200$;
• $g_1 = 0.5 \cdot 0.4200 = 0.2100$, $\delta_1 = 0.2100 \cdot (1 - 0.7616^2) = 0.0882$;
• $\partial L / \partial h_0 = w\,\delta_1 = 0.0441$.

Parameter gradients (sums over time, one term per use of the weight):

• $\partial L / \partial w = \delta_3 h_2 + \delta_2 h_1 + \delta_1 h_0 = 0.3517 + 0.3198 + 0 = 0.6715$;
• $\partial L / \partial u = \delta_1 x_1 + \delta_2 x_2 + \delta_3 x_3 = 0.0882 + 0 + 0 = 0.0882$.

Each backward step multiplies the gradient by $w \cdot \tanh'(a_t) \le 0.5$. The only input sits at step 1, so $\partial L/\partial u$ is small: it went through the whole product.

(b) Eigenvalues against singular values. Take $W = \begin{pmatrix} 0.5 & 1 \\ 0 & 0.5 \end{pmatrix}$ (a Jordan block). Its eigenvalues are $0.5, 0.5$, so $\rho(W) = 0.5$. But $\|W\|_2 = 1.207 > 1$, so the theorem (with $\gamma = 1$) says nothing. In the linear case,

because $n\,0.5^{n-1} \to 0$. The norms are $\|W\|_2 = 1.207$, $\|W^2\|_2 = 1.059$ and $\|W^{10}\|_2 = 0.020$: the norm stays above 1 for two steps, then $\rho = 0.5$ takes over. The norm condition is sufficient, not necessary.

5. In code #

PyTorch's autograd checks the worked example; then the Jacobian products, the LSTM equations and clipping. The notebook has these cells and the exercises.

import torch

# (a) the worked example with autograd
w = torch.tensor(0.5, requires_grad=True)
u = torch.tensor(1.0, requires_grad=True)
h = [torch.tensor(0.0, requires_grad=True)]          # h_0
for x_t in [1.0, 0.0, 0.0]:
    h.append(torch.tanh(w * h[-1] + u * x_t))
h[3].backward()                                      # L = h_3
print([round(v.item(), 4) for v in h])               # [0.0, 0.7616, 0.3634, 0.1797]
print(round(w.grad.item(), 4), round(u.grad.item(), 4), round(h[0].grad.item(), 4))  # 0.6715 0.0882 0.0441

Jacobian products for $T = 50$ steps and $d_h = 32$ (column convention, $h \leftarrow \tanh(Wh + Ux)$). The linear RNN follows $\rho(W)$; the tanh RNN obeys the bound, and saturation can make it vanish even when $\rho(W) > 1$:

from torch.autograd.functional import jacobian

torch.manual_seed(0)
H, D, T = 32, 8, 50
xs, U, W0 = torch.randn(T, D), torch.randn(H, D) / D**0.5, torch.randn(H, H)
norm2 = lambda M: torch.linalg.matrix_norm(M, ord=2).item()

def tanh_rnn_last(h0, W):
    h = h0
    for x in xs:
        h = torch.tanh(W @ h + U @ x)
    return h

# linear RNN: dh_T/dh_0 = W^T, ruled by rho(W)
for rho in [0.9, 1.0, 1.1]:
    W = W0 * rho / torch.linalg.eigvals(W0).abs().max()
    print("linear, rho", rho, f"{norm2(torch.linalg.matrix_power(W, T)):.0e}")
# linear, rho 0.9 8e-03 / linear, rho 1.0 2e+00 / linear, rho 1.1 2e+02

# tanh RNN with ||W||_2 = 0.9: the bound gives at most 0.9^50 = 5e-3
W = W0 * 0.9 / torch.linalg.matrix_norm(W0, ord=2)
print(f"{norm2(jacobian(lambda h0: tanh_rnn_last(h0, W), torch.zeros(H))):.0e}")   # 6e-25

# tanh RNN with rho(W) = 1.5: the linear part would explode, but saturation wins here
W = W0 * 1.5 / torch.linalg.eigvals(W0).abs().max()
print(f"{norm2(jacobian(lambda h0: tanh_rnn_last(h0, W), torch.zeros(H))):.0e}")   # 5e-05

The LSTM equations against nn.LSTMCell, the effect of the forget-gate bias, and clipping:

torch.manual_seed(0)
H, D, T = 32, 8, 50
xs = torch.randn(T, D)

# the LSTM by hand matches nn.LSTMCell (gate order in the weights: i, f, g, o)
cell = torch.nn.LSTMCell(D, H)
x, h, c = torch.randn(1, D), torch.randn(1, H), torch.randn(1, H)
z = x @ cell.weight_ih.T + cell.bias_ih + h @ cell.weight_hh.T + cell.bias_hh
i, f, g, o = z.chunk(4, dim=1)
c_new = torch.sigmoid(f) * c + torch.sigmoid(i) * torch.tanh(g)
h_new = torch.sigmoid(o) * torch.tanh(c_new)
h_ref, c_ref = cell(x, (h, c))
print(torch.allclose(h_new, h_ref, atol=1e-6), torch.allclose(c_new, c_ref, atol=1e-6))  # True True

# ||dc_T/dc_0|| after 50 steps for three forget-gate biases
def cell_last(c0):
    h, c = torch.zeros(1, H), c0[None]
    for x in xs:
        h, c = cell(x[None], (h, c))
    return c[0]

with torch.no_grad():
    cell.bias_hh.zero_()
for b_f in [0.0, 1.0, 3.0]:
    with torch.no_grad():
        cell.bias_ih[H:2 * H] = b_f                  # the forget-gate block of the bias
    J = jacobian(cell_last, torch.zeros(H))
    print(b_f, f"{torch.linalg.matrix_norm(J, ord=2).item():.0e}")   # 2e-10, 1e-03, 2e+01

# global-norm clipping keeps the direction and caps the length
p = torch.nn.Parameter(torch.zeros(3))
p.grad = torch.tensor([30.0, -40.0, 0.0])            # norm 50
total = torch.nn.utils.clip_grad_norm_([p], max_norm=1.0)
print(total.item(), [round(v, 3) for v in p.grad.tolist()])   # 50.0 [0.6, -0.8, 0.0]

With $b_f = 0, 1, 3$ the forget gate is about $s(b_f) = 0.5, 0.73, 0.95$; the gradient survives only when $f$ is close to 1. At $b_f = 3$ it even grows slightly: the paths through $h$ and the gates add to the direct cell path.

6. Where it shows up #

• Language and speech. LSTMs were the standard for machine translation and speech recognition in the mid-2010s; Google's 2016 neural translation system was a deep LSTM. Attention was first added to an RNN encoder–decoder (Bahdanau, Cho, Bengio 2015) and later replaced the recurrence.
• Language models. Tokens, embeddings, teacher forcing and the next-token cross-entropy of this lesson are the training setup of every GPT-style model; only the network in the middle changed.
• Linear recurrences return. Mamba, RWKV, LRU and other state-space models are designed around the problems of this lesson: linear updates, controlled eigenvalues, parallel training.
• ML for science. A learned time stepper $u_{t+1} = F_\theta(u_t)$ (weather and climate emulators) is an RNN without input. Training on multi-step rollouts backpropagates through it exactly like BPTT. If $\partial F_\theta / \partial u$ has eigenvalues with magnitude above 1, long rollouts blow up; this is the role of the amplification factor in a von Neumann stability analysis. The adjoint method for neural ODEs is BPTT in continuous time. Earthquake phase pickers such as EQTransformer first shorten the waveform with convolutions and only then apply LSTMs and attention, because a raw record has thousands of steps.
• Practice. Global-norm clipping at 1.0 is a common default in transformer training as well (nanoGPT uses grad_clip = 1.0).

7. Common mistakes #

• Using eigenvalues in the sufficient condition. The proof needs $\|W\|_2$. When the diagonal factors change from step to step, $\rho(W) < 1$ does not bound the product of a tanh RNN.
• "Clipping fixes vanishing gradients." It only shrinks large gradients.
• "LSTMs solve vanishing gradients." They give a path that does not vanish when $f \approx 1$. With $f < 1$ the gradient still decays like $\prod_i f_i$; this is intended forgetting.
• Forgetting the sum over time. $W_{hh}$ is used at every step, so $\partial L / \partial W_{hh}$ adds $T$ terms.
• Forgetting $s' \le 1/4$. A sigmoid RNN vanishes much faster than a tanh RNN.
• Expecting PyTorch to set the forget bias to 1. It does not; set bias_ih[d_h:2*d_h] by hand.
• Truncated BPTT with window $k$ and long dependencies. The forward state carries information further than $k$ steps, but the gradient cannot teach the model to use it.
• Shifting the targets the wrong way. The target at position $t$ is the token at $t + 1$. A model trained with inputs and targets aligned learns to copy its input.

8. Check yourself #

1. A sigmoid RNN has $\|W_{hh}\|_2 = 3$. Does Pascanu's sufficient condition for vanishing hold?
   Answer

   Yes. $\gamma = 1/4$, so $\gamma\|W_{hh}\|_2 = 0.75 < 1$: the long-term contributions shrink at least like $0.75^{\,t-k}$.
2. How many parameters does one LSTM layer with $D = 128$ inputs and $d_h = 256$ hidden units have (one bias per gate)? How many does PyTorch report?
   Answer

   $4 \cdot 256 \cdot (128 + 256 + 1) = 394{,}240$. PyTorch has two bias vectors: $4 \cdot 256 \cdot (128 + 256 + 2) = 395{,}264$.
3. A forget gate stays at $f = 0.99$. What fraction of the cell-path gradient survives 100 steps? Compare a vanilla RNN whose per-step factor is 0.9.
   Answer

   $0.99^{100} \approx 0.37$ against $0.9^{100} \approx 2.7 \times 10^{-5}$. The memory time is about $1/(1 - f) = 100$ steps.
4. Why does clipping not help with vanishing gradients?
   Answer

   It changes $g$ only when $\|g\| > \tau$, and then it rescales the whole sum by one number. The long-term contributions are tiny next to the short-term ones, and rescaling does not change their ratio.
5. Why is $\partial L / \partial W_{hh}$ a sum over time steps?
   Answer

   The same $W_{hh}$ is used at every step (tied weights). By the multivariable chain rule, the gradient adds the contribution of each use: $\sum_t h_{t-1}^\top \delta_t$.
6. For the text "hello" with character tokens, write the inputs and the targets used with teacher forcing.
   Answer

   Inputs `h e l l`, targets `e l l o`. At each position the target is the next character, and the input is always the true previous character, not the model's own prediction.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. Write the vanilla RNN update (row-vector convention).

   Show answer

2. What is backpropagation through time (BPTT)?

   Show answer

3. Give the one-step Jacobian $\partial h_t/\partial h_{t-1}$ of a tanh RNN.

   Show answer

4. A scalar linear RNN has $w = 0.9$. What is $\partial h_{50}/\partial h_0$?

   Show answer

5. What does the spectral radius $\rho(W)$ decide for an RNN?

   Show answer

6. State Pascanu's sufficient condition for vanishing gradients.

   Show answer

7. Write global-norm gradient clipping.

   Show answer

8. Write the LSTM cell-state update and say why it helps gradients.

   Show answer

9. Why are RNNs slow to train on long sequences?

   Show answer

10. How does an embedding layer compute the vector of token $s$?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Why do RNN gradients vanish or explode?" — The gradient over $t - k$ steps is a product of Jacobians $\operatorname{diag}(\sigma')W_{hh}^\top$, bounded by $(\gamma\|W_{hh}\|_2)^{t-k}$; in the linear case $\rho(W_{hh})$ decides.
• "Write the LSTM and say why it helps." — Gates $f, i, o$ and candidate $g$; additive cell update; the direct path has Jacobian $\operatorname{diag}(f_t)$.
• "LSTM or GRU?" — Similar quality on most tasks; the GRU has 3 weight blocks instead of 4 and no separate cell state, so it is a little cheaper.
• "How would you initialize $W_{hh}$?" — Orthogonal (all singular values 1), or the identity for ReLU RNNs (Le, Jaitly, Hinton 2015); set the LSTM forget bias to about 1.
• "What is teacher forcing, and what is its weakness?" — Training with the true previous tokens as inputs. At generation time the model sees its own outputs, which can contain errors it never saw in training (exposure bias).
• "Why did transformers replace RNNs, and what came back?" — Parallel training and a path of length 1 between any two tokens; linear RNNs and SSMs came back for constant-memory generation.

10. Practice #

The notebook for this lesson runs in Colab (NumPy and a small PyTorch model on the CPU). It has five exercises, each with a check cell and a folded solution:

1. Write the forward pass and BPTT of a vanilla RNN in NumPy and check the gradients against autograd.
2. Measure $\|\partial h_T/\partial h_0\|_2$ for several spectral radii and compare it with $\rho^T$ and with Pascanu's bound.
3. Write one LSTM step by hand and match nn.LSTMCell (gate order $i, f, g, o$).
4. Implement global-norm clipping and check it against torch.nn.utils.clip_grad_norm_.
5. Build inputs and targets for next-token prediction, write the loss, then train a tiny character-level RNN on a short text and sample from it.

11. Watch and read #

Lecture 10 | Recurrent Neural NetworksStanford CS231n (2017) · Justin Johnson · 73 min

RNNs as language models, BPTT and truncated BPTT, and the vanishing-gradient argument for LSTMs (from about minute 50).

Recurrent Neural Networks (RNNs), Clearly Explained!!!StatQuest with Josh Starmer · 17 min

A slow, visual introduction to unrolling and to why the same weight raised to a power makes gradients vanish or explode.

Long Short-Term Memory (LSTM), Clearly ExplainedStatQuest with Josh Starmer · 21 min

The gates of the LSTM with numbers, step by step.

The spelled-out intro to language modeling: building makemoreAndrej Karpathy · 118 min

Tokens, next-character prediction and the negative log-likelihood loss, built from scratch in code.

• Pascanu, Mikolov, Bengio, On the difficulty of training recurrent neural networks (2013): §2.1 (the conditions with $\gamma$), §2.3 (the "wall" picture), §3.2 (norm clipping, Algorithm 1).
• Dive into Deep Learning, §9.7 Backpropagation through time and §10.1 LSTM.
• Christopher Olah, Understanding LSTM Networks: the standard diagrams of the gates.
• Goodfellow, Bengio, Courville, Deep Learning, chapter 10: sequence modeling, BPTT, long-term dependencies and gated RNNs.
• Andrej Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks: character-level language models in practice.
• Jozefowicz, Zaremba, Sutskever, An Empirical Exploration of Recurrent Network Architectures (2015): the forget-gate bias of 1, and LSTM against GRU.