Skip to content
LearnMLByDoing
Free

3.5 Recurrent networks, LSTMs and backprop through time

36 minProintermediateHOMLDLD2L

A recurrent network applies one cell again and again along a sequence, so its gradient is a long product of the same Jacobian: it fades or blows up like a matrix power unless a gate opens an additive path.

01Sequences need a state

Text, audio, sensor readings and clickstreams arrive in order, and their length changes from one example to the next. A feed-forward network takes a fixed-size input, so it cannot read a review of 12 words and one of 400 with the same weights unless you pad, truncate or average away the order. A recurrent network instead reads one element at a time and carries a summary forward: the state.

The simplest state is a running total. Reading the numbers 1, 2, 3 and keeping their sum gives h=1,3,6h = 1, 3, 6: each step adds the new input to the old state, ht=ht−1+xth_t = h_{t-1} + x_t. Give the old state a weight ww and the total becomes leaky: with w=0.5w = 0.5, ht=0.5 ht−1+xth_t = 0.5\,h_{t-1} + x_t gives 1, 2.5, 4.251,\ 2.5,\ 4.25, and an input kk steps old counts 0.5k0.5^k times as much.

Loading figure
Step forward through h1,h2,h3h_1, h_2, h_3, then keep stepping: the derivative of h3h_3 with respect to an older state is a power of ww. The bottom slider unrolls further: 0.530≈9.3×10−100.5^{30} \approx 9.3\times10^{-10}.

The viewport shows the same idea with vectors. Each dark block is one copy of the cell, T=20\tok{T_len}{T} = 20 of them; the cyan bars under each copy are the input xt\tok{x_in}{x_t} (three numbers per step) and the emerald bars above are the state ht\tok{h_state}{h_t} (six numbers, drawn five times taller than their value). The green pulse runs left to right: a copy's bars appear only after the state has reached it.

Hands-On Machine Learning, Aurélien Géron, Ch. 15: Recurrent neurons and layers

Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 9.1 Working with sequences

02The recurrence, unrolled

A vanilla RNN replaces the scalar weight with matrices and squashes the result with tanh⁡\tanh. In the row-vector convention of PyTorch and D2L, one step is

ht=tanh⁡ ⁣(xt Wxh+ht−1 Whh+b),h0=0\tok{h_state}{h_t}=\tanh\!\big(\tok{x_in}{x_t}\,W_{xh}+\tok{h_state}{h_{t-1}}\,\tok{whh}{W_{hh}}+b\big),\qquad h_0=0
The same WxhW_{xh}, WhhW_{hh} and bb at every step. Unrolling draws one copy per step; it does not create new weights.
X_t (one step of a batch)[,]

A batch of B sequences is processed in lockstep. The full input is a (T, B, D) tensor, read one time slice at a time.

W_xh[,]18 values

Input-to-hidden weights.

W_hh[,]36 values

Hidden-to-hidden weights, square because the state feeds back into itself. The parameter count, D·H + H·H + H, does not depend on T.

H_t[,]

Hidden state after step t. Stacking all steps gives (T, B, H); a classifier usually reads only H_T, a tagger reads every H_t.

The scene's weights are chosen so the maths stays exact: Whh=ρ Q\tok{whh}{W_{hh}} = \rho\,Q with QQ orthogonal, so every eigenvalue of WhhW_{hh} has modulus ρ\tok{rho}{\rho}, the cockpit's spectral radius (0.9 now), and so does every singular value. The inputs are a slow wave of amplitude 0.05, small enough that the units stay in the near-linear part of tanh⁡\tanh: the grey gauges above the copies show the mean slope tanh⁡′(at)=1−ht2\tanh'(a_t) = 1 - h_t^2 per step, 0.996 on average here.

Raise the spectral radius to ρ=1.5\rho = 1.5 and watch the forward pass.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 10.1 Unfolding computational graphs; 10.2 Recurrent neural networks

Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 9.4 Recurrent neural networks

03Backprop through time: a product of Jacobians

Let the loss read only the last state, L=wout⋅hTL = w_{\text{out}}\cdot h_T with ∥wout∥=1\|w_{\text{out}}\| = 1, so ∂L/∂hT=wout\partial L/\partial h_T = w_{\text{out}}. Backpropagation through time (BPTT) is ordinary backprop on the unrolled graph: walk from step TT back to step 1, multiplying by the local Jacobian of each step. The rose ribbon is the result, its thickness at copy tt equal to ∥∂L/∂ht∥\|\tok{grad}{\partial L/\partial h_t}\|.

Why the gradient is a matrix power1/5
  1. 1
    ak=xkWxh+hk−1Whh+b,hk=tanh⁡(ak)  ⇒  ∂L∂hk−1=(∂L∂hk⊙(1−hk2))Whh⊤a_k = x_k W_{xh} + h_{k-1}W_{hh} + b,\quad h_k=\tanh(a_k)\;\Rightarrow\; \frac{\partial L}{\partial h_{k-1}}=\Big(\frac{\partial L}{\partial h_k}\odot(1-h_k^2)\Big)W_{hh}^{\top}

    One step back: multiply elementwise by tanh′, then by the transposed recurrent matrix. This is the line the challenge asks you to write.

The viewport is at ρ=0.9\rho = 0.9, T=20T = 20. The ribbon starts at ∥∂L/∂h20∥=1\|\partial L/\partial h_{20}\| = 1 on the right and thins to ∥∂L/∂h1∥=0.122\|\partial L/\partial h_1\| = 0.122 on the left, just under the bound 0.919=0.1350.9^{19} = 0.135: the HUD's factor per step is 0.895, ρ\rho times a mean tanh⁡′\tanh' a little below 1. This is lesson 1.1's picture: applying a matrix again and again scales each eigen-direction by its eigenvalue every time, so ∣λ∣<1|\lambda| < 1 fades and ∣λ∣>1|\lambda| > 1 grows. Hover ρ\tok{rho}{\rho} to draw the bound ρT−t\rho^{T-t} as a grey outline on the ribbon.

Keep ρ=0.9\rho = 0.9 and unroll 60 steps.

Back to T=20T = 20 with ρ=0.5\rho = 0.5.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 10.2.2 Computing the gradient in an RNN; 10.7 The challenge of long-term dependencies

Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 9.7 Backpropagation through time

The rest of this lesson is in Pro

  • 04Exploding gradients and clipping
  • 05LSTM and GRU: gates and an additive path
  • 06What the additive path buys: copy the first token
  • 07Sequence models today
  • Python challenge: Run, backpropagate and gate an RNN
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is backpropagation through time?
It is ordinary backpropagation applied to a recurrent network after unrolling it: the network is drawn as one copy of the cell per time step, all copies sharing the same weights, and the gradient is passed backward from the last step to the first. Each step back multiplies by that step's Jacobian, and the gradient of the shared weights is the sum of the contributions from every copy. Truncated BPTT stops the backward pass after a fixed number of steps to bound memory and compute.
Why do recurrent neural networks have vanishing gradients?
The gradient that reaches an early time step is a product of one Jacobian per step, and each Jacobian contains the same recurrent weight matrix times the activation's slope. If the largest singular value of the recurrent matrix is below 1, and the tanh slope can only make it smaller, the product shrinks geometrically with distance, like a matrix power. After a few dozen steps the early inputs contribute almost nothing to the weight update, so the network cannot learn long-range dependencies.
What is the difference between an LSTM and a GRU?
Both add gates that create an additive path through time so gradients survive longer. The LSTM keeps a separate cell state with input, forget and output gates, four weight blocks in all. The GRU merges the cell and hidden state and uses an update gate that interpolates between the old state and a candidate plus a reset gate, three weight blocks, so it is smaller and often trains as well. Neither removes the need for gradient clipping.

Sources

  • Hands-On Machine Learning, Aurélien Géron
  • Deep Learning, Goodfellow, Bengio & Courville
  • Dive into Deep Learning, Zhang, Lipton, Li & Smola