Backprop Through Time

Unrolling Recurrent Networks in Time

Recurrent Neural Networks reuse the same weight matrices ($W_x, W_h, W_y$) at every time step $t$.

To calculate gradients using backpropagation, we unroll the recurrent loop across time steps into a standard feedforward computational graph:

  RECURRENT LOOP:                      UNROLLED GRAPH ACROSS TIME STEPS:

      ┌──────┐                         Input x_1 ──► [ Cell W_h ] ──► Loss L_1
      │      │ h_t                                         │
      ▼      │                                             ▼ h_1
  [ RNN Cell ] ──► Output y_t          Input x_2 ──► [ Cell W_h ] ──► Loss L_2
      ▲                                                    │
      │                                                    ▼ h_2
   Input x_t                           Input x_3 ──► [ Cell W_h ] ──► Loss L_3

The BPTT Mathematical Gradient Sum

Total Loss $L$ across a sequence of length $T$:

$$L = \sum_{t=1}^T L_t$$

Because weight matrix $W_h$ is shared across all time steps, its total gradient is the sum of gradients at each individual step:

$$\frac{\partial L}{\partial W_h} = \sum_{t=1}^T \frac{\partial L_t}{\partial W_h}$$

Using the chain rule backwards through hidden states $h_t, h_{t-1}, \dots, h_1$:

$$\frac{\partial L_t}{\partial W_h} = \sum_{k=1}^t \frac{\partial L_t}{\partial h_t} \cdot \left( \prod_{j=k+1}^t \frac{\partial h_j}{\partial h_{j-1}} \right) \cdot \frac{\partial h_k}{\partial W_h}$$

Notice the product term $\prod \frac{\partial h_j}{\partial h_{j-1}}$. This repeated matrix multiplication across time steps causes vanishing or exploding gradients.

Truncated BPTT (Managing VRAM Limits)

Unrolling a sequence of 10,000 tokens creates a 10,000 layer computational graph.

Caching 10,000 intermediate hidden activation tensors in VRAM causes immediate out of memory crashes.

Truncated BPTT limits backpropagation to a fixed window $k_1$ steps:

  Full Sequence (1,000 Tokens) ──► Split into Sub Chunks of 50 Tokens:
  - Chunk 1: Forward 50 steps ──► Backward 50 steps ──► Pass hidden state h_50 to Chunk 2!
  - Chunk 2: Forward 50 steps ──► Backward 50 steps ──► Pass hidden state h_100 to Chunk 3!

Truncated BPTT maintains continuous forward memory while capping backpropagation computational graph memory to a safe fixed size.

Say this out loud

Backpropagation Through Time unrolls recurrent networks across time steps into a computational graph. Because weight matrices are shared across time, total gradient dL/dW is the sum of gradients calculated at each individual time step. Truncated BPTT limits backward chain rule passes to fixed windows like 50 steps to prevent GPU out of memory crashes.

Followups to expect

1. How does Truncated BPTT impact long term dependency learning? Gradients cannot flow backward past the truncation boundary (like 50 steps), so the model cannot learn direct gradient dependencies longer than the truncation window.
2. Why do Transformers avoid BPTT entirely? Transformers do not pass hidden states sequentially through time. Self attention connects all sequence tokens directly in parallel, avoiding time unrolling and BPTT computational bottlenecks.