RNNs, LSTMs & GRUs

Sequential Processing with Recurrent Networks

Traditional feedforward networks process inputs independently without memory of past items.

Recurrent Neural Networks (RNNs) process sequence data step by step, passing a Hidden State Memory Vector ($h_t$) forward across time steps:

  Input x_1 ──► [ RNN Cell ] ──► Output y_1
                    │
                    ▼ Hidden State h_1
  Input x_2 ──► [ RNN Cell ] ──► Output y_2
                    │
                    ▼ Hidden State h_2
  Input x_3 ──► [ RNN Cell ] ──► Output y_3

Equation for standard RNN hidden state:

h_t = tanh( W_h * h_{t-1} + W_x * x_t + bias )

The Vanishing Memory Problem

In a standard RNN, processing a sequence of 100 words multiplies the hidden weight matrix $W_h$ 100 times in a row.

During backpropagation, multiplying by $W_h$ 100 times causes gradients to decay exponentially to zero.

The network forgets early words in a long sentence. By the time it reaches word 50, information from word 1 has vanished.

Long Short Term Memory (LSTM - Hochreiter & Schmidhuber, 1997)

LSTM solves vanishing gradients by adding a Constant Memory Cell State ($C_t$) that acts like a clean conveyor belt.

Information flows down the cell state with minimal modifications, regulated by three sigmoid gates:

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. FORGET GATE           │ 2. INPUT GATE            │ 3. OUTPUT GATE           │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Decides what old memory  │ Decides what new incoming│ Decides what part of the │
│ to discard from cell.    │ information to store.    │ cell memory to output as │
│ Sigmoid (0 = drop, 1=keep│ Sigmoid * Candidate Tanh │ hidden state h_t.        │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

                       LSTM CELL ARCHITECTURE
                        Cell State C_{t-1} ─────────────────────────────► C_t (Conveyor Belt!)
                                              ▲           ▲
                                              │           │
                                       [ Forget Gate ] [ Input Gate ]
                                              ▲           ▲
   Hidden State h_{t-1} ──┬───────────────────┴───────────┴─────────────► h_t
   Input Vector x_t     ──┘

Because additions to the cell state bypass repeated matrix multiplications, gradients flow backward across hundreds of time steps without vanishing.

Gated Recurrent Unit (GRU - Cho et al., 2014)

GRU simplifies the LSTM cell:

1. Combines Cell State and Hidden State into a single state vector $h_t$.
2. Reduces gates from 3 down to 2: Reset Gate (how to combine new input with past memory) and Update Gate (how much past memory to keep).

GRU trains faster than LSTM with fewer parameters while offering comparable performance.

Say this out loud

Standard RNNs pass hidden states across time steps but suffer from vanishing gradients on long sequences. LSTMs solve this using an additive Cell State conveyor belt controlled by Forget, Input, and Output gates. GRUs simplify LSTMs by combining cell and hidden states into one vector using Reset and Update gates, running faster with fewer parameters.

Followups to expect

1. What is a Bidirectional RNN? Running two independent RNNs on a sequence simultaneously: one processing left to right, and one processing right to left, combining both contexts at each time step.
2. Why have Transformers largely replaced LSTMs in modern AI? LSTMs process sequences sequentially step by step, which cannot be parallelized effectively on GPUs. Transformers process all tokens simultaneously in parallel.