Vanishing & Exploding Gradients

The Chain Rule Problem

Backpropagation calculates gradients by multiplying local layer derivatives together backward from the final loss step:

Gradient at Layer 1 = Layer5 Derivative * Layer4 Derivative * Layer3 Derivative * Layer2 Derivative * Layer1 Derivative

When you stack dozens of layers together, this repeated multiplication causes two major issues.

1. Vanishing Gradients

If the local derivatives at each layer are smaller than 1 (for example, 0.25):

0.25 * 0.25 * 0.25 * 0.25 * 0.25 = 0.00097

As gradients travel backward through many layers, they shrink exponentially until they reach nearly zero.

Result: Early layers in deep networks receive near zero weight updates. The front of the network stays stuck at random initialization, failing to learn basic features.

Legacy activations like Sigmoid and Tanh are notorious for this because their maximum derivatives are 0.25 and 1.0.

2. Exploding Gradients

If local derivatives or weight values are consistently greater than 1 (for example, 2.0):

2.0 * 2.0 * 2.0 * 2.0 * 2.0 * 2.0 * 2.0 * 2.0 = 256.0

As gradients travel backward, they grow exponentially large.

Result: Weight updates become massive, causing the loss to bounce wildly or crash into NaN (Not a Number) numerical overflow errors.

Summary Comparison

1. Vanishing Gradients: Gradients shrink to zero. Early layers stop learning. Caused by Sigmoid activations and deep un-shortcutted layers.
2. Exploding Gradients: Gradients grow to infinity. Loss bounces or turns into NaN. Caused by high weights and recurrent loops.

How Modern Deep Learning Solves Both Problems

1. Use ReLU Activations: Derivative of ReLU is 1.0 for positive inputs, avoiding gradient shrinkage.
2. Use Residual Skip Connections (ResNet): Adds identity shortcuts where derivative is 1.0, giving gradients a direct path back to early layers.
3. Use Batch or Layer Normalization: Keeps activations centered and scaled, preventing extreme value growth.
4. Use Gradient Clipping: Caps gradient vector lengths to a maximum threshold like 1.0, stopping exploding updates instantly.
5. Use Proper Weight Initialization: Xavier or He initialization sets starting weight scale appropriately based on layer dimensions.

Say this out loud

Vanishing gradients happen when repeated chain rule multiplications shrink gradients to near zero, stopping early layers from learning. Exploding gradients happen when repeated multiplications grow exponentially, causing unstable updates or NaN crashes. We fix vanishing gradients using ReLU, skip connections, and normalization. We fix exploding gradients using gradient clipping and proper weight initialization.

Followups to expect

1. Why are Recurrent Neural Networks (RNNs) extra vulnerable to vanishing gradients? RNNs reuse the exact same weight matrix repeatedly across time steps. Multiplying by the same matrix over 100 time steps exponentially amplifies small or large values.
2. How do LSTMs solve vanishing gradients in sequences? LSTMs use constant error carousel cell states with additive memory gates, allowing information and gradients to flow unchanged across hundreds of time steps.