Gradient Clipping

A simple safety net that prevents exploding gradients from crashing deep learning models.

🟡 intermediate4 min readtraining

Gradient Clipping is a practical optimization technique that prevents exploding gradients during neural network training. If the length or magnitude of a gradient vector exceeds a maximum threshold value, it is rescaled back down to the threshold while preserving its direction. Gradient clipping is essential when training Recurrent Neural Networks, Transformer Language Models, and Deep Reinforcement Learning agents.

What is Gradient Clipping?

When training deep neural networks, sudden massive gradient spikes can occur.

These huge gradient steps cause model weights to swing wildly, sending loss to infinity or producing NaN (Not a Number) errors that crash training.

Gradient Clipping acts as a safety harness. If a gradient step is reasonably sized, nothing changes. If a gradient step becomes dangerously huge, clipping caps it to a safe maximum limit before updating weights.

Two Types of Gradient Clipping

Gradient Value Clipping: Caps every individual number in the gradient vector between a minimum and maximum range (such as minus 1.0 to plus 1.0).
Gradient Norm Clipping (Preferred): Measures the total length of the entire gradient vector. If the total length exceeds a maximum threshold (such as 1.0), the entire vector is scaled down proportionally.

Norm clipping is preferred because scaling the entire vector down preserves the exact direction of the gradient step.

  Unclipped Gradient Vector (Length = 8.5, Dangerously Large!)
  ────────────────────────────────────────────────────────►

  After Norm Clipping (Max Norm = 1.0, Direction Preserved!)
  ─────►

How Norm Clipping Works Mathematically

Suppose your total gradient vector length is 5.0, but your maximum allowed threshold is 1.0:

scaling_factor = max_norm / actual_norm = 1.0 / 5.0 = 0.2

new_gradient = original_gradient * 0.2

The new gradient points in the exact same direction as before, but its step length is now safely reduced to 1.0.

Code Example in PyTorch

Gradient clipping is placed right after backpropagation and right before the optimizer update step:

loss.backward()

# Clip total gradient norm to 1.0
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

optimizer.step()

Say this out loud

Gradient Clipping caps dangerously large gradient updates during training. Value clipping limits individual elements, while norm clipping rescales the total gradient vector length when it exceeds a threshold like 1.0. Norm clipping is essential for Recurrent Networks and Transformers because it stops exploding gradients and NaN loss crashes while preserving gradient direction.

Followups to expect

Does Gradient Clipping solve vanishing gradients? No. Gradient clipping only prevents exploding gradients. It does not help when gradients shrink to zero.
How do you pick max_norm threshold? Standard values range between 0.5 and 5.0. A common default for language models is 1.0.

Check yourself

Question 1 of 3

How does Gradient Norm Clipping modify a gradient vector g when its length ||g|| exceeds maximum threshold max_norm?

More in Deep Learning

See all →

Activation Functions4 min Dropout4 min Backpropagation5 min