Deep Learning

Learning Rate Schedules & Warmup

How dynamically adjusting learning rates during training accelerates convergence and improves model accuracy.

🟡 intermediate4 min readoptimization
Learning Rate Schedules change the optimizer step size over the course of training. Using a fixed learning rate is sub optimal: high rates cause divergence or bouncing around minima, while low rates slow training. Common schedules include Linear Warmup (gradually raising rate at step one), Cosine Decay (smoothly reducing rate to near zero), and Step Decay.

Why Change Learning Rates During Training?

The learning rate controls how large a step your optimizer takes when updating weights.

  High Learning Rate:  Fast early progress, but bounces around or diverges near the minimum.
  Low Learning Rate:   Stable settling into minima, but takes forever to train from scratch.

The ideal strategy takes large steps early in training to make fast progress across flat landscapes, and smaller steps later in training to settle precisely into deep minima.

Common Learning Rate Schedules

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. LINEAR WARMUP         │ 2. COSINE ANNEALING      │ 3. STEP DECAY            │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Gradually increases rate │ Smoothly lowers rate to  │ Drops learning rate by   │
│ from 0 to peak over initial│ near zero following a    │ a factor like 0.1 every  │
│ warmup steps.            │ half cosine wave.        │ set number of epochs.    │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Linear Warmup

When training starts, weights are randomly initialized. Initial gradients can be chaotic and massive.

Taking large steps with chaotic gradients early can destabilize the model.

Linear Warmup ramps up the learning rate from 0 to its peak target rate over the first few thousand steps, allowing layer representations to stabilize smoothly.

2. Cosine Decay with Warmup

This is the standard schedule for modern LLMs, Transformers, and ConvNeXt models:

  Learning Rate
   Peak ┤      / \
        │     /   \  Cosine Decay Wave
        │    /     \
      0 ┴───/───────\────────────────► Training Steps
          Warmup   Final Step
  1. Warmup Phase: Linearly increase learning rate from 0 to peak over initial 1 to 5 percent of steps.
  2. Cosine Phase: Smoothly decrease learning rate following a half cosine curve until reaching near zero at the final step.

3. Reduce on Plateau

Monitors validation loss after each epoch. If validation loss stops improving for a specified number of epochs (patience), the learning rate is multiplied by a reduction factor like 0.1.

Great for classical ML and smaller vision models where total step count is unknown in advance.

Say this out loud

Learning rate schedules adjust step size during training. Linear Warmup gradually ramps up the rate from zero to prevent chaotic early gradients from destabilizing initial weights. Cosine Decay smoothly reduces the learning rate along a half cosine curve to near zero at the final step, letting the model settle into deep minima.

Followups to expect

  1. What is Cosine Annealing with Warm Restarts (SGDR)? Periodically resetting the learning rate back to peak value in repeating cycles, helping the optimizer escape local sub optimal minima.
  2. Why must total training step count be known for Cosine Decay? The cosine curve formula relies on total step count T_max to calculate the decay rate so it reaches zero exactly at the final step.

Check yourself

Question 1 of 3

Why is Linear Warmup applied during the first few thousand steps of training large transformer models?

More in Deep Learning

See all →
Activation Functions4 minDropout4 minBackpropagation5 min