SGD, Momentum, Adam & AdamW

The Evolution of Optimizers

Optimizers decide how to adjust network weights after backpropagation computes loss gradients.

┌──────────────────────────┬──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. STOCHASTIC GD (SGD)   │ 2. SGD WITH MOMENTUM     │ 3. ADAM                  │ 4. ADAMW                 │
├──────────────────────────┼──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Takes direct steps along │ Adds velocity vector.    │ Adaptive rates per weight│ Decouples weight decay   │
│ negative gradient.       │ Smooths oscillations and │ tracking mean and        │ from Adam momentum for   │
│ Can bounce in ravines.   │ accelerates along slopes.│ variance of gradients.   │ proper regularization.   │
└──────────────────────────┴──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Stochastic Gradient Descent (SGD)

Takes steps proportional to the negative gradient:

w = w - learning_rate * gradient

If the loss surface forms a narrow steep ravine, standard SGD bounces wildly back and forth across the walls instead of moving down the floor.

2. SGD with Momentum

Acts like a heavy ball rolling down a hill. It builds up velocity along consistent directions:

velocity = beta * velocity + ( 1 - beta ) * gradient

w = w - learning_rate * velocity

Momentum smooths out noisy perpendicular bounces, accelerating movement down the main floor.

3. Adam (Adaptive Moment Estimation)

Adam combines Momentum with adaptive learning rates for every individual parameter.

It tracks two exponential moving averages:

1. First Moment (m_t): Exponential average of past gradients (like Momentum).
2. Second Moment (v_t): Exponential average of past squared gradients (uncentered variance).

Formula:

w = w - learning_rate * ( m_t_hat / ( sqrt( v_t_hat ) + epsilon ) )

Weights with large frequent gradients get smaller learning rates. Weights with rare sparse gradients get larger learning rates.

4. AdamW (Decoupled Weight Decay)

In standard SGD, L2 regularization is mathematically equivalent to Weight Decay.

In Adam, adding L2 regularization directly to gradients breaks this equivalence because Adam scales gradients by $1 / \sqrt{v_t}$. This weakens regularization for weights with large gradients.

AdamW fixes this by applying weight decay directly to the weights after computing the Adam update step:

w = w - learning_rate * Adam_Step - learning_rate * weight_decay * w

AdamW is the default optimizer for modern Transformers, LLMs, and ConvNeXt models.

Say this out loud

SGD takes steps along the negative gradient. Momentum adds velocity to smooth out noisy oscillations down steep ravines. Adam adapts learning rates per parameter using moving averages of first and second gradient moments. AdamW decouples weight decay regularization from gradient moments, applying weight decay directly to parameters, making it the standard optimizer for LLMs and transformers.

Followups to expect

1. What is RMSprop? An adaptive optimizer that scales learning rates using an exponential moving average of squared gradients, solving exploding step issues in Recurrent Networks.
2. Why is hyperparameter beta1 usually 0.9 and beta2 0.999 in Adam? Beta1 controls the momentum memory window across past steps. Beta2 controls the variance memory window over a longer history.