Double Descent & Grokking

Breaking Classical Statistical Wisdom

For decades, introductory statistics taught the U-Shaped Bias-Variance Curve:

As model complexity increases, training error decreases, but test error eventually increases due to Overfitting.

  CLASSICAL BIAS-VARIANCE U-CURVE:
  Test Error
    │         \             / ◄── Classical Overfitting Spike!
    │          \           /
    │           \_________/  ◄── Optimal Capacity Point
  0 ┴─────────────────────────────────────────► Model Capacity / Parameters

However, modern deep neural networks (like 70B LLMs) have far more parameters than training samples, yet they achieve outstanding test generalization!

Mikhail Belkin et al. (2019) and Preetum Nakkiran et al. (2019) explained this paradox by discovering Deep Double Descent.

  DEEP DOUBLE DESCENT CURVE:
  Test Error
    │         \             /\
    │          \           /  \
    │           \_________/    \________________ ◄── Over-Parameterized Regime
  0 ┴───────────────────┼─────────────────────────► Model Capacity / Parameters
                    Interpolation
                      Threshold (N = D)

The 3 Regimes of Double Descent

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. UNDER-PARAMETERIZED   │ 2. INTERPOLATION SPIKE   │ 3. OVER-PARAMETERIZED    │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Classical U-Curve regime.│ Parameter count N equals │ Parameter count N >> D.  │
│ Increasing capacity      │ sample count D. Model is │ Multiple interpolating   │
│ reduces bias and test    │ forced to fit noise.     │ solutions exist. SGD     │
│ error toward minimum.    │ HUGE TEST ERROR SPIKE!   │ selects smooth low-norm! │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Under-Parameterized Regime ($N < D$): Standard classical behavior. Adding parameters improves generalization.
2. Interpolation Threshold ($N \approx D$): The exact point where model capacity is just large enough to fit $100%$ of training samples. The model is forced to fit noisy outlier points, causing a severe test error spike!
3. Over-Parameterized Regime ($N \gg D$): As parameter count grows far past $D$, an infinite number of interpolating solutions exist. Gradient descent (SGD) acts as an implicit regularizer, selecting the smoothest, lowest-norm solution, causing test error to decrease again!

Types of Double Descent

Double Descent manifests across three axes:

• Model-Size Double Descent: Test error decreases as model parameter count $N$ grows.
• Sample-Size Double Descent: Test error can temporarily increase as dataset size $D$ grows near $N \approx D$.
• Epoch-Wise Double Descent: Test error decreases, spikes midway through training, and decreases again as epoch count increases.

What is Grokking? (Power et al., 2022 / OpenAI)

Grokking is an extreme temporal form of epoch-wise double descent observed in algorithmic tasks (like modular arithmetic or symbolic logic):

  Accuracy (%)
   100 ┤              TRAINING ACCURACY = 100%
       │  ═══════════════════════════════════════════════════
       │                                          GROKKING SPIKE!
    50 ┤                                                 /
       │                                                /
     0 ┼───────────────────────────────────────────────/─────► Training Epochs
          0             1,000         10,000        50,000

1. The model achieves $100%$ Training Accuracy within 1,000 epochs, but $0%$ Validation Accuracy (pure memorization!).
2. Training continues for $50,000$ extra epochs with $0.0$ training loss.
3. Suddenly, validation accuracy spikes to $100%$ in a few steps!

The network transitions from memorizing training rows into discovering the underlying general mathematical algorithm long after training loss hit zero.

Say this out loud

Double Descent describes how test error decreases, spikes near the interpolation threshold where parameter count equals sample count, and then decreases again in over parameterized regimes. Past the interpolation threshold, implicit SGD regularization selects smooth low norm solutions. Grokking is an extreme form where models suddenly transition from memorization to perfect test generalization long after training loss hits zero.

Followups to expect

1. How does L2 Weight Decay impact Double Descent? Proper L2 regularization suppresses or completely flattens the interpolation error spike, allowing models to transition smoothly into the over-parameterized regime.
2. What is Implicit Bias in SGD? Stochastic Gradient Descent naturally favors solutions with smaller L2 norm of weights or minimal curvature, explaining why massive over-parameterized models generalize well without explicit regularization.