Scaling Laws & Chinchilla

The Predictability of LLM Scaling

Building Large Language Models requires spending millions of dollars on GPU compute clusters.

How do engineering teams know whether spending $10 Million on a 70B model will yield a specific performance jump before launching the training run?

Scaling Laws provide empirical power-law formulas that predict exact pretraining loss reduction as a function of Model Parameters ($N$), Dataset Tokens ($D$), and Total Compute ($C \approx 6 N D$).

  Pretraining Loss L
    │
  4 ┤  \
    │   \  Predictable Power-Law Slope!
  3 ┤    \
    │     \
  2 ┴──────┴──────────┴──────────► Total Compute Budget C (FLOPs)
        10^21       10^23      10^25

1. Kaplan Scaling Laws (OpenAI, 2020)

Jared Kaplan et al. (OpenAI) published the first systematic LLM scaling study:

$$L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}, \quad L(D) = \left(\frac{D_c}{D}\right)^{\alpha_D}$$

Key Kaplan Claim

Kaplan claimed that when compute budget $C$ increases by $10\times$:

• Model Parameters ($N$) should scale by $7.3\times$.
• Dataset Tokens ($D$) should scale by only $1.37\times$.

This led to a wave of massive, under-trained LLMs (e.g. GPT-3 175B trained on only 300 Billion tokens).

2. Chinchilla Scaling Laws (DeepMind, 2022)

Jordan Hoffmann et al. (DeepMind) published "Training Compute-Optimal Large Language Models" (Chinchilla), proving that Kaplan's learning rate schedules were suboptimal.

The Chinchilla Finding

When compute budget $C$ increases:

• Model Parameters ($N$) and Dataset Tokens ($D$) should scale in equal 1:1 proportion!

$$\text{Chinchilla Optimal Ratio: } D \approx 20 \cdot N$$

  CHINCHILLA BENCHMARK COMPARISON:
  - Gopher (DeepMind): 280B Parameters trained on 300B Tokens (Kaplan-style, Under-trained!)
  - Chinchilla (DeepMind): 70B Parameters trained on 1.4 Trillion Tokens (Chinchilla-Optimal!)
  RESULT: Chinchilla (70B) OUTPERFORMED Gopher (280B) while using 4x lower VRAM memory!

3. Beyond Chinchilla: Inference-Aware Scaling

Chinchilla optimizes for Pretraining Compute Efficiency.

However, in real-world production, an LLM will be served to millions of users for years (Inference Cost).

  Inference Cost = Total User Requests * Model Parameter Size N

Models like LLaMA 3 (8B trained on 15 Trillion tokens) intentionally over-train smaller models past Chinchilla limits by a factor of 100x!

While over-training costs more pretraining compute, it yields a small, ultra-fast 8B model that runs at high throughput on single GPUs, saving millions of dollars in downstream serving costs.

Say this out loud

Scaling Laws predict pretraining loss reduction as a power law of compute, parameters, and tokens. Kaplan laws claimed parameters should scale faster than tokens. DeepMind Chinchilla corrected this, proving parameters and tokens should scale equally in 1:1 proportion (20 tokens per parameter). Modern models like LLaMA 3 intentionally over train past Chinchilla bounds to optimize downstream inference costs.

Followups to expect

1. What is the compute equation for a Transformer forward-backward pass? $C \approx 6 N D$ floating point operations (FLOPs), where $N$ is parameter count and $D$ is total training token count.
2. What is Test-Time Compute Scaling? Scaling inference compute (such as generating longer Chain-of-Thought reasoning steps or running Monte Carlo tree search) to improve problem-solving accuracy on hard math and coding tasks without increasing model parameter size $N$.