Catastrophic Forgetting

What is Catastrophic Forgetting?

When humans learn a new skill (like learning to play Tennis), we do not suddenly forget how to speak English or ride a bicycle.

Neural networks suffer from Catastrophic Forgetting:

When a model pretrained on Task A is fine-tuned sequentially on Task B, backpropagation updates parameter weights to optimize Task B, overwriting knowledge required for Task A:

  PRETRAINED BASE MODEL:    Task A Accuracy = 95%  | Task B Accuracy = 20%
                                        │
                                        ▼ (Fine-Tune sequentially on Task B!)
  FINE-TUNED MODEL:         Task A Accuracy = 15%  | Task B Accuracy = 92%  <-- FORGOT TASK A!

Catastrophic Forgetting is a major barrier to continuous lifelong learning in artificial intelligence.

Root Causes of Forgetting

1. Shared Parameter Overwriting: Neural networks use distributed representations where a single weight parameter contributes to multiple tasks. Updating that parameter for Task B corrupts Task A.
2. Data Distribution Shift: Training exclusively on Task B data causes gradient descent to shift parameter weights out of the local minima bowl associated with Task A.

Production Defenses Against Forgetting

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. PEFT / LORA ADAPTERS  │ 2. ELASTIC WEIGHT (EWC)  │ 3. EXPERIENCE REPLAY     │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ FREEZE base model weights│ Penalizes updates to     │ Interleave a small fraction│
│ completely. Train tiny   │ parameters that were     │ of Task A dataset rows   │
│ isolated LoRA matrices.  │ critical for Task A      │ into Task B training     │
│ Zero base weight erosion!│ (Fisher Information Matrix) batches (Replay Buffer). │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Parameter-Efficient Fine-Tuning (LoRA / PEFT)

The most popular defense in modern LLM engineering!

By freezing $100%$ of base model parameters and training separate LoRA adapter matrices, the original pretraining knowledge base remains completely untouched.

2. Elastic Weight Consolidation (EWC - Kirkpatrick et al., 2017)

EWC measures the importance of each parameter $\theta_i$ to Task A using the Fisher Information Matrix $F_i$:

$$\mathcal{L}_{\text{EWC}}(\theta) = \mathcal{L}B(\theta) + \sum_i \frac{\lambda}{2} F_i \left( \theta_i - \theta{A, i}^* \right)^2$$

• If parameter $\theta_i$ was critical for Task A ($F_i$ is large), EWC applies a huge quadratic penalty if the optimizer tries to change $\theta_i$.
• If $\theta_i$ was irrelevant for Task A ($F_i \approx 0$), the optimizer is free to modify $\theta_i$ for Task B.

3. Experience Replay & Data Mixing

Interleave a small percentage (e.g. $10%$) of historical Task A dataset samples into Task B training mini-batches to keep gradient steps aligned with both objectives.

Say this out loud

Catastrophic Forgetting occurs when sequential fine tuning overwrites shared weight parameters learned during past tasks. Fixes include PEFT methods like LoRA that freeze base model parameters completely, Elastic Weight Consolidation (EWC) which penalizes changes to parameters critical to past tasks using Fisher Information, and Experience Replay data mixing.

Followups to expect

1. What is Continual Learning (Lifelong Learning)? The subfield of machine learning dedicated to building algorithms capable of learning continuous streams of tasks over time without catastrophic forgetting.
2. How does Model Merging (MergeKit) mitigate catastrophic forgetting? Merging a fine-tuned domain model with its original base model using spherical linear interpolation (SLERP) or TIES-Merging to retain general capabilities while gaining domain skill.