Transfer Learning & Fine-Tuning

The Decision Matrix

Transfer learning strategy depends on two axes: Target Dataset Size and Domain Similarity to Pretraining Source:

                       Domain Similarity to Source Data
                               High                        Low
                 ┌───────────────────────────┬───────────────────────────┐
      Small      │ Strategy 1:               │ Strategy 2:               │
                 │ Freeze backbone, train    │ Train linear head on      │
                 │ linear head only.         │ shallow layer features.   │
  Dataset Size   ├───────────────────────────┼───────────────────────────┤
                 │ Strategy 3:               │ Strategy 4:               │
      Large      │ Fine-tune all layers with │ Full fine-tuning from     │
                 │ low learning rate.        │ pretrained weights.       │
                 └───────────────────────────┴───────────────────────────┘

Three Fine-Tuning Frameworks

1. Linear Probing (Feature Extraction):
   • Freeze all pretrained backbone weights $W_0$.
   • Train only new classification head $W_{head}$.
   • Fast, low memory, zero catastrophic forgetting, but lower accuracy ceiling.
2. Full Fine-Tuning:
   • Unfreeze all layers. Train end-to-end with low learning rate (e.g. 1/10th pretraining rate).
   • High accuracy, but high GPU memory cost and risk of catastrophic forgetting.
3. Parameter-Efficient Fine-Tuning (PEFT / LoRA):
   • Freeze base model weights $W_0$. Inject small trainable rank matrices $\Delta W = B \cdot A$.
   • Achieves full fine-tuning performance with $< 1%$ trainable parameters.

Best Practices to Avoid Degradation

• Learning Rate Ratio: Use learning rates 10x to 100x smaller than initial pretraining rates.
• Layer-wise LR Decay (LLRD): Decay learning rate by factor $\eta < 1$ for each layer moving from output back to input: $\alpha_l = \alpha \cdot \eta^{(L - l)}$.
• Gradual Unfreezing: Train output head first for $K$ epochs, then unfreeze top layers, and finally unfreeze bottom layers.

Say this out loud

"Transfer learning reuses features pretrained on massive datasets for target tasks. When target data is small and domain-similar, we freeze the backbone and train a linear head to prevent overfitting. When target data is large, we fine-tune with small learning rates or LoRA adapters. We use layer-wise learning rate decay and gradual unfreezing to prevent catastrophic forgetting."

Follow-ups to expect

• What is In-Context Learning vs Fine-Tuning in LLMs? In-context learning passes task demonstrations in the prompt without updating any model weights ($W$ is fixed). Fine-tuning updates model weights via backpropagation ($W \to W + \Delta W$).
• Why is pretraining with Self-Supervised Learning better than Supervised ImageNet pretraining? Self-supervised pretraining (MAE, DINOv2, CLIP) learns richer, less task-biased visual representations than class-label supervised models, generalizing better to downstream medical, aerial, or robotics tasks.