Which Layers to Freeze

What Does Freezing Layers Mean?

In deep learning, Freezing a Layer means disabling weight parameter updates during backpropagation.

In PyTorch, freezing is accomplished by setting:

for param in model.backbone.parameters():
    param.requires_grad = False # Disables gradient calculations for these weights!

When requires_grad = False:

1. Autograd skips computing partial derivatives for those weights.
2. Optimizer steps make zero changes to frozen parameter values.
3. GPU VRAM memory for storing backward gradient tensors is saved!

Why Early Layers Are Ideal for Freezing

Pretrained deep networks build a natural hierarchy of representations:

  INPUT DATA ──► [ EARLY LAYERS ] ────────► [ MIDDLE LAYERS ] ──────► [ DEEP LAYERS ] ──► OUTPUT
                 Low-Level Universal       Domain Textures            Task-Specific Concepts
                 (Edges, Lines, Grammar)   (Shapes, Phrases)          (Faces, Sentiment)
                 ───────────────────────   ─────────────────          ──────────────────
                 ALWAYS FREEZE!            OPTIONAL FREEZE            UNFREEZE & TRAIN!

Early layers extract universal domain agnostic features (like horizontal lines or basic subwords) that apply across almost every vision or text problem.

Freezing early layers preserves these rich features while allowing deep layers to adapt to your specific target task.

Transfer Learning Decision Matrix

How many layers should you freeze? Use this standard matrix based on Target Dataset Size and Domain Similarity:

┌─────────────────────────────────────┬─────────────────────────────────────┐
│ 1. SMALL DATASET + SIMILAR DOMAIN   │ 2. SMALL DATASET + DIFFERENT DOMAIN │
├─────────────────────────────────────┼─────────────────────────────────────┤
│ Freeze ENTIRE backbone.             │ Freeze EARLY layers.                │
│ Train ONLY the final linear         │ Fine tune DEEP layers + Head.       │
│ classification head.                │ Prevents severe overfitting.        │
├─────────────────────────────────────┼─────────────────────────────────────┤
│ 3. LARGE DATASET + SIMILAR DOMAIN   │ 4. LARGE DATASET + DIFFERENT DOMAIN │
├─────────────────────────────────────┼─────────────────────────────────────┤
│ Fine tune ALL layers (or top half)  │ Fine tune ALL layers from scratch.  │
│ with a tiny learning rate.          │ Pretrained weights provide good     │
│ Maximum accuracy potential!         │ initial starting point.             │
└─────────────────────────────────────┴─────────────────────────────────────┘

Practical Tips for Fine Tuning

1. Use Differential Learning Rates: Use tiny learning rates ($10^{-5}$) for early unfrozen backbone layers and larger learning rates ($10^{-3}$) for the new task head.
2. Gradual Unfreezing (Howard & Ruder, 2018): Start by fine-tuning only the classification head for 1 epoch, then unfreeze the last layer group, repeating backward to avoid shocking pretrained weights (Catastrophic Forgetting).

Say this out loud

Freezing layers sets parameter requires_grad to False to disable gradient updates during backpropagation. Early layers extract universal low level features like edges and grammar roots that apply across domains, so freezing them prevents overfitting and saves GPU memory on small target datasets.

Followups to expect

1. How does Batch Normalization behave when early layers are frozen? If a layer is frozen (requires_grad = False), set model.eval() or keep BatchNorm running statistics frozen (model.bn.eval()), otherwise updating running mean and variance will corrupt frozen activations.
2. What is LoRA (Low Rank Adaptation)? Instead of unfreezing full weight matrices, LoRA freezes 100 percent of pretrained base parameters and inserts tiny low rank trainable adapter matrices alongside layers.