Distilling a Large Model into a Small One

Compressing a 175B teacher model's reasoning capabilities into a fast, cheap 8B student model.

🔴 advanced5 min readefficiency

Model Distillation transfers knowledge from a large, expensive Teacher Model (e.g. GPT-4o) into a compact, lightweight Student Model (e.g. LLaMA-3-8B). In classic Logit Distillation (Hinton et al., 2015), the student is trained to match the teacher's soft probability output distribution by minimizing KL divergence. In modern LLM Sequence Distillation, the student is fine-tuned on high-quality synthetic text and reasoning chains generated by the teacher (Synthetic SFT / Rejection Sampling).

Two Forms of Model Distillation

  1. CLASSIC LOGIT DISTILLATION (Hinton et al., 2015)
  Teacher Model ──► Softmax Logits P_T (T=5.0) ──┐
                                                 ├──► Minimize KL Divergence D_KL(P_T || P_S)
  Student Model ──► Softmax Logits P_S (T=5.0) ──┘   (Transfers "Dark Knowledge")

  2. LLM SEQUENCE DISTILLATION (Synthetic Instruction Tuning)
  Frontier Teacher (GPT-4o) ──► Generates 100k Synthetic Reasoning Chains ──► SFT Fine-Tune 8B Student

Hinton Logit Distillation Loss

Total Loss combines hard target Cross-Entropy $\mathcal{L}{CE}$ and soft teacher KL Divergence $\mathcal{L}{KL}$:

$$\mathcal{L}{\text{distill}} = (1 - \alpha) \mathcal{L}{CE}(y, \sigma(z_S)) + \alpha T^2 D_{KL}\left( \sigma\left(\frac{z_T}{T}\right) \parallel \sigma\left(\frac{z_S}{T}\right) \right)$$

$z_T, z_S$: Raw logits from Teacher and Student.
$T$: Temperature parameter ($T = 2.0\text{--}5.0$) that softens output probabilities to reveal dark knowledge correlations.
$T^2$ multiplier balances gradient scales.

Distillation vs Quantization vs Pruning

Efficiency Technique	Mechanics	Accuracy Impact	Best Used For
Distillation	Trains a small model to mimic a large teacher	Preserves high task performance	Specialized task adaptation (8B replacement)
Quantization	Lowers weight precision (FP16 $\to$ INT4)	$< 1%$ drop (AWQ)	General inference VRAM reduction
Structured Pruning	Removes entire attention heads / FFN layers	Moderate drop	Shrinking layer depth of existing models

Say this out loud

"Model Distillation transfers knowledge from a large teacher to a small student. Classic Logit Distillation minimizes KL divergence between soft teacher and student logits at high temperature to transfer 'dark knowledge'. In modern GenAI, Sequence Distillation uses frontier models like GPT-4o to generate synthetic reasoning chains for fine-tuning 8B models, delivering 50x lower serving cost."

Follow-ups to expect

What is On-Policy Distillation (MiniLLM / GDKD)? Standard sequence distillation suffers from exposure bias when the student model drifts from teacher trajectories. On-policy distillation samples outputs directly from the student and evaluates student trajectory probabilities against the teacher model.
What is Speculative Decoding via Distillation? Fine-tuning a 1B draft model directly on target 70B model logit distributions to maximize draft token acceptance rates during speculative execution.

Check yourself

Question 1 of 3

Why does Hinton's classic Logit Distillation minimize KL divergence between teacher Softmax logits P_T and student Softmax logits P_S?

More in LLMs & GenAI

See all →

Pretraining → SFT → RLHF5 min Fine-Tune vs RAG vs Prompt: Choosing5 min Retrieval-Augmented Generation5 min