Instruction Tuning & Data Curation

What is Instruction Tuning?

A pretrained Base Model is an autocomplete engine. It predicts raw text continuations.

Instruction Tuning (Supervised Fine-Tuning / SFT) fine-tunes base models on structured Instruction Prompt-Response Pairs:

  Base Model Behavior:
  Input:  "Translate this sentence to French: Hello world"
  Output: "Translate this sentence to Spanish: Hello world. Translate this sentence to German:"

  Instruction-Tuned Model Behavior:
  Input:  "Translate this sentence to French: Hello world"
  Output: "Bonjour le monde"

Instruction tuning teaches the model how to act as a helpful assistant that follows human commands.

The LIMA Hypothesis (Quality > Quantity)

Early SFT datasets relied on massive web scraping (e.g. 50,000 to 100,000 noisy crowdsourced pairs).

Meta's LIMA paper (Zhou et al., 2023 - "Less Is More for Alignment") challenged this consensus:

  LIMA EXPERIMENT:
  - Dataset A: 50,000 crowdsourced instruction pairs (Noisy quality).
  - Dataset B: 1,000 hand-curated, exceptionally high-quality instruction pairs (LIMA Dataset).
  RESULT: Model trained on 1,000 LIMA samples EQUALED or OUTPERFORMED the 50,000-sample model!

The LIMA Principle

Pretraining endows an LLM with 99 percent of its world knowledge and reasoning capacity.

Instruction Tuning merely teaches the model how to format and present that knowledge. A small, pristine dataset is all that is required.

The Data Curation Pipeline

To build a high-performing SFT dataset:

  Raw Seed Instructions ──► [ Deduplication & Similarity Filtering ] ──► [ Topic Clustering ] ──► [ LLM Quality Scoring ] ──► Final Pristine SFT Dataset!

1. Deduplication: Remove near-identical prompts using MinHash LSH or Sentence-BERT embedding similarity ($>0.85$ similarity).
2. Topic Diversity Balancing: Cluster instructions using BERTopic. Ensure equal representation across Code, Math, Writing, Reasoning, and Multi-Turn Chat.
3. LLM Quality Filtering (AlpacaEval / DeITA): Use GPT-4 as a Quality Scorer to rate response Complexity and Quality, discarding low-scoring samples.

Synthetic Data Generation (Self-Instruct)

Human data writing is slow and expensive.

Self-Instruct (Wang et al., 2022) uses a teacher model (e.g. GPT-4) to bootstrap new instruction datasets automatically:

  1. Start with 175 human seed instruction pairs.
  2. Prompt Teacher LLM: "Generate 5 new diverse instruction tasks similar to seed examples."
  3. Prompt Teacher LLM: "Generate input context and expert response for each new instruction."
  4. Filter out duplicate or low quality generated samples.
  5. Repeat loop to generate 50,000 synthetic instruction pairs (e.g. Stanford Alpaca)!

Say this out loud

Instruction Tuning fine-tunes base models on instruction prompt-response pairs to learn chat formatting and task execution. The LIMA hypothesis proved that dataset quality drastically outweighs dataset quantity, with 1,000 hand-curated instruction pairs matching 50,000 noisy samples. Data curation uses deduplication, topic balancing, and synthetic Self-Instruct generation.

Followups to expect

1. What is System Prompt Fine-Tuning? Including system prompts ("You are a helpful coding assistant") in SFT training pairs to train models to obey system persona instructions.
2. What is Data Packing in SFT? Concatenating multiple short instruction examples into a single 4096-token training sequence separated by EOS tokens, using Per-Token Loss Masking to maximize GPU training efficiency.