Pretraining → SFT → RLHF

The LLM Lifecycle Pipeline

Building a production Large Language Model (like LLaMA 3, ChatGPT, or Claude 3) is not a single training run.

It requires a Three-Stage Training Pipeline:

  TRILLIONS OF RAW WEB TOKENS
              │
              ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ STAGE 1: PRETRAINING (Self-Supervised Causal LM)             │ ──► Produces BASE MODEL
  │ Consumes 99% of total compute budget across 10,000 GPUs.   │     (Raw Autocomplete Generator!)
  └──────────────────────────┬──────────────────────────────────┘
                             │
                             ▼
  HIGH-QUALITY INSTRUCTION PAIRS (10,000 - 100,000 Samples)
                             │
                             ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ STAGE 2: SUPERVISED FINE-TUNING (SFT)                       │ ──► Produces SFT INSTRUCT MODEL
  │ Teaches the model how to act as a conversational assistant. │     (Follows Chat Formats!)
  └──────────────────────────┬──────────────────────────────────┘
                             │
                             ▼
  PREFERENCE PAIRS (Chosen vs Rejected Responses)
                             │
                             ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ STAGE 3: PREFERENCE ALIGNMENT (RLHF / DPO / KTO)            │ ──► Produces FINAL ALIGNED MODEL
  │ Aligns outputs for Helpfulness, Honesty, and Safety (HHH).  │     (Production Ready Assistant!)
  └─────────────────────────────────────────────────────────────┘

Stage 1: Self-Supervised Pretraining

• Data Input: Trillions of unlabelled text tokens scraped from web pages, books, Wikipedia, GitHub code, and scientific papers (e.g. 15 Trillion tokens for LLaMA 3).
• Objective: Causal Next-Token Prediction ($L = -\sum \log P(x_t \mid x_{<t})$).
• Output: Base Model (e.g. LLaMA-3-8B-Base).
• Behavior: Raw text completion engine. If you ask a base model "What is the capital of France?", it might complete with "What is the capital of Germany? What is the capital of Italy?" because it learned list formatting patterns!

Stage 2: Supervised Fine-Tuning (SFT)

• Data Input: 10,000 to 100,000 curated, high-quality Instruction Prompt-Response Pairs written by human experts or distilled from frontier models.
• Objective: Per-Token Loss Masked Cross Entropy on response tokens.
• Output: SFT Instruct Model.
• Behavior: Understands user commands, system prompts, and multi-turn chat roles. Responds directly to questions ("The capital of France is Paris.").

Stage 3: Preference Alignment (RLHF / DPO)

• Data Input: Pairwise Preference Datasets containing Prompt ($x$), Chosen Response ($y_w$), and Rejected Response ($y_l$).
• Objective: Direct Preference Optimization (DPO) or PPO Reinforcement Learning.
• Output: Final Aligned Assistant (e.g. LLaMA-3-8B-Instruct).
• Behavior: Aligned with human values: helpful, honest, concise, and safe against jailbreaks. Refuses malicious requests while avoiding over-refusal of safe technical queries.

Summary Comparison Matrix

Say this out loud

Building production LLMs follows a three stage pipeline: Pretraining acquires raw world knowledge by consuming trillions of web text tokens via next token prediction (99% of compute). Supervised Fine Tuning (SFT) fine tunes base models on instruction prompt response pairs to learn chat formatting. Preference Alignment (RLHF/DPO) aligns model outputs for helpfulness, honesty, and safety.

Followups to expect

1. What is the LIMA Hypothesis (Less Is More for Alignment)? Zhou et al. (2023) proved that 1,000 carefully curated, exceptionally high-quality SFT instruction pairs can match the instruction-following performance of 50,000 noisy samples, proving that pretraining holds all knowledge while SFT merely unlocks formatting.
2. What is Continuous Pretraining? Continuing Stage 1 pretraining on specialized domain corpora (e.g. 50 Billion medical or legal tokens) before running SFT and DPO.