DPO vs PPO
Contrasting implicit preference optimization against traditional actor critic reinforcement learning.
The Alignment Paradigm Comparison
Post-SFT preference alignment ensures LLMs follow human values (Helpful, Honest, Harmless).
Two main algorithms dominate post-SFT alignment:
┌──────────────────────────┬──────────────────────────┐
│ 1. PPO RLHF (2017/2022) │ 2. DPO ALIGNMENT (2023) │
├──────────────────────────┼──────────────────────────┤
│ Actor-Critic RL loop. │ Direct supervised loss │
│ Requires Reward Model. │ on preference pairs. │
│ 4 Models in GPU VRAM. │ 2 Models in GPU VRAM. │
│ High VRAM, unstable RL. │ Low VRAM, stable SFT-like│
└──────────────────────────┴──────────────────────────┘
Detailed Architectural Breakdown
PPO Architecture (4 Models in VRAM)
To run PPO RLHF, your GPU cluster must hold four distinct neural networks:
1. Policy Model (Actor - π_θ): Active model generating text & receiving updates.
2. Critic Model (Value - V_ϕ): Estimates expected future cumulative reward.
3. Reference Model (Fixed - π_ref): Frozen copy of SFT model for KL penalty.
4. Reward Model (Fixed - R_ψ): Frozen scalar judge scoring generated outputs.
Generating outputs and updating 4 massive models in VRAM requires complex GPU parallel orchestration and fine-grained hyperparameter tuning (clipping ranges, learning rates, GAE lambda, KL beta).
DPO Architecture (2 Models in VRAM)
DPO proves that the optimal policy under Bradley-Terry preferences has an exact closed-form solution:
$$\mathcal{L}{\text{DPO}}(\theta; \pi{\text{ref}}) = -\mathbb{E}{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]$$
DPO requires only two models:
- Active Policy Model ($\pi_\theta$).
- Frozen Reference Model ($\pi_{\text{ref}}$).
Training runs as a standard supervised binary cross-entropy classification pass on offline preference datasets!
Head-to-Head Comparison Matrix
| Dimension | PPO RLHF | DPO Alignment |
|---|---|---|
| Underlying Math | Reinforcement Learning (Actor-Critic) | Implicit Reward Classification |
| Separate Reward Model? | Yes (Mandatory) | No (Implicit in Policy) |
| GPU Models in VRAM | 4 Models ($\pi_\theta, V_\phi, \pi_{\text{ref}}, R_\psi$) | 2 Models ($\pi_\theta, \pi_{\text{ref}}$) |
| Training Stability | Unstable (Sensitive to RL hyperparams) | Stable (Supervised Cross-Entropy) |
| Online Exploration | Yes (Generates new outputs on the fly) | No (Operates on offline static pairs) |
| VRAM Memory Requirement | Extremely High | 50% Lower VRAM |
When to Choose Which?
- Choose DPO: For 90 percent of open-source fine-tuning workflows (Zephyr, LLaMA 3 Instruct, Mistral). DPO is fast, stable, and fits on standard hardware.
- Choose PPO: When building frontier systems with dynamic interactive environments (e.g. RL for mathematical reasoning or code execution) where online sampling out-of-distribution yields better results than static offline preference datasets.
Say this out loud
PPO is an actor-critic RL framework requiring 4 models in GPU VRAM and complex reward model tuning. DPO reformulates policy optimization to eliminate the reward model and critic, optimizing policy parameters directly on offline preference pairs using binary cross entropy. DPO cuts VRAM by 50 percent and provides stable, supervised-style alignment.
Followups to expect
- What is Online DPO (Iterative DPO)? Generating fresh responses from the active policy $\pi_\theta$ at every iteration, scoring them with an external judge (like GPT-4), and running DPO on the updated pairs to combine DPO stability with online PPO exploration.
- What is RLAIF (RL from AI Feedback)? Using an AI model (Anthropic Claude or GPT-4) instead of human annotators to rank preference pairs for DPO or PPO training.
Check yourself
What primary architectural advantage makes DPO significantly easier to train than PPO in production?