KTO & ORPO

Post-DPO Alignment Innovations

While DPO simplified alignment by eliminating PPO reward models, it still carries two major operational requirements:

1. Paired Data Requirement: Requires explicit $(x, y_w, y_l)$ triplets (both a winning AND a losing response for the exact same prompt $x$).
2. Reference Model VRAM Overhead: Requires keeping a frozen Reference Model $\pi_{\text{ref}}$ in GPU VRAM alongside the active policy model $\pi_\theta$.

Next-generation methods KTO and ORPO break these remaining limits:

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ ALIGNMENT METHOD         │ DATA REQUIREMENT         │ VRAM MEMORY REQUIREMENT  │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ DPO                      │ Paired (Chosen + Reject) │ 2 Models (Policy + Ref)  │
│ KTO                      │ Unpaired (Thumbs Up/Down)│ 2 Models (Policy + Ref)  │
│ ORPO                     │ Paired (Chosen + Reject) │ 1 Model ONLY (Policy)!   │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Kahneman-Tversky Optimization (KTO - Ethayarajh et al., 2024)

KTO is inspired by Prospect Theory (Kahneman & Tversky, 1979 Nobel Prize), which proved that humans do not maximize expected utility—we perceive losses much more intensely than equivalent gains (Loss Aversion).

Key Data Advantage

KTO does NOT require paired responses $(y_w, y_l)$ for the same prompt.

It takes unpaired single responses labeled simply as:

• $(x, y)$ $\to$ Desirable (Thumbs Up / Good)
• $(x, y)$ $\to$ Undesirable (Thumbs Down / Bad)

  Real-World Production Advantage:
  Users upvote good customer support responses and downvote bad ones on DIFFERENT queries.
  KTO trains directly on real un-paired user feedback logs without forcing synthetic pair generation!

$$\mathcal{L}{\text{KTO}} = \mathbb{E}{x,y} \left[ w(y) \left( 1 - \sigma\left( \lambda_y \left( r_\theta(x,y) - z_{\text{ref}} \right) \right) \right) \right]$$

2. Odds Ratio Preference Optimization (ORPO - Hong et al., 2024)

ORPO merges Supervised Fine Tuning (SFT) and Preference Alignment into a single unified loss step!

Key VRAM Advantage

ORPO eliminates the separate Reference Model $\pi_{\text{ref}}$ entirely!

It modifies standard SFT Cross-Entropy loss by appending an Odds Ratio Penalty:

$$\mathcal{L}{\text{ORPO}} = \mathcal{L}{\text{SFT}} + \lambda \cdot \mathcal{L}_{\text{OR}}$$

$$\text{Odds Ratio} = \frac{\text{Odds}\theta(y_w \mid x)}{\text{Odds}\theta(y_l \mid x)} = \frac{\frac{\pi_\theta(y_w \mid x)}{1 - \pi_\theta(y_w \mid x)}}{\frac{\pi_\theta(y_l \mid x)}{1 - \pi_\theta(y_l \mid x)}}$$

$$\mathcal{L}{\text{OR}} = -\log \sigma \left( \log \frac{\text{Odds}\theta(y_w \mid x)}{\text{Odds}_\theta(y_l \mid x)} \right)$$

ORPO penalizes the model for assigning high odds to dispreferred responses, achieving DPO-level alignment with 50 percent lower GPU VRAM overhead in a single training run.

Say this out loud

KTO and ORPO simplify post-DPO alignment. KTO eliminates paired data requirements by optimizing directly on unpaired binary thumbs up or thumbs down feedback using Prospect Theory loss aversion. ORPO eliminates the separate reference model entirely, adding an odds ratio penalty to SFT loss to perform alignment in a single 1-model GPU training step.

Followups to expect

• What is SimPO (Simple Preference Optimization)? A reference-free variant of DPO that replaces the reference model log ratio with an explicit length-normalized reward margin $\frac{1}{|y|} \log \pi_\theta(y|x) - \gamma$.
• When should you choose KTO over DPO? Choose KTO when collecting real-world user feedback logs where thumbs up and thumbs down ratings exist on independent prompts without paired alternatives.