RLHF Explained

What is RLHF?

Supervised Fine Tuning (SFT) teaches an LLM how to format responses to instructions.

However, SFT cross-entropy loss treats all valid response words equally. It cannot easily teach subtle human preferences like tone, concise formatting, or safe refusal of dangerous requests.

Reinforcement Learning from Human Feedback (RLHF) uses human preference rankings to fine-tune the model policy using Reinforcement Learning.

  SFT Model ──► Generate Pairwise Responses ──► Human Preference Ranking (Chosen vs Rejected)
                                                      │
                                                      ▼
  Train Scalar REWARD MODEL R_psi(x, y) ◄─────────────┘
        │
        ▼
  PPO REINFORCEMENT LEARNING LOOP:
  Actor Model pi_theta generates response y ──► Reward Model R_psi scores y ──► PPO Update pi_theta!
  (Constrained by KL Penalty relative to SFT Model!)

The 2-Phase RLHF Workflow

Phase 1: Train a Reward Model $R_\psi(x, y)$

1. Collect a dataset of prompts $x$ and multiple candidate outputs $(y_1, y_2)$.
2. Human annotators select the preferred response $y_w$ over $y_l$.
3. Train a scalar Reward Model $R_\psi(x, y)$ using the Bradley-Terry preference loss:

$$\mathcal{L}R(\psi) = -\mathbb{E}{(x, y_w, y_l)} \left[ \log \sigma \left( R_\psi(x, y_w) - R_\psi(x, y_l) \right) \right]$$

The Reward Model learns to output a single real-number scalar score (e.g. $+2.5$ for good, $-1.8$ for bad) for any prompt response pair.

Phase 2: PPO Policy Optimization

Fine-tune the active LLM policy $\pi_\theta$ using Proximal Policy Optimization (PPO).

Total Reward Function optimized during PPO:

$$\text{Reward}{\text{PPO}}(x, y) = R\psi(x, y) - \beta D_{KL}\left( \pi_\theta(y \mid x) \parallel \pi_{\text{SFT}}(y \mid x) \right)$$

• $R_\psi(x, y)$: Scalar score from the trained Reward Model.
• $D_{KL}(\cdot)$: KL Divergence Penalty measuring how far the active policy $\pi_\theta$ has drifted from the initial frozen $\pi_{\text{SFT}}$ model.
• $\beta$: Hyperparameter controlling penalty strength.

Why the KL Divergence Penalty is Mandatory

Without the KL Divergence penalty ($\beta = 0$):

The LLM policy will discover exploited loopholes in the Reward Model (Reward Hacking).

For example, the model might discover that generating repetitive strings of punctuation or specific nonsense words triggers high scores from the Reward Model. The model collapses into generating gibberish.

The KL penalty acts as an anchor, forcing the model to stay close to natural language capabilities learned during SFT.

Say this out loud

RLHF aligns LLMs with human preferences. Phase 1 trains a scalar Reward Model on human pairwise preference rankings. Phase 2 fine-tunes the LLM policy using PPO reinforcement learning to maximize reward scores. A KL divergence penalty prevents policy drift and reward hacking by anchoring updates near the initial SFT model.

Followups to expect

1. How many models must be loaded into GPU VRAM during PPO RLHF? 4 Models! Active Policy Model $\pi_\theta$, Value/Critic Model $V_\phi$, Frozen SFT Reference Model $\pi_{\text{ref}}$, and Frozen Reward Model $R_\psi$. This massive VRAM footprint motivated DPO.
2. What is PPO Clipped Surrogate Objective? Clips probability ratio $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$ within range $[1 - \epsilon, 1 + \epsilon]$ (typically $\epsilon = 0.2$), preventing destructively large policy updates.