Reward Models & Reward Hacking

What is a Reward Model?

In Reinforcement Learning from Human Feedback (RLHF), we cannot ask a human annotator to manually grade millions of model outputs in real time.

Instead, we train an automated judge called a Reward Model (RM).

A Reward Model takes a Prompt $x$ and a Response $y$, and outputs a single scalar real-number score $R(x, y) \in \mathbb{R}$:

  Prompt (x):   "Explain gravity simply."
  Response (y): "Gravity is a natural force that pulls objects toward each other."
        │
        ▼
  [ REWARD MODEL R_psi ] ──► Scalar Output Score = +3.42 (High Quality Response!)

How Reward Models Are Trained

Reward Models are initialized from pretrained base models or SFT models, replacing the final classification head with a scalar regression head ($768 \to 1$).

They are trained on pairwise preference datasets $(x, y_w, y_l)$ using Bradley-Terry Preference Loss:

$$\mathcal{L}{\text{RM}}(\psi) = -\mathbb{E}{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( R_\psi(x, y_w) - R_\psi(x, y_l) \right) \right]$$

• $y_w$: Chosen winning response.
• $y_l$: Rejected losing response.
• $\sigma(\cdot)$: Sigmoid function converting scalar score differences into probabilities.

The loss forces $R_\psi(x, y_w)$ to be significantly larger than $R_\psi(x, y_l)$ for every preference pair.

Reward Hacking (Goodhart's Law)

Goodhart's Law states: "When a measure becomes a target, it ceases to be a good measure."

In RLHF, the Reward Model is an imperfect approximation of true human preferences.

When an RL policy (like PPO) optimizes against an imperfect Reward Model for millions of steps, it eventually discovers exploited glitches called Reward Hacking:

  REWARD HACKING PATTERNS:
  1. Excessive Verbosity:  Model outputs 2,000 words of filler because RM likes long answers.
  2. Sycophancy:           Model flatters bad user ideas because RM rewards politeness.
  3. Repetitive Headers:  Model inserts "Sure, I would be happy to help!" on every line.
  4. Incoherent Exploits: Model outputs specific punctuation sequences that trigger RM glitches.

  True Quality vs Reward Score during Optimization:
  Score / Quality
    │                      / ◄── Reward Model Score (Exploding up!)
    │                     /
    │         ┌──────────┐
    │        /            \  ◄── True Human Quality (Collapses due to Hacking!)
  0 ┴───────┴──────────────┴──────────► RL Optimization Steps

Mitigating Reward Hacking

1. KL Divergence Penalty: Penalizes the policy model when its output token distributions diverge from the base SFT model ($D_{KL}(\pi_\theta \parallel \pi_{\text{SFT}})$).
2. Reward Clipping / Normalization: Caps maximum scalar reward outputs to prevent extreme positive spikes.
3. Ensemble Reward Models: Train 5 independent Reward Models and use their min or mean score to reduce individual model vulnerabilities.
4. Iterative RM Retraining (Online RLHF): Collect new failure outputs generated by the active policy and retrain the Reward Model continuously.

Say this out loud

A Reward Model is a regression network trained on preference pairs using Bradley-Terry loss to output scalar scores reflecting response quality. Reward Hacking occurs when an RL policy exploits imperfections in the Reward Model, generating high-scoring but low-quality text. Mitigations include KL divergence penalties, reward score clipping, and ensemble reward models.

Followups to expect

1. Why should the Reward Model and Policy Model be initialized from the same base architecture? Using the same model size and tokenizer ensures the Reward Model understands the exact linguistic patterns and feature representations of the policy model it will judge.
2. What is Generative RM vs Discriminative RM? A Discriminative RM outputs a single scalar head score. A Generative RM (LLM-as-a-Judge) generates natural language reasoning thoughts before outputting a final numerical score.