Curating Preference Data

What is a Preference Dataset?

Supervised Fine Tuning (SFT) teaches an LLM how to format responses using single prompt response pairs $(x, y)$.

Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) require Pairwise Preference Datasets:

$$\mathcal{D}{\text{preference}} = { (x^{(i)}, y_w^{(i)}, y_l^{(i)}) }{i=1}^N$$

1. Prompt ($x$): The user instruction or query.
2. Chosen Response ($y_w$): The preferred, winning response.
3. Rejected Response ($y_l$): The non preferred, losing response.

  Prompt (x): "Explain quantum computing in one sentence."

  Chosen Response (y_w):  "Quantum computing uses quantum mechanics principles like superposition to process complex information faster than classical computers."

  Rejected Response (y_l): "It is a new fast computer that uses Qubits to run programs." (Lacks precision!)

How Preference Pairs Are Created

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. HUMAN ANNOTATION      │ 2. MODEL SELF PAIRING    │ 3. AI JUDGE (RLAIF)      │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Human crowd workers      │ Sample two outputs from  │ Generate responses from  │
│ write responses or rank  │ target model: one with   │ multiple models. Use     │
│ model candidate outputs. │ high temperature, one    │ GPT-4 judge with rubrics │
│ High cost, high quality. │ with low temperature.    │ to rank win/loss pairs.  │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Common Biases in Preference Data

Human annotators introduce systematic biases that can corrupt alignment models:

1. Verbosity Bias: Annotators consistently rate longer responses as higher quality, even when they contain unnecessary filler words.
2. Formatting Bias: Annotators favor responses with bold headers, bullet points, and markdown styling over plain text.
3. Sycophancy Bias: Annotators prefer responses that politely agree with their opinions, even when the user premise is factually wrong.

Mitigation Strategies

• Length Normalization: Filter or crop preference pairs so chosen and rejected responses have similar character lengths.
• Multi-Aspect Rubric Scoring: Grade responses on separate independent dimensions (Helpfulness, Factual Accuracy, Formatting, Safety) before computing final rankings.

Key Open Preference Datasets

1. HH-RLHF (Anthropic): 160k human preference pairs covering Helpful and Harmless dimensions.
2. UltraFeedback (OpenBMB): 64k prompts with responses generated from 4 distinct LLMs, annotated by GPT-4 across fine-grained rubrics.
3. LMSYS Chatbot Arena: Real-world crowd-sourced side-by-side battle rankings across frontier models.

Say this out loud

Preference Data Curation builds pairwise datasets containing prompt x, chosen response y_w, and rejected response y_l. These pairs train Reward Models and DPO alignment algorithms. High-quality curation requires filtering out human biases like verbosity and formatting bias, using multi-aspect rubrics or AI LLM judges like UltraFeedback to generate preferences at scale.

Followups to expect

• What is Inter-Rater Reliability (Fleiss Kappa / Cohen Kappa)? A statistical metric measuring how consistently different human annotators agree on preference rankings. Low agreement indicates ambiguous guidelines or noisy data.
• What are Hard Negatives in Preference Data? Paired responses where $y_w$ and $y_l$ are subtle and closely matched in quality, forcing the model to learn fine-grained reasoning rather than easy superficial differences.