Inter-Annotator Agreement

Why Human Annotations Can Be Noisy

When building labeled datasets, developers assume human hand-labeling provides perfect ground truth.

In reality, human annotators frequently disagree:

• Ambiguous Task Guidelines: Annotators interpret rules differently.
• Subjective Tasks: Deciding whether a social post is toxic or sarcastic involves personal subjective judgment.
• Annotator Fatigue: Careless human workers rush through labeling tasks.

If human annotators disagree on 30 percent of training samples, the ground truth dataset is noisy, setting an artificial ceiling on model accuracy.

Inter Annotator Agreement (IAA) measures labeling consistency across annotators.

Annotator A Labels ──┐
Annotator B Labels ──┼─► [ STATISTICAL AGREEMENT METRIC (KAPPA) ] ──► Consensus Quality Score
Annotator C Labels ──┘

Statistical Agreement Metrics

Simple percentage agreement (for example $85%$ match) is misleading because two annotators could agree purely by random guessing.

Statistical IAA metrics adjust for chance agreement probability:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

• $p_o$: Observed percentage agreement.
• $p_e$: Expected agreement by chance.

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. COHEN KAPPA           │ 2. FLEISS KAPPA          │ 3. KRIPPENDORFF ALPHA    │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Measures agreement       │ Extends Kappa to THREE OR│ Flexible metric for ANY  │
│ between EXACTLY TWO      │ MORE annotators across   │ number of annotators,    │
│ annotators.              │ categorical data.        │ missing data, or scales! │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Interpreting Kappa Scores ($\kappa$)

  κ < 0.00:  Poor Agreement (Worse than random chance!).
0.01 - 0.40:  Slight to Fair Agreement (Unacceptable for ML datasets).
0.41 - 0.60:  Moderate Agreement (Requires guideline revisions).
0.61 - 0.80:  Substantial Agreement (Good quality benchmark!).
0.81 - 1.00:  Almost Perfect Agreement (Ideal dataset quality!).

How to Fix Low Annotator Agreement

1. Clarify Guidelines: Provide explicit edge-case examples and negative definitions in annotator instruction manuals.
2. Conduct Qualification Tests: Screen annotators using benchmark test sets before allowing them to label production datasets.
3. Adjudication for Disagreements: Route samples with low agreement to a senior domain expert to make final tie-breaker decisions.

Say this out loud

Inter Annotator Agreement measures consistency among human data annotators. Metrics like Cohen Kappa for two raters and Fleiss Kappa for multiple raters quantify agreement beyond random chance. A Kappa score above 0.70 represents strong agreement, while low scores signal vague task instructions or noisy labels requiring guideline revisions.

Followups to expect

1. What is Majority Voting in multi-annotator pipelines? Assigning the final label based on the most frequent class selected by multiple independent annotators per sample.
2. What is Dawid-Skene model? A probabilistic model that estimates individual annotator error rates and true hidden class labels simultaneously using expectation maximization.