A/B Test Pitfalls: Peeking & Novelty
Avoiding severe statistical traps like peeking, novelty effects, and sample ratio mismatches in A/B experiments.
Top 4 A/B Testing Pitfalls
A/B EXPERIMENTATION PITFALLS
┌──────────────────────────┬──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. CONTINUOUS PEEKING │ 2. SAMPLE RATIO MISMATCH │ 3. NOVELTY EFFECT │ 4. SPILLED OVERLAY │
├──────────────────────────┼──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Checking p-values daily │ Observed user split (e.g.│ Temporary metric spike │ Treatment users interact │
│ and stopping early │ 52/48) breaks expected │ driven by user curiosity │ with Control users, │
│ inflates false positives.│ 50/50 ratio. │ that fades over weeks. │ corrupting independence. │
└──────────────────────────┴──────────────────────────┴──────────────────────────┴──────────────────────────┘
1. Continuous Peeking (The P-Hacking Trap)
If you pledge to run a 14-day test, but check p-values every morning and stop the moment p < 0.05:
Your true False Positive Rate jumps from 5% to over 30%!
Random metric fluctuations naturally dip below 0.05 temporarily. If you stop the test at that exact moment, you are harvesting false positives.
Fix: Pre-commit to fixed sample size N, or use Sequential Testing frameworks (mSPRT / mAB) that adjust alpha boundaries dynamically for continuous monitoring.
2. Sample Ratio Mismatch (SRM)
You set up a 50/50 experiment split. After 7 days, you observe:
- Control: 100,000 users
- Treatment: 94,000 users
A Chi-Square test returns p < 0.0001 for the split ratio.
This is a Sample Ratio Mismatch. It means your assignment mechanism is broken:
- Treatment variant crashed low-end Android phones before logging events.
- Redirect scripts dropped slow network users in Treatment.
Rule: If SRM is detected, the entire experiment is invalid. Do not analyze metrics until assignment bugs are fixed!
3. Novelty & Primacy Effects
- Novelty Effect: Existing users click a bright new button out of curiosity. Metrics spike for 5 days, then decay back to normal.
- Primacy Effect: Existing users hate a redesigned navigation menu because muscle memory broke. Metrics drop for 10 days, then recover as users adapt.
Fix: Compare New Users (who never saw the old design) against Existing Power Users.
Say this out loud
A/B testing pitfalls ruin experiment decisions. Peeking at p-values daily without sequential corrections inflates false positives to over 30%. Sample Ratio Mismatch (SRM) indicates broken user assignment, invalidating all results. Novelty effects create temporary curiosity spikes. Always pre-calculate sample sizes, run SRM Chi-Square checks first, and segment cohorts to verify true long-term impact.
Follow-ups to expect
- What is an A/A Test? Running an experiment where both Group A and Group B receive the exact same control experience. Used to verify that your randomization engine works and that your false positive rate is truly 5%.
- How do network effects cause Interference (Network Spillover)? In social networks or ride-sharing apps (Uber/Lyft), treating Driver A affects control Driver B's supply/demand pricing. Solve using Cluster-Based Randomization (randomizing entire cities rather than individual users).
Check yourself
Why does checking p-values daily and stopping an A/B test early as soon as p < 0.05 (Continuous Peeking) ruin test validity?