A/B Testing End to End
The gold standard for product decision-making at tech companies.
A/B testing evaluates product changes by splitting users randomly into Control (A) and Treatment (B) variants. End-to-end execution requires hypothesis definition, metric selection (Guardrail vs Primary), sample size calculation based on Minimum Detectable Effect (MDE) and Power (80%), randomized unit assignment, AA validation, and statistical testing. Key interview topics include sample ratio mismatch (SRM), peeking pitfall, novelty effects, and network spillovers.
The End-to-End A/B Testing Workflow
1. Hypothesis & Metrics ──► 2. Sample Size & Duration ──► 3. Randomization & AA ──► 4. Run & Monitor ──► 5. Decision
1. Formulate Hypothesis & Select Metrics
- Primary Metric: Key conversion goal (e.g., Checkout Rate).
- Secondary Metrics: Diagnostic signals (e.g., Cart Additions).
- Guardrail Metrics: Latency, App Crash Rate, Unsubscribe Rate. (Do not ship if guardrail degrades).
2. Sample Size Calculation & Duration
Sample size per group N depends on:
N ≈ 16 · σ² / ( MDE )²
- Variance
σ²: Metric baseline variance. - Minimum Detectable Effect (MDE): Smallest relative lift business cares to detect (e.g., 1% lift).
- Significance
α = 0.05(5% False Positive Rate). - Power
1 - β = 0.80(80% True Positive Rate).
Always run experiments in full 7-day weekly cycles (typically 2 weeks) to capture day-of-week seasonality.
3. Unit of Randomization & Assignment
- User-level Randomization: Hash
md5(user_id + exp_id) % 100for consistent bucket assignment. - SUTVA Assumption: Stable Unit Treatment Value Assumption — treatment of unit A does not affect unit B.
Common A/B Testing Pitfalls
| Pitfall | Problem | Solution |
|---|---|---|
| Sample Ratio Mismatch (SRM) | Assignment pipeline is biased (e.g., 52% Control / 48% Treatment) due to latency drops or bot filters. | Run Chi-Square test on raw counts. If p < 0.001, invalidate experiment immediately. |
| Peeking (Early Stopping) | Checking p-values daily inflates Type I error from 5% to 30%+. | Use fixed horizon testing or Sequential Testing (mSPRT / Always Valid p-values). |
| Novelty / Primacy Effect | Users react positively/negatively just because UI changed, then fade over time. | Run experiment for 2+ weeks; plot metric trajectory over time. |
| Interference / Network Effects | Uber drivers or social network users interact, violating SUTVA. | Cluster Randomization (graph/city-level) or Switchback Experiments (time-based toggling). |
Say this out loud
"An A/B test requires defining primary, secondary, and guardrail metrics upfront, calculating sample size based on baseline variance and Minimum Detectable Effect, and randomizing at user level while respecting SUTVA. Before trusting results, check for Sample Ratio Mismatch via Chi-Square test. Avoid peeking at p-values to prevent Type I error inflation, and run for full weekly cycles to account for seasonality."
Follow-ups to expect
- What is variance reduction using CUPED? Controlled-experiment Using Pre-Experiment Data (CUPED) subtracts pre-experiment baseline metric variance
Y_adj = Y - θ(X - E[X]), reducing variance by up to 50% and shrinking required sample size N. - How do Multi-Armed Bandits (MAB) differ from A/B tests? A/B tests allocate 50/50 traffic statically to learn ground truth. Bandits dynamically shift traffic toward the winning variant during the test, minimizing regret (lost conversions) at the cost of statistical rigor.
- What are Switchback experiments? Used in 2-sided marketplaces (Doordash, Lyft) where spatial units interact: toggle the algorithm globally across the entire city every 30 minutes to evaluate treatment vs control.
Check yourself
Question 1 of 3
What is Sample Ratio Mismatch (SRM) in an A/B test configured for a 50/50 traffic split?