Multiple Testing & Bonferroni

The Multiple Testing Problem

If you set significance level alpha = 0.05, every single test carries a 5% chance of a false positive.

What happens when you evaluate 20 metrics on a single A/B test (or test 20 different UI variations)?

Family-Wise Error Rate (FWER):

FWER = 1 - (1 - alpha)^k = 1 - (0.95)^20 = 0.642 (64.2%)

With 20 metrics, you have a 64.2% chance of finding a false win purely due to random chance noise!

Correction Method 1: Bonferroni (Ultra-Strict)

The simplest correction divides your target significance alpha by the number of tests k:

alpha_new = alpha / k

For 10 metrics at total alpha = 0.05: alpha_new = 0.05 / 10 = 0.005

A metric must achieve p < 0.005 to be declared statistically significant.

• Pros: 100% guarantees Family-Wise Error Rate stays below 0.05.
• Cons: Extremely conservative! Drastically reduces statistical power, causing many real wins to be missed (high Type II errors).

Correction Method 2: Benjamini-Hochberg (False Discovery Rate)

Instead of guaranteeing zero false positives across all tests, the Benjamini-Hochberg (BH) procedure controls the False Discovery Rate (FDR):

FDR = Expected ( False Positives / Total Declared Positives )

The BH Procedure Algorithm

1. Sort all k p-values in ascending order: p_1 <= p_2 <= ... <= p_k.
2. Find the largest index i such that:

p_i <= (i / k) * Q

Where Q is your target False Discovery Rate (like 0.10).

3. Declare all tests from 1 to i as statistically significant.

FDR offers much higher statistical power than Bonferroni while keeping false discoveries under control.

When to Apply Corrections

1. Multiple Variants (A/B/C/D tests): Mandatory! Apply Bonferroni or Dunnett's test across treatment variants.
2. Multiple Metrics: If checking 30 secondary metrics, apply Benjamini-Hochberg FDR correction.
3. Subgroup Analysis: If slicing data across 15 country cohorts, apply corrections to prevent cherry-picking random cohort spikes.

Say this out loud

The Multiple Testing Problem occurs when running many tests simultaneously. Evaluating 20 metrics at alpha = 0.05 inflates your chance of a false positive to 64%. Bonferroni divides alpha by k, which is very strict and loses statistical power. Benjamini-Hochberg controls the False Discovery Rate instead, preserving power while keeping false positives under control.

Follow-ups to expect

• What is Holm-Bonferroni? A step-down procedure that is strictly more powerful than standard Bonferroni while still controlling Family-Wise Error Rate 100%.
• Do you need multiple testing correction if you specified one primary metric upfront? No! If you pre-registered a single Primary Metric (like 7-day retention) before launching the test, you evaluate that primary metric at standard alpha = 0.05 without correction.