Statistical Power & Sample Size

What is Statistical Power?

Statistical Power is the likelihood that your test will spot a real difference when a real difference actually exists.

Mathematically, Power = 1 - beta, where beta is the Type II error rate (false negative rate).

Standard practice sets significance level alpha = 0.05 and Power = 0.80. This means you accept a 5% chance of a false positive and a 20% chance of a false negative.

The Four Connected Levers

Power calculations connect four key variables. Changing any one lever impacts the others:

1. Sample Size (N): More users in your test gives higher statistical power.
2. Effect Size (MDE): Larger product changes are easier to spot than tiny subtle shifts.
3. Significance Level (alpha): A stricter alpha threshold (like 0.01 instead of 0.05) requires more data for equal power.
4. Variance: Lower noise in your metric makes real signals stand out faster.

Why Underpowered Tests Damage Businesses

Running a test with low power (like 30% power) is dangerous:

1. Wasted Effort: You might build a fantastic new checkout flow that genuinely boosts sales by 2%.
2. False Negative: Because your test was underpowered, the p-value comes back at 0.14 (not significant).
3. Wrong Decision: Your team throws away a winning feature, concluding it did not work.

How to Increase Test Power

1. Increase Sample Size: Run the experiment longer or allocate a larger percentage of live traffic.
2. Reduce Variance: Use variance reduction techniques like CUPED (Using pre-experiment data to remove baseline noise).
3. Target Larger Effects: Test bolder product changes rather than tiny color tweaks.

Say this out loud

Statistical power is the chance that your test catches a real effect when one exists. Industry standard targets 80% power. Power depends on sample size, variance, significance level, and effect size. Running underpowered A/B tests is a major business risk because you end up throwing away winning features due to false negative results.

Follow-ups to expect

• How does CUPED increase test power? CUPED uses historical pre-experiment user behavior to predict and subtract baseline variance, shrinking metric noise so smaller effects become statistically significant faster.
• Why can't you just let an underpowered test run indefinitely until it becomes significant? Peeking at results continuously inflates your false positive rate (Type I error). You must pre-calculate required sample size upfront.