Confidence Intervals

What is a Confidence Interval?

When you measure a metric on a sample of users, your sample mean is just an estimate. A Confidence Interval puts upper and lower bounds around that number to show how precise your estimate really is.

For a normal distribution, a 95% Confidence Interval is calculated as:

Sample Mean +/- (1.96 * Standard Error)

Where Standard Error = Standard Deviation / sqrt(N).

The Frequentist Meaning of 95%

A common mistake is saying "There is a 95% chance the true answer is inside this specific range."

In frequentist statistics, the true population number is a fixed un-moving reality. Either your interval contains it, or it does not.

The true meaning of 95% Confidence is: If you ran this experiment 100 times and built 100 intervals, 95 of those intervals would successfully capture the true population number.

Why Point Estimates Are Dangerous

Imagine two A/B tests reporting a 5% revenue increase:

1. Test A (Small Sample): Lift = 5%, 95% CI = [-2%, +12%]
2. Test B (Large Sample): Lift = 5%, 95% CI = [+4%, +6%]

Test A could easily be a loss (-2%) or a massive win (+12%). The range is too wide to be sure.

Test B gives a tight, confident range between 4% and 6%. You can safely ship Test B.

Three Factors That Shrink Interval Width

1. Larger Sample Size (N): Dividing by square root of N shrinks the margin of error.
2. Lower Variance: Less noisy data produces tighter bounds.
3. Lower Confidence Level: A 90% CI is narrower than a 99% CI because it requires less coverage guarantee.

Say this out loud

A Confidence Interval provides upper and lower bounds around your point estimate. A 95% interval means that if you repeated the experiment many times, 95% of those calculated ranges would contain the true population parameter. Always report confidence intervals alongside p-values because intervals show both the magnitude of business impact and the certainty of your result.

Follow-ups to expect

• How do you construct confidence intervals for non-normal metrics like medians or ratios? Use Non-Parametric Bootstrap Resampling: draw 1,000 samples with replacement from your dataset, calculate the metric on each draw, and take the 2.5th and 97.5th percentiles.
• What does it mean if two 95% confidence intervals overlap slightly? Overlapping intervals do NOT necessarily mean the difference is statistically insignificant. You must calculate the confidence interval of the difference itself.