Hypothesis Testing & p-values
How to make statistical decisions under uncertainty without being tricked by random noise.
Hypothesis testing evaluates whether an observed effect is statistically significant or merely random chance. We set a Null Hypothesis H₀ (no effect) and Alternative Hypothesis H₁ (effect exists), compute a test statistic, and derive a p-value: P(Data | H₀). If p-value < significance level α (typically 0.05), we reject H₀. Key interview topics include Type I vs Type II errors, p-value misinterpretations, Z-test vs t-test, and effect size vs p-value.
The Hypothesis Testing Framework
Every hypothesis test follows 5 rigid steps:
- Formulate Hypotheses:
- Null Hypothesis
H₀: No difference / no treatment effect (μ_A = μ_B). - Alternative Hypothesis
H₁: Treatment effect exists (μ_A ≠ μ_Borμ_B > μ_A).
- Null Hypothesis
- Choose Significance Level (
α): Threshold for Type I error (commonlyα = 0.05). - Calculate Test Statistic: e.g.,
t = (X̄_B - X̄_A) / SE_diff. - Compute p-value:
P( Statistic ≥ Observed | H₀ is True ). - Decision Rule:
- If
p-value < α: RejectH₀(Statistically significant result). - If
p-value ≥ α: Fail to rejectH₀(Insufficient evidence).
- If
Decision Matrix & Errors
| Decision \ Truth | H₀ is True (No Effect) | H₁ is True (Real Effect) |
|---|---|---|
| Reject H₀ (Fire Alert) | Type I Error (α) — False Alarm | Correct Decision — Power (1 - β) |
| Fail to Reject H₀ | Correct Decision (1 - α) | Type II Error (β) — Missed Detection |
- Type I Error Rate (
α): False positive rate, usually set to 5%. - Type II Error Rate (
β): False negative rate. - Statistical Power (
1 - β): Probability of detecting a real effect when one exists (target: 80%+).
Z-test vs Student's t-test vs Chi-Square vs ANOVA
| Test | Data Type | Scenario |
|---|---|---|
| Z-test | Continuous / Rates | Large sample (N > 30), known population σ, or Bernoulli rates (CTR). |
| t-test | Continuous | Small sample (N < 30) with unknown σ (Two-sample t-test for A/B testing). |
| Chi-Square (χ²) | Categorical | Independence of 2 categorical features (e.g. Device Type vs Conversion). |
| ANOVA (F-test) | Continuous | Comparing means across 3+ groups simultaneously (prevents alpha inflation). |
Say this out loud
"Hypothesis testing determines whether an observed metric difference is statistically significant. The p-value is the probability of seeing data as extreme as ours assuming the null hypothesis H₀ is true. If p < 0.05, we reject H₀. In large-scale online systems with millions of samples, even tiny non-meaningful lifts become statistically significant, so we must pair p-values with practical effect size and confidence intervals."
Follow-ups to expect
- What is the p-value misinterpretation trap? p-value is NOT the probability that H₀ is true, nor is it P(Result is due to chance). It is P(Data | H₀).
- Why is Welch's t-test preferred over Student's t-test in A/B testing? Welch's t-test does not assume equal variances between Control and Treatment groups (
σ_A² ≠ σ_B²), making it safer for real-world product experiments. - How does sample size N impact p-values? Because
t = (X̄_B - X̄_A) / (s / √N), increasing N inflates the t-statistic linearly with √N, driving p-values down even for microscopic effect sizes.
Check yourself
Question 1 of 3
What is the exact definition of a p-value?