Type I vs Type II Errors

The Error Matrix

                          Truth in Population
                     H₀ is True (No Effect)       H₁ is True (Real Effect)
                  ┌───────────────────────────┬───────────────────────────┐
  Reject H₀       │   Type I Error (α)        │    Correct Decision       │
  (Positive)      │   False Positive / Alarm  │    Power = 1 - β          │
                  ├───────────────────────────┼───────────────────────────┤
  Fail to Reject  │    Correct Decision       │   Type II Error (β)       │
  (Negative)      │    Confidence = 1 - α     │   False Negative / Miss   │
                  └───────────────────────────┴───────────────────────────┘

• Type I Error (α): Convicting an innocent person. Flagging healthy patient as sick.
• Type II Error (β): Acquitting a guilty criminal. Missing a cancerous tumor.
• Power (1 - β): Ability to detect a real difference (typically aimed at 80%).

The Trade-off & Levers

        Shift Decision Threshold Left (More Sensitive)
     ◄─────────────────────────────────────────────────►
  Type I Error (α) ↑   False Positives ↑  Recall ↑
  Type II Error (β) ↓  False Negatives ↓  Precision ↓

For a fixed sample size N, you cannot decrease α without increasing β.

To decrease both α and β simultaneously:

1. Increase Sample Size N: Shrinks standard error SE = σ / √N, narrowing distribution curves.
2. Increase Effect Size: Larger real differences are easier to separate from noise.
3. Reduce Measurement Noise: Improve logging precision or use variance reduction (CUPED).

Say this out loud

"Type I error alpha is a false positive—rejecting a true null hypothesis. Type II error beta is a false negative—missing a real effect. Statistical power is 1 - beta. For a fixed sample size, lowering alpha increases beta, so balancing them requires cost asymmetry analysis. The only way to lower both error rates simultaneously is to increase sample size N."

Follow-ups to expect

• How does alpha selection depend on application? In cancer screening, false negatives (Type II) kill patients, so set high sensitivity (accept higher Type I α). In automated account banning, false positives (Type I) ruin innocent users, so enforce low α (high precision).
• What is the relationship between Type I error and p-values? Significance level α is your pre-registered threshold for maximum acceptable Type I error rate (e.g. α = 0.05). The p-value is the probability of observing test data under H₀. Reject H₀ if p < α.