Is Model A Really Better Than B?
Determining whether Model A's 0.5% offline metric gain over Model B is statistically significant or random sampling noise.
Why Point Estimates Are Misleading
Model A scores AUC = 0.885; Model B scores AUC = 0.880.
Is Model A genuinely superior, or did it get lucky on a favorable test set split?
95% Confidence Intervals of Metric Difference ΔAUC
Scenario 1: [==== (0.001 to 0.009) ====] ──► Statistically Significant (Interval excludes 0!)
Scenario 2: [====== (-0.003 to 0.013) ======] ──► NOT Statistically Significant (Interval crosses 0!)
If the 95% confidence interval of $\Delta = \text{Metric}_A - \text{Metric}_B$ crosses zero, Model A is NOT provably better than Model B.
Three Rigorous Model Comparison Tests
1. McNemar's Test (Paired Binary Predictions)
Fast, non-parametric test operating on a $2 \times 2$ contingency table of paired predictions on the same fixed test set:
Model B Correct Model B Incorrect
Model A Correct n_00 n_01 (A right, B wrong)
Model A Incorrect n_10 (B right, A wrong) n_11
McNemar Test Statistic (Chi-Square with 1 degree of freedom):
$$\chi^2 = \frac{( |n_{01} - n_{10}| - 1 )^2}{n_{01} + n_{10}}$$
If $\chi^2 > 3.841$ ($p < 0.05$), the disagreement rates are statistically significant!
2. 5x2-Fold Cross-Validation Paired t-Test (Dietterich, 1998)
Corrects for overlapping fold correlations:
- Run 2-fold cross-validation 5 times (total 10 folds).
- Compute variance strictly within each 2-fold replication to avoid cross-replication correlation bugs.
3. Bootstrap Resampling Confidence Intervals (Gold Standard)
No model re-training required!
- Given saved test predictions $\hat{y}_A$ and $\hat{y}_B$ for $N$ test samples.
- For $b = 1 \dots 1000$:
- Sample $N$ indices with replacement from test set.
- Compute $\Delta^{(b)} = \text{Metric}_A^{(b)} - \text{Metric}_B^{(b)}$.
- Sort 1,000 $\Delta^{(b)}$ values. 95% CI is $[\Delta_{(25)}, \Delta_{(975)}]$.
Say this out loud
"To prove Model A is statistically better than Model B, we construct confidence intervals around metric differences Δ = Metric_A - Metric_B. Standard paired t-tests over 10-fold CV fail due to training set overlap correlations. We use McNemar's test for paired classification outputs, or Bootstrap Resampling over saved test predictions to construct 95% confidence intervals around Δ metric gains."
Follow-ups to expect
- What is the Bonferroni Correction for multi-model comparison? If comparing 10 candidate models against a baseline, running 10 hypothesis tests at $\alpha = 0.05$ yields a high multi-testing false positive rate ($1 - 0.95^{10} \approx 40%$). Bonferroni divides significance threshold by number of tests: $\alpha_{\text{new}} = \alpha / K$.
- How large does a test set need to be to detect a 0.5% AUC lift? Depends on variance. Using power calculations for paired proportions, detecting a 0.5% difference with 80% power typically requires $N > 20,000$ paired test samples.
Check yourself
Why does running a standard Paired Student's t-test across 10 folds of standard 10-fold cross-validation yield an artificially inflated Type I error rate?