Implement ROC-AUC from Scratch

Math Foundations

The ROC Curve plots True Positive Rate (TPR / Recall) against False Positive Rate (FPR) across all possible decision probability thresholds $\tau \in [0.0, 1.0]$:

$$\text{TPR}(\tau) = \frac{\text{TP}(\tau)}{\text{TP}(\tau) + \text{FN}(\tau)}, \quad \text{FPR}(\tau) = \frac{\text{FP}(\tau)}{\text{FP}(\tau) + \text{TN}(\tau)}$$

ROC-AUC calculates the total Area Under the ROC Curve:

$$\text{AUC} = \int_{0}^{1} \text{TPR}(\text{FPR}) , d(\text{FPR})$$

TPR (Recall)
1.0 ┼───────────────────────┐ ◄── Perfect Classifier (AUC = 1.0)
    │                      ╱│
    │                    ╱  │
0.5 ┼                  ╱    │ ◄── Random Classifier (AUC = 0.50)
    │                ╱      │
0.0 ┴──────────────┴────────┘
   0.0            0.5      1.0
             FPR (False Alarm Rate)

Pairwise Rank Order Interpretation

Mathematically, ROC-AUC equals the probability that a classifier ranks a randomly selected positive sample $i$ higher than a randomly selected negative sample $j$:

$$\text{AUC} = P(\hat{p}_i > \hat{p}_j \mid y_i = 1, y_j = 0)$$

This makes ROC-AUC completely threshold-independent and scale-invariant!

Efficient NumPy Implementation

import numpy as np


class ROCAUCFromScratch:

  def compute(self, y_true, y_scores):
    # Ensure numpy array inputs
    y_true = np.asarray(y_true)
    y_scores = np.asarray(y_scores)

    # 1. Sort predictions in descending order of score
    desc_indices = np.argsort(-y_scores)
    y_true_sorted = y_true[desc_indices]
    y_scores_sorted = y_scores[desc_indices]

    # 2. Identify distinct threshold indexes where score values change
    distinct_value_indices = np.where(np.diff(y_scores_sorted))[0]
    threshold_idxs = np.r_[distinct_value_indices, y_true_sorted.size - 1]

    # 3. Compute Cumulative True Positives and False Positives
    tps = np.cumsum(y_true_sorted)[threshold_idxs]
    fps = (1 + threshold_idxs) - tps

    # Add (0,0) starting origin point
    tps = np.r_[0, tps]
    fps = np.r_[0, fps]

    total_positives = tps[-1]
    total_negatives = fps[-1]

    if total_positives == 0 or total_negatives == 0:
      return 0.5  # Undefined single-class baseline

    # 4. Compute True Positive Rates and False Positive Rates
    tpr = tps / total_positives
    fpr = fps / total_negatives

    # 5. Compute Area Under Curve using Trapezoidal Rule
    # AUC = sum( (FPR_i - FPR_{i-1}) * (TPR_i + TPR_{i-1}) / 2 )
    auc = np.trapz(tpr, fpr)
    return auc

Say this out loud

ROC-AUC evaluates classifier ranking ability across all decision thresholds. It represents the probability that a randomly chosen positive sample receives a higher score than a randomly chosen negative sample. The implementation sorts predictions by score, calculates cumulative TPR and FPR across unique thresholds, and integrates area under the curve using the trapezoidal rule.

Followups to expect

1. Why can ROC-AUC be misleading on heavily imbalanced datasets? On highly imbalanced datasets with tiny positive class counts, FPR remains artificially low because TN is huge, making ROC-AUC look inflated. Use Precision-Recall AUC (PR-AUC) instead.
2. What is Wilcoxon-Mann-Whitney U Test connection to ROC-AUC? ROC-AUC is mathematically equivalent to the normalized Mann-Whitney U statistic computed on pair ranks: $\text{AUC} = U / (N_{\text{pos}} \times N_{\text{neg}})$.