ROC-AUC vs PR-AUC

With 0.1% positives, ROC-AUC of 0.97 can mean a useless model.

🟡 intermediate4 min readmetricsmust-know

ROC-AUC plots true positive rate against false positive rate across all thresholds and equals the probability the model ranks a random positive above a random negative. It is threshold-free and prevalence-independent — which is exactly its weakness. Because FPR's denominator is the huge negative class, thousands of false positives barely move it. PR-AUC uses precision, whose denominator is what you flagged, so it collapses honestly when the positive class is rare. Use PR-AUC for imbalanced problems.

What each curve plots

ROC: TPR = TP/(TP+FN) on the y-axis against FPR = FP/(FP+TN) on the x-axis, sweeping the threshold from 1 to 0.

PR: Precision = TP/(TP+FP) against Recall = TP/(TP+FN).

Both are threshold-free summaries of a ranking. ROC-AUC has a beautiful interpretation: it is exactly the probability that a randomly chosen positive scores higher than a randomly chosen negative. (It equals the normalised Mann–Whitney U statistic.)

The denominator argument

This is the whole answer, and it is worth being able to state crisply.

FPR's denominator is TP + TN for the negative class — with 1,000,000 negatives, going from 0 to 10,000 false positives moves FPR from 0 to 0.01. Invisible on the plot.

Precision's denominator is TP + FP — what you actually flagged. Those same 10,000 false positives against 900 true positives give precision of 0.08. Devastating, and correctly so, because a human now has to sift 10,900 alerts to find 900 real cases.

ROC hides false positives behind a huge negative class. PR does not.

Choosing between them

Use ROC-AUC	Use PR-AUC
Roughly balanced classes	Heavy imbalance (< ~10% positive)
Both classes equally interesting	You only care about the positive class
Comparing across datasets with different prevalence	Fraud, disease, anomaly, spam, retrieval
You want a prevalence-stable number	You want a number that reflects reviewer workload

Note that "prevalence-independent" cuts both ways. It is a feature when you want to compare a model across two markets with different fraud rates. It is a bug when you want to know whether the model is deployable at today's rate.

The traps

PR-AUC's baseline is not 0.5. It is prevalence. A PR-AUC of 0.30 at 1% prevalence is a 30× lift — genuinely strong. Always report the baseline alongside.
Interpolation. Naive linear interpolation of a PR curve is wrong; use average precision (sklearn.metrics.average_precision_score), which is a proper step-wise sum, rather than trapezoidal auc().
Neither metric picks your threshold. Both summarise ranking quality across thresholds. You still have to choose an operating point from cost or capacity constraints.
Neither measures calibration. A model can rank perfectly (AUC 1.0) while every predicted probability is wrong. If you use the probabilities in downstream maths — expected value, bidding, triage cost — check calibration separately with a reliability curve or Brier score.

Say this out loud

"ROC-AUC is the probability a random positive outranks a random negative — it's prevalence-independent, which makes it stable but also lets it hide false positives when negatives dominate. For a rare positive class I'd report PR-AUC, or average precision, alongside the prevalence baseline, plus precision at whatever alert volume the review team can handle. And I'd check calibration separately if the probabilities feed a downstream decision."

Check yourself

Question 1 of 3

A dataset has 0.1% positives. ROC-AUC is 0.97 but PR-AUC is 0.08. The most likely reading is

More in Metrics & Evaluation

See all →

Precision, Recall & F14 min Why Accuracy Lies4 min Data Leakage4 min