Metrics & Evaluation

Why Accuracy Lies

Why 99 percent accuracy can trick you into keeping a completely useless machine learning model.

🟢 beginner4 min readmetricsmust-know
The accuracy trap happens when you use simple accuracy to judge a model on unbalanced datasets. In a rare disease dataset where 99 percent of people are healthy, a simple guessing model that predicts healthy for everyone achieves 99 percent accuracy while missing every single sick person. Solving this problem requires switching to metrics like precision, recall, and cost matrices.

The Math Behind the Trap

Accuracy is calculated as:

Accuracy = (True Positives + True Negatives) / Total Count

Imagine a bank fraud detection system:

  1. Total Transactions: 100000
  2. Normal Transactions: 99900 (99.9 percent)
  3. Fraudulent Transactions: 100 (0.1 percent)

Model A predicts normal for every single transaction. Model B is a smart model that catches 80 frauds but makes 1900 false alarms.

  1. Model A Accuracy: 99.90 percent. Recall: 0 percent.
  2. Model B Accuracy: 98.08 percent. Recall: 80 percent.

If you rely on raw accuracy, Model A looks better than Model B. But Model A is completely useless because it catches zero fraud!

Better Metrics to Use Instead

  1. Precision and Recall: Precision measures how many predicted positive cases were actually positive. Recall measures what percentage of real positive cases the model managed to catch.
  2. F1 Score: A single combined score that balances precision and recall so a model cannot cheat by ignoring one of them.
  3. Cost Matrices: Weighing errors by real financial cost. A missed fraud might cost 5000 dollars while a false alarm costs only 5 dollars.

Cost Weighted Decision Making

In real business settings, mistakes do not cost the same amount:

  1. Missed Fraud (False Negative): Costs 5000 dollars in stolen money.
  2. False Alarm (False Positive): Costs 5 dollars for a quick SMS check.

Calculating expected business loss:

Total Cost = (Missed Frauds * 5000) + (False Alarms * 5)

A model with lower accuracy that catches all fraud will save millions compared to a high accuracy model that lets fraud slip through.

Say this out loud

Accuracy is misleading on unbalanced datasets because common normal cases dominate the overall score. A dummy model predicting 100 percent normal achieves high accuracy while catching zero real events. For unbalanced problems like fraud or disease detection, we use precision, recall, F1 score, or business cost matrices.

Followups to expect

  1. When is raw accuracy a valid metric? Accuracy is fine when your dataset classes are roughly balanced, like a 50 50 split, and the costs of both types of mistakes are equal.
  2. What is balanced accuracy? Balanced accuracy takes the average recall score across each individual class, making sure no single class masks the rest.

Check yourself

Question 1 of 3

A fraud detection dataset contains 99900 normal transactions and 100 fraudulent transactions. A model predicts normal for every transaction. What is its accuracy and recall?

More in Metrics & Evaluation

See all →
Precision, Recall & F14 minROC-AUC vs PR-AUC4 minData Leakage4 min