Handling Imbalanced Datasets

The Accuracy Trap in Imbalanced Data

Consider a fraud detection dataset containing 99,900 legitimate transactions and 100 fraudulent transactions ($99.9%$ vs $0.1%$).

If you train a classifier using standard accuracy:

A naive dummy model that predicts "Legitimate" for every single transaction achieves 99.9% Accuracy!

The model receives an A+ score while catching zero fraud cases, completely failing its business objective.

  99.9% Legitimate / 0.1% Fraud ──► Naive Dummy Classifier ("Always Legitimate") ──► 99.9% Accuracy! (0% Fraud Detected!)

Toolbox for Handling Imbalanced Data

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. DATA RESAMPLING       │ 2. COST SENSITIVE        │ 3. PROPER METRICS        │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Undersample majority.    │ Assign higher loss       │ Evaluate via PR-AUC,     │
│ Oversample minority      │ weight to minority       │ F1-Score, Macro Recall.  │
│ (SMOTE interpolation).   │ class (class_weight).    │ Ditch raw Accuracy!      │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Data Resampling Strategies

A. Random Undersampling

Randomly drop majority class samples to match minority count.

• Pros: Reduces dataset size and speeds up training.
• Cons: Discards potentially valuable majority class information.

B. SMOTE (Synthetic Minority Over-sampling Technique)

Instead of simply duplicating minority rows, SMOTE (Chawla et al., 2002) creates synthetic new samples by interpolating feature space between k-nearest minority neighbors:

$$x_{\text{new}} = x_i + \lambda \cdot (x_{\text{neighbor}} - x_i), \quad \lambda \sim U(0, 1)$$

  Minority Point A ──────●────── Minority Point B (k-NN Neighbor)
                         ▲
                Synthetic SMOTE Sample Created Here!

Rule: Apply SMOTE ONLY to the training split. Never apply SMOTE to validation or test data!

2. Cost-Sensitive Learning (Loss Reweighting)

Instead of changing dataset rows, change the Loss Function:

$$\text{Class Weight}_k = \frac{N}{C \cdot N_k}$$

• Assigns a large weight penalty to misclassifying minority samples ($N_k$ is small).
• Assigns a tiny weight penalty to majority samples.

In PyTorch or scikit-learn, set class_weight='balanced' or use Focal Loss to automatically down-weight easy background samples.

3. Evaluation Metrics for Imbalanced Data

Say this out loud

Imbalanced datasets cause models to achieve fake high accuracy by predicting majority classes while failing minority events. Fixes include SMOTE synthetic minority oversampling, cost sensitive loss reweighting (class_weight = balanced), and evaluating models using Precision Recall Area Under the Curve (PR-AUC) and F1-Score instead of raw accuracy.

Followups to expect

1. What is Borderline-SMOTE? A variant of SMOTE that generates synthetic samples only near class decision boundaries (where minority samples are surrounded by majority neighbors) rather than in safe deep interior regions.
2. How do decision thresholds impact imbalanced classification? Default decision threshold is 0.5. Lowering threshold to 0.1 increases recall for minority fraud events at the cost of lower precision, tuned using precision-recall curves.