Detecting Drift: PSI, KS, KL

Why Models Degrade in Production

A machine learning model trained on historical data assumes that future production data will follow the exact same statistical distribution.

In the real world, user behavior, macroeconomic trends, and seasonality change constantly. When production distributions move away from training distributions, model accuracy drops. This is called Model Drift.

Training Data Distribution P(X)  ◄── Compare Statistical Distance ──►  Live Production Data Distribution Q(X)

Types of Drift

1. Data Drift (Covariate Shift): The distribution of input features $P(X)$ changes, but the underlying decision mapping stays the same. For example, user age distributions shift toward older demographics.
2. Concept Drift: The relationship between input features and target labels $P(Y \mid X)$ changes. For example, what constituted normal online transaction behavior pre-pandemic became fraudulent during lockdown.

Key Drift Detection Methods

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. PSI (Stability Index) │ 2. KS TEST               │ 3. KL DIVERGENCE         │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Bins features into       │ Non-parametric test for  │ Information theory metric│
│ buckets and measures     │ continuous features;     │ measuring relative       │
│ percentage shifts.       │ measures max distance    │ entropy between two      │
│ Standard in banking!     │ between distributions.   │ probability distributions│
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Population Stability Index (PSI)

PSI divides continuous or categorical feature values into $B$ buckets and compares baseline distribution percentages $B_i$ against target distribution percentages $T_i$:

$$\text{PSI} = \sum_{i=1}^B \left( T_i - B_i \right) \times \ln\left( \frac{T_i}{B_i} \right)$$

• PSI $< 0.10$: No significant distribution change.
• $0.10 \le \text{PSI} \le 0.20$: Moderate shift; flag for monitoring.
• PSI $> 0.20$: Significant distribution drift; trigger model retraining!

2. Kolmogorov Smirnov (KS) Test

The KS test is a non-parametric statistical test used for continuous numerical features. It compares the Cumulative Distribution Functions (CDF) of baseline data $F_1(x)$ and production data $F_2(x)$:

$$D = \max_x |F_1(x) - F_2(x)|$$

If the maximum distance $D$ exceeds critical p-value thresholds, the system flags the feature as drifted.

3. Kullback Leibler (KL) Divergence

KL Divergence measures how much information is lost when approximating a production probability distribution $Q(x)$ using baseline training distribution $P(x)$:

$$D_{\text{KL}}(P \parallel Q) = \sum_{x} P(x) \log\left( \frac{P(x)}{Q(x)} \right)$$

Say this out loud

Drift detection monitors statistical changes between training baselines and live production data. Data drift measures shifts in input feature distributions, while concept drift measures changes in the relationship between features and target labels. Techniques like Population Stability Index, Kolmogorov Smirnov tests, and KL Divergence quantify distribution distance to trigger automated retraining before model accuracy drops.

Followups to expect

1. How often should drift detection monitors run? Run feature level data drift checks daily or hourly on streaming pipelines, while concept drift checks run whenever ground truth labels arrive.
2. What is Jensen Shannon Divergence? A symmetric, bounded variation of KL divergence that provides a normalized distance score between zero and one, making it easier to compare drift across different features.