Data Drift vs Concept Drift
Why models that hit 99% accuracy in offline testing decay silently 3 months after deployment.
Data Drift (Covariate Shift) occurs when feature input distributions P(X) change over time while target rules P(Y|X) remain fixed. Concept Drift occurs when the relationship between features and target P(Y|X) changes. Detecting drift requires statistical tests like Population Stability Index (PSI), Kolmogorov-Smirnov (KS) test, and adversarial drift classifiers. Mitigations range from feature re-scaling to retraining schedules.
Defining Covariate Shift vs Concept Shift vs Prior Shift
Model predictions depend on the joint distribution P(X, Y) = P(X) · P(Y | X) = P(Y) · P(X | Y).
Type of Drift What Changes? What Stays Constant?
──────────────────────────────────────────────────────────────────────────
Data Drift (Covariate) P(X) changes P(Y | X) unchanged
Concept Drift P(Y | X) changes P(X) can be constant
Prior Shift P(Y) changes P(X | Y) unchanged
- Data Drift (Covariate Shift): Demographic shifts in app users. Inputs
P(X)change, but a given user profile still defaults on loans at the same historical rateP(Y|X). - Concept Drift: Sudden fraud tactic shift or economic shock. The same input vector
Xnow maps to a completely different default probabilityP(Y|X). - Prior Shift: Fraud rate spikes from 1% to 10% overall (
P(Y)), but fraud patterns within classesP(X|Y)remain identical.
Detection Methods (PSI, KS Test, Drift Classifiers)
| Method | Data Type | Mechanics |
|---|---|---|
| Population Stability Index (PSI) | Categorical & Binned Numeric | PSI = ∑ (Actual% - Expected%) × ln(Actual% / Expected%). PSI ≥ 0.25 signals severe drift. |
| Kolmogorov-Smirnov (KS) Test | Continuous Numeric | Non-parametric test comparing cumulative distributions (CDF). Low p-value (< 0.05) rejects null hypothesis of identical distributions. |
| Adversarial Drift Classifier | High-dimensional vectors / text | Train GBDT on (Train Data = 0, Serving Data = 1). If classifier AUC > 0.7, significant drift exists. Feature importances reveal which features drifted. |
Mitigation Strategies & Retraining Triggers
- Rule-Based Triggers: Retrain when PSI > 0.25 or when production performance drops below SLA.
- Scheduled Retraining: Retrain daily, weekly, or monthly on rolling time windows.
- Domain Adaptation / Importance Weighting: Weight training samples by
P_prod(X) / P_train(X)to correct covariate shift without full retraining. - Feature Dropping: Remove highly volatile features that drift rapidly without driving major predictive signal.
Say this out loud
"Data drift is a shift in feature distributions P(X), while concept drift is a shift in the target relationship P(Y|X). We detect data drift without labels using PSI, KS tests, or adversarial classifiers trained to separate training logs from production logs. When concept drift occurs, model performance degrades and requires retraining on newly collected, newly labelled data."
Follow-ups to expect
- How do you monitor drift when true labels arrive weeks or months late (e.g. credit default)? Monitor input feature drift P(X) and prediction drift P(y_hat) as proxy indicators, since true labels P(Y) are delayed.
- What is upstream data pipeline drift? Systemic bugs (e.g. app update changing string formats, missing telemetry fields, time zone bugs) that alter feature values without any real-world user behavior shift.
- How does online continuous learning differ from batch retraining? Online learning updates weights incrementally with each streaming sample (e.g. SGD/FTRL in ad click prediction), while batch retraining re-runs full training jobs on sliding historical data windows.
Check yourself
Question 1 of 3
A sudden macroeconomic shock changes consumer purchasing behaviour so that a credit scoring model's inputs X remain identical, but default rates P(Y|X) double. This is an example of