Monitoring When Labels Arrive Late
Evaluating production model performance when ground truth labels take days, weeks, or months to arrive.
The Delayed Feedback Problem
In ideal machine learning tutorials, ground truth labels arrive instantly.
In real world production systems, labels are frequently delayed:
- Ad Conversion Rates: A user clicks an ad today, but buys the product 14 days later.
- Credit Risk: A bank issues a loan today, but loan default events occur 12 to 36 months later.
- Customer Churn: A user stops using an app gradually, with churn confirmed only after 60 days of inactivity.
Timestamp T_0: Model makes Prediction (Loan Approved)
│
▼ (Delay Window: 30 to 365 Days)
│
Timestamp T_1: True Ground Truth Label Arrives (Loan Defaulted or Paid)
If you wait for true labels before monitoring your model, a failing model could operate undetected for months!
How to Monitor Models Under Delayed Labels
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. FEATURE DRIFT (PSI) │ 2. OUTPUT SCORE DRIFT │ 3. PROXY LABELS │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Monitor input features │ Track shifts in predicted│ Use early short term │
│ for statistical shifts │ probability score │ signals that correlate │
│ using PSI or KS tests. │ distributions over time. │ with long term outcomes. │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘
1. Monitor Input Feature Drift
Even if true labels are missing, you can monitor whether input features $P(X)$ match historical training features. If feature drift (PSI $> 0.20$) is detected, model predictions are no longer reliable.
2. Monitor Prediction Output Distributions
Track statistical distributions of predicted probabilities $\hat{Y}$. If a credit model historically predicted a 3 percent default rate, and suddenly begins predicting a 12 percent default rate over 24 hours, the system has experienced an environment shift.
3. Use Short Term Proxy Labels
Instead of waiting 12 months for final outcomes, define early Proxy Labels:
- Long term target: Loan default over 2 years.
- Short term proxy: Late payment within the first 30 days.
Training early iteration models or monitoring proxy metrics provides faster feedback loops.
Temporal Join Architecture
When delayed labels finally arrive, join them back to the original feature vector recorded at prediction time $T_0$:
Log Event at T_0: [ Prediction ID: #8492 | Features: X_0 | Prediction: 0.12 ]
│
(Time Gap 14 Days)
▼
Log Event at T_1: [ Prediction ID: #8492 | Ground Truth Label: Y = 1 ]
│
▼
Joined Dataset ──► Used for Model Retraining!
Ensure point in time correctness during joins to prevent data leakage.
Say this out loud
Monitoring under delayed labels requires tracking surrogate metrics because ground truth outcomes take days or months to observe. Teams monitor input feature drift, output prediction score distributions, and short term proxy labels to catch model failures early. When true labels finally arrive, asynchronous temporal joins reconnect outcomes to original prediction features for retraining.
Followups to expect
- What is Attribution Window in ad conversion modeling? The maximum allowed time window (for example 7 days or 30 days) between an ad click and a purchase event for the conversion to be credited to that ad prediction.
- What is Survival Analysis in delayed label modeling? Statistical techniques used to estimate the time elapsed until an event occurs, accounting for censored data samples where outcomes have not yet happened.
Check yourself
Why is real time performance monitoring difficult for credit default prediction models?