What to Monitor in Production

The core observability pillars required to detect model failure, data drift, and latency degradation in production.

🟡 intermediate5 min readmonitoringmust-know

Model Monitoring tracks the health of deployed ML systems across four distinct layers: Software System Metrics (Latency, QPS, Memory), Input Data Drift (Feature distribution PSI/KS-tests), Prediction Drift (Output score distribution shifts), and Model Quality (Ground-truth performance). When true labels arrive with long delays (e.g. 30-day loan defaults), input and prediction drift act as early proxy indicators of performance degradation.

The 4 Pillars of ML Production Monitoring

                                  PRODUCTION MONITORING STACK
┌──────────────────────────┬──────────────────────────┬──────────────────────────┬──────────────────────────┐
│  1. SYSTEM INFRASTRUCTURE│   2. INPUT DATA DRIFT    │   3. PREDICTION DRIFT    │   4. MODEL PERFORMANCE   │
├──────────────────────────┼──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ - Latency (p50, p99)     │ - Feature PSI / KS-test  │ - Output Score Shifts    │ - Precision / Recall     │
│ - QPS / Throughput       │ - Missing Value Spikes   │ - Positive Class Rate    │ - ROC-AUC / RMSE         │
│ - Memory / GPU Util      │ - Schema Invalidation    │ - Anomaly Score Spikes   │ - Business Conversion    │
└──────────────────────────┴──────────────────────────┴──────────────────────────┴──────────────────────────┘

Immediate vs Delayed Label Monitoring

  [ IMMEDIATE LABELS ] (Search / Ad Clicks)      ──► Calculate Real-Time CTR, Precision@k, NDCG
  [ DELAYED LABELS ]   (Credit Default / Fraud)  ──► Monitor Input Drift P(X) & Prediction Drift P(Ŷ)

For delayed labels, Prediction Drift is your primary alarm:

If baseline model predicted 2% positive rate, and live serving predictions suddenly spike to 12% positive rate, the input distribution has shifted or an upstream feature pipeline broke.

Threshold Reference for PSI (Population Stability Index)

$$PSI = \sum_{i=1}^B \left( \text{Actual}_i - \text{Expected}_i \right) \times \ln\left( \frac{\text{Actual}_i}{\text{Expected}_i} \right)$$

PSI < 0.10: No significant distribution shift. No action needed.
0.10 ≤ PSI < 0.25: Moderate drift. Issue warning, monitor closely, prepare retraining.
PSI ≥ 0.25: Significant drift! Trigger alert, initiate fallback rules or model retraining.

Say this out loud

"Production ML monitoring requires tracking four layers: System SLAs (p99 latency, QPS), Input Data Drift (PSI, schema checks), Prediction Drift (output score distribution shifts), and True Performance (precision, recall). When ground-truth labels arrive late, input feature drift P(X) and output prediction drift P(Ŷ) act as critical early proxies to catch model failures before business metrics suffer."

Follow-ups to expect

What is an Upstream Pipeline Failure vs Data Drift? Pipeline failures are sudden engineering bugs (e.g. app update sending NULL strings, timezone bugs, currency conversion errors) yielding instant PSI spikes. Data drift is gradual real-world user behavior shift over months.
How do you handle shadow deployments? Run new candidate model in parallel with live production model, feeding it real-time queries but discarding outputs. Compare latency, prediction drift, and outputs against live model safely before switching traffic.

Check yourself

Question 1 of 3

What should an MLOps team monitor in real-time when true target labels Y arrive with a 60-day delay (e.g. credit default)?

More in MLOps & Production

See all →

Data Drift vs Concept Drift4 min Point-in-Time Correct Feature Joins5 min Debugging a Production Model Incident5 min