Debugging a Production Model Incident

Production ML Incident Playbook

                  ┌───────────────────────────────────────────────┐
                  │ 1. INCIDENT ALERT (Conversion Drops / SLA Peak)│
                  └───────────────────────┬───────────────────────┘
                                          │
                                          ▼
                  ┌───────────────────────────────────────────────┐
                  │ 2. IMMEDIATE MITIGATION & FALLBACK (Stop Bleeding)
                  │    - Flip Kill-Switch to Rule Baseline / Rollback
                  └───────────────────────┬───────────────────────┘
                                          │
                                          ▼
                  ┌───────────────────────────────────────────────┐
                  │ 3. ROOT-CAUSE TRIAGE (Isolate Layer)          │
                  │    - Data / Infra / Model / Environment       │
                  └───────────────────────┬───────────────────────┘
                                          │
                                          ▼
                  ┌───────────────────────────────────────────────┐
                  │ 4. FIX & RE-DEPLOYMENT (Validate & Canary)    │
                  └───────────────────────┬───────────────────────┘
                                          │
                                          ▼
                  ┌───────────────────────────────────────────────┐
                  │ 5. POST-MORTEM & AUTOMATED SAFEGUARDS         │
                  └───────────────────────────────────────────────┘

Step 2: Immediate Mitigation (Kill-Switch & Rollback)

Never debug in a live production fire. Protect revenue first:

1. Model Rollback: Revert traffic pointer to previous stable model container tag ($V_{t-1}$).
2. Circuit Breaker / Fallback: Switch model traffic to static heuristic baseline (e.g. top-trending items, global popularity).

Step 3: Root-Cause Triage Decision Tree

                                  Where is the Failure?
                   ┌────────────────────────┴────────────────────────┐
                   ▼                                                 ▼
          Data & Pipeline Issues                            Serving & Infra Issues
- Upstream app schema change (e.g. Nulls)          - Feature Store latency timeout (>50ms)
- Unit conversion mismatch (cents vs $)            - GPU OOM (Out-of-Memory) crash
- Point-in-time feature store lag                  - Un-batched inference QPS bottleneck
- Missing categorical encodings                    - Model server pod crash-looping

Top Suspect: Upstream Schema Break

Check feature null counts and distribution histograms between serving logs and baseline training. In 80% of incidents, an unannounced frontend/backend app release changed telemetry logging formats.

Step 5: Post-Mortem & Preventative Safeguards

1. Great Expectations / Deequ: Enforce automated data validation contracts at ingestion boundaries.
2. Shadow Deployments: Route live traffic to new models in shadow mode for 48 hours before promoting.
3. Integration Tests: Add CI/CD tests verifying model input/output shapes, non-NaN bounds, and unit conversions.

Say this out loud

"When responding to a production ML incident, first mitigate business impact by triggering a circuit breaker fallback or rolling back to a known stable model version. Next, isolate the layer by checking upstream data pipelines for schema breaks, feature store lag, or null spikes—which cause 80% of outages. Once fixed, re-deploy via canary traffic and add automated data quality contracts to prevent recurrence."

Follow-ups to expect

• What is a Canary Deployment? Gradually shifting live user traffic to a new model version (e.g., 1% -> 5% -> 25% -> 100%) while continuously monitoring automated error metrics to catch regressions early.
• How do you test a model rollback system? Perform regular GameDay fire-drills where engineers deliberately trigger circuit breakers and measure Time-to-Detect (TTD) and Time-to-Mitigate (TTM).