Online vs Offline Evaluation

Comparison Matrix

       HISTORICAL LOGGED DATA                         LIVE PRODUCTION USERS
 ┌─────────────────────────────────┐           ┌─────────────────────────────────┐
 │       OFFLINE EVALUATION        │           │        ONLINE EVALUATION        │
 │  - ROC-AUC, PR-AUC, NDCG@k      │   ─────►  │  - A/B Testing, Interleaving   │
 │  - Fast, cheap, repeatable      │           │  - CTR, Conversion, Latency     │
 │  - Prone to offline-online gap  │           │  - Slow, expensive, real risk   │
 └─────────────────────────────────┘           └─────────────────────────────────┘

Why the Offline-Online Gap Happens

1. Position & Logging Bias: Offline data records clicks on items recommended by the previous model. Items never shown have zero logs, even if users would have loved them.
2. Static vs Dynamic Distribution: Offline data assumes user behavior is static. Online, showing new recommendations changes user exploration and long-term engagement.
3. Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." Over-optimizing an offline metric (e.g. optimizing pure CTR) leads to clickbait, degrading online long-term retention.

Bridge Techniques: IPS & Interleaving

1. Inverse Propensity Scoring (IPS) for Counterfactual Offline Eval

Reweights logged interactions by the inverse probability that the old logging policy $\pi_0$ displayed action $a$ in context $x$:

$$\hat{V}{IPS}(\pi{\text{new}}) = \frac{1}{N} \sum_{i=1}^N \frac{\pi_{\text{new}}(a_i | x_i)}{\pi_0(a_i | x_i)} \cdot r_i$$

Corrects historical logging bias, providing unbiased offline estimates of new policies!

2. Team Draft Interleaving

Instead of 50/50 user splitting in A/B testing:

• Interleave recommendations from Model A and Model B into a single blended list shown to every user.
• Count relative clicks on items attributed to A vs B. Requires 100x smaller sample size to detect winner!

Say this out loud

"Offline evaluation filters weak models quickly using historical logs and metrics like NDCG and AUC. Online evaluation validates business ROI using live A/B testing and metrics like CTR and conversion. The offline-online gap occurs due to position bias and logging bias in static data. We bridge this gap using Inverse Propensity Scoring for counterfactual offline evaluation and Interleaving for rapid online pre-testing."

Follow-ups to expect

• What is Guardrail Metric in online evaluation? A production metric (e.g. Page Load Latency, Crash Rate, Unsubscribe Rate) that must NOT degrade below a threshold during an A/B test, even if primary CTR increases.
• How do you handle delayed labels in online evaluation? In conversion rate (CVR) modeling, users may click an ad today but purchase 14 days later. Evaluating short-term CVR undercounts conversions; use survival analysis or windowed attribution.