CTR Prediction & Calibration

What is CTR Prediction?

In online advertising (Google Ads, Meta Ads) and recommendation feeds:

Click-Through Rate (CTR) Prediction estimates the exact probability that a user $u$ will click a candidate item or ad $i$ in context $c$:

$$\text{pCTR} = P(\text{Click} = 1 \mid \text{User } u, \text{Ad } i, \text{Context } c)$$

  USER IMPRESSION ──► [ CTR PREDICTION MODEL ] ──► Predicted pCTR = 0.042 (4.2% Probability!)

Why Calibration is Mandatory in Ad Auctions

In standard classification, a model with high ROC-AUC is sufficient. As long as Positive items receive higher scores than Negative items, the exact numerical scale of predictions does not matter.

In Ad Auctions, numerical scale is everything!

Ad networks rank ads by Expected Revenue per Impression (eCPM):

$$\text{eCPM} = \text{pCTR} \times \text{Bid}_{\text{CPC}} \times 1000$$

• $\text{Bid}_{\text{CPC}}$: Cost-Per-Click dollar bid placed by advertiser (e.g. $$2.00$).

  UNCALIBRATED MODEL FAILURE:
  Model predicts pCTR = 0.40 (10x Over-confident! True CTR is 0.04).
  eCPM = 0.40 * $2.00 * 1000 = $800 eCPM!
  The un-calibrated ad wins the auction illegally, advertiser gets 0 clicks, and platform loses trust!

If predicted probabilities are un-calibrated, auction rankings collapse and advertisers get overcharged.

Measuring Calibration: Expected Calibration Error (ECE)

To evaluate calibration, divide predictions into $M$ probability bins (e.g. $[0.0, 0.1], (0.1, 0.2], \dots$):

$$\text{ECE} = \sum_{m=1}^M \frac{|B_m|}{N} \left| \text{acc}(B_m) - \text{conf}(B_m) \right|$$

• $\text{conf}(B_m)$: Average predicted probability inside bin $B_m$.
• $\text{acc}(B_m)$: True empirical click fraction inside bin $B_m$.

A perfectly calibrated model has $\text{ECE} = 0.0$ (Reliability Diagram follows a 45-degree line!).

  RELIABILITY DIAGRAM (CALIBRATION PLOT)
  True Click Rate
  1.0 ┤                      / ◄── Perfect Calibration (45-degree line!)
      │                     /
  0.5 ┤            •  •  • / ◄── Un-calibrated Over-confident Model!
      │       •  •
  0.0 ┴───────┴───────────┴──────────► Predicted Probability pCTR
             0.5         1.0

Post-Processing Calibration Techniques

If a trained neural network outputs un-calibrated probabilities:

┌──────────────────────────┬──────────────────────────┐
│ 1. PLATT SCALING         │ 2. ISOTONIC REGRESSION   │
├──────────────────────────┼──────────────────────────┤
│ Fits a 1D Logistic       │ Fits a non-parametric    │
│ Regression model over    │ monotonic step-function  │
│ raw model logits:        │ mapping raw predictions  │
│ p_cal = σ(a * logit + b).│ to true frequencies.     │
└──────────────────────────┴──────────────────────────┘

Key CTR Model Architectures

1. Factorization Machines (FM): Models $2\text{nd}$-order feature interactions using dot products of low-dimensional vectors.
2. Deep & Cross Network (DCNv2 - Wang et al., 2021): Applies explicit cross layers to compute degree-1, 2, 3 feature interactions automatically alongside deep MLP layers.
3. Deep Interest Network (DIN - Zhou et al., 2018 / Alibaba): Uses attention mechanisms over user click history items to activate relevant historical preferences for candidate ads.

Say this out loud

CTR Prediction estimates click probabilities for ad auctions and feeds. Probability Calibration is mandatory because ad auctions rank by Expected Value = pCTR * Bid. Un calibrated probabilities distort auction economics. Models are evaluated using Log Loss and Expected Calibration Error (ECE), and calibrated post training using Platt Scaling or Isotonic Regression.

Followups to expect

1. What is CVR Prediction vs CTR Prediction? CTR predicts click probability $P(\text{Click} \mid \text{Impression})$. CVR predicts conversion probability $P(\text{Conversion} \mid \text{Click})$. Entire conversion pipelines predict ESMM (Entire Space Multi-task Model) to avoid sample selection bias.
2. Why is Log Loss preferred over ROC-AUC for CTR optimization? Log Loss measures exact probabilistic cross-entropy divergence, directly punishing un-calibrated probability errors, whereas ROC-AUC measures rank ordering independent of probability scale.