Position Bias & Debiasing Clicks

What is Position Bias?

In search engines and recommendation feeds, users examine top items far more frequently than bottom items.

Eye-tracking studies show:

• Rank 1: $80%$ of users examine the item.
• Rank 10: $<10%$ of users examine the item.

Because a user cannot click an item they never saw, raw click data measures Examination Probability $\times$ True Relevance, rather than True Relevance alone:

$$P(\text{Click} = 1) = P(\text{Examination} \mid \text{Position}) \times P(\text{Relevance} \mid \text{Item}, \text{User})$$

  Top Item (Rank 1):   High Examination (80%) x Low Relevance (10%)  ──► 8% Click Rate!
  Bottom Item (Rank 10): Low Examination (5%) x High Relevance (90%) ──► 4.5% Click Rate!

If you train a model directly on raw click logs, the model will learn that Rank 1 items are superior simply because they received more clicks!

1. Inverse Propensity Scoring (IPS)

Inverse Propensity Scoring (IPS) reweights click loss by the inverse of the Examination Propensity $P(\text{Examine} \mid \text{Rank})$:

$$\mathcal{L}{\text{IPS}}(\theta) = \sum{i=1}^N \frac{Y_i \cdot \text{Loss}(f(x_i), Y_i)}{P(\text{Examine} \mid \text{Rank}_i)}$$

• Clicks at Rank 1 (high examination $P=0.80$) are down-weighted ($\text{Weight} = 1 / 0.80 = 1.25$).
• Clicks at Rank 10 (low examination $P=0.05$) are heavily up-weighted ($\text{Weight} = 1 / 0.05 = 20.0$).

This un-biases the loss function, allowing the model to learn true item relevance $P(\text{Relevance})$.

2. Position Feature Trick (The Production Standard)

Instead of complex propensity estimation, add Item Position as an explicit input feature during training:

  TRAINING PHASE:
  Inputs: [ User Features, Item Features, Rank Position (e.g. Rank 4) ] ──► Predict Click!

Because Position is an explicit feature, the neural network isolates the position effect into the position feature weights.

  INFERENCE PHASE (Serving Real-Time Predictions):
  Inputs: [ User Features, Item Features, DEFAULT FIXED POSITION (Rank 1) ] ──► Rank Items!

By setting the Position feature to a fixed constant (e.g. Rank 1) for all candidate items during serving, position bias is completely stripped from inference predictions!

3. Intervention A/B Swap Testing

How do you estimate true examination probabilities $P(\text{Examine} \mid \text{Rank})$?

Run a small Intervention A/B Swap Experiment:

• Randomly swap Rank 1 and Rank 2 items for $1%$ of live production search queries.
• Measure the resulting click drop to calculate exact position propensity ratios.

Say this out loud

Position Bias occurs when top rank items receive more clicks due to higher visual examination probability rather than true relevance. Debiasing techniques include Inverse Propensity Scoring (IPS) which reweights clicks by inverse examination probability, and passing Position as an input feature during training while fixing Position to a constant during inference serving.

Followups to expect

1. What is Selection Bias vs Position Bias in RecSys? Position bias is user preference for top visual ranks. Selection bias occurs because users can only interact with items presented by the previous recommendation model, leaving un-presented items un-observed.
2. What is the Doberman / Cascade Model of Examination? Assumes users scan search results sequentially from top to bottom, stopping examination once a satisfactory relevant item is found and clicked.