Feedback Loops in Recommenders

What is a Recommender Feedback Loop?

In offline machine learning, training datasets are static.

In online production recommendation systems, the model actively shapes the dataset it will train on tomorrow:

  1. Recommender Model outputs Item A at Rank 1.
                       │
                       ▼
  2. Users click Item A because it is prominently displayed.
                       │
                       ▼
  3. System logs record 10,000 Clicks on Item A (Logged Training Data!).
                       │
                       ▼
  4. Retrained Model becomes 10x MORE CONFIDENT that Item A is the best!
                       │
                       ▼
  5. REPEAT LOOP: Item A dominates 100% of user feeds! (FEEDBACK LOOP!)

This creates a Self-Reinforcing Feedback Loop (System Degeneracy).

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ POPULARITY AMPLIFICATION │ FILTER BUBBLES           │ CATALOG COLLAPSE         │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Popular items get richer;│ Users receive narrow,    │ 95% of catalog items     │
│ niche long-tail items    │ echo-chamber content     │ receive ZERO exposure,   │
│ starve for exposure.     │ matching past clicks.    │ wasting licensing costs. │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

The Consequences of Un-Mitigated Feedback Loops

1. Popularity Bias Explosion: Rich-get-richer dynamics where top $0.1%$ of items absorb $90%$ of traffic.
2. Homogenization: Users with different tastes receive increasingly identical, generic recommendations.
3. Loss of Novelty: Users stop discovering new creators, products, or genres, leading to platform boredom.

Production Defenses Against Feedback Loops

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. BANDIT EXPLORATION    │ 2. CAUSAL DEBIASING (IPS)│ 3. COLD-START REPLAYS    │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Reserve 5% of traffic to │ Apply Inverse Propensity │ Inject fresh un-tried    │
│ serve random long-tail   │ Weights to down-weight   │ items into feeds to measure│
│ items via Thompson Sampling| clicks on top items.   │ true organic CTR.        │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Multi-Armed Bandits & Exploration

Inject controlled Exploration into real-time feeds using Thompson Sampling or $\epsilon$-Greedy:

• $95%$ of recommendations are served by the primary ranking model (Exploitation).
• $5%$ of recommendations are reserved for exploring un-tested long-tail items (Exploration).

This collects unbiased interaction data for long-tail items, feeding true signals into future retraining runs.

2. Causal IPS Weighting

Apply Inverse Propensity Scoring (IPS) during training to penalize clicks generated on high-exposure items, boosting positive feedback signals received by low-exposure items.

3. Model Ensembling & Diversity Constraints

Enforce hard catalog diversity constraints (e.g. Maximal Marginal Relevance) to prevent single popular categories from dominating user feeds.

Say this out loud

Feedback Loops occur when recommendation models train on user interaction logs generated by their own past predictions. This creates self-reinforcing loops that amplify popularity bias, cause filter bubbles, and trigger catalog collapse. Defenses include Multi-Armed Bandit exploration, Inverse Propensity Scoring, and catalog diversity constraints.

Followups to expect

1. What is Off-Policy Evaluation (OPE)? Evaluating a new recommendation algorithm $B$ using historical interaction logs collected under an old recommendation policy $A$ without deploying policy $B$ to live production users.
2. What is Simpson's Paradox in Feedback Loops? Aggregated click data can show an item performs well overall, while user-level disaggregated data reveals the item only succeeded due to forced high exposure at position 1.