Diversity, Novelty & Filter Bubbles

Beyond Pure Accuracy & Click-Through Rate

Standard recommendation systems optimize for immediate Accuracy or Click-Through Rate (CTR).

However, a system that optimizes $100%$ for CTR creates severe product degradation:

  PURE CTR OPTIMIZATION FAILURE:
  User watches 1 Star Wars clip ──► System recommends 50 Star Wars videos in a row!
  (User gets bored of repetitive recommendations and leaves the platform!)

This phenomenon is called a Filter Bubble (Echo Chamber).

To maintain long-term user engagement, industrial recommendation platforms measure and optimize Three Non-Accuracy Metrics:

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. DIVERSITY             │ 2. NOVELTY               │ 3. SERENDIPITY           │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Dissimilarity among items│ Measure of long-tail     │ Measures items that are  │
│ inside a single top-K    │ obscurity. Recommending  │ BOTH unexpected (novel)  │
│ recommendation list.     │ un-popular items.        │ AND relevant/delightful! │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Intra-List Diversity (ILD)

Measures average pairwise dissimilarity among items in a recommended list $L$:

$$\text{ILD}(L) = \frac{2}{|L|(|L|-1)} \sum_{i \in L} \sum_{j \in L, j \neq i} \left( 1 - \text{CosineSim}(v_i, v_j) \right)$$

A high ILD score guarantees the list contains a mix of categories (e.g. 2 Comedy movies, 1 Tech video, 1 News article, 1 Music track).

2. Novelty (Self-Information)

Measures how unexpected or rare an item $i$ is relative to global item popularity:

$$\text{Novelty}(i) = -\log_2 P(i) = -\log_2 \left( \frac{\text{Clicks}(i)}{\text{Total Clicks}} \right)$$

Recommending blockbusters (like Avengers) yields low novelty. Recommending obscure indie films yields high novelty.

3. Serendipity (Surprising Relevance)

Novelty alone is not enough: recommending a random obscure Hungarian cooking video to an American sports fan is novel, but useless.

Serendipity requires an item to be both Unexpected AND Relevant:

$$\text{Serendipity} = \text{Novelty} \times \text{Relevance} \times (1 - \text{Similarity}(\text{User History}))$$

Serendipity measures the "I didn't know I would love this!" user reaction.

Maximal Marginal Relevance (MMR - Carbonell & Goldstein)

How do production rerankers inject diversity into top-$K$ recommendations?

Maximal Marginal Relevance (MMR) iteratively selects items by balancing Relevance Score against Similarity to already-selected items $S$:

$$\text{MMR} = \arg\max_{i \in R \setminus S} \left[ \lambda \cdot \text{Sim}1(i, \text{User}) - (1 - \lambda) \max{j \in S} \text{Sim}_2(i, j) \right]$$

• $\lambda = 1.0 \implies$ Pure Relevance (Zero Diversity).
• $\lambda = 0.0 \implies$ Pure Diversity (Ignores User Relevance).
• $\lambda = 0.7 \implies$ Production Sweet Spot (High Relevance + Good Diversity).

Say this out loud

Optimizing recommendation engines purely for CTR creates Filter Bubbles and user fatigue. Diversity measures dissimilarity across items in a list. Novelty measures long tail item obscurity. Serendipity measures items that are both unexpected and relevant. Rerankers like Maximal Marginal Relevance (MMR) use parameter lambda to balance relevance against diversity.

Followups to expect

1. What is Determinantal Point Processes (DPP) for diversity? A probabilistic model that samples subsets of items proportional to the volume spanned by their feature vectors, providing elegant diversity constraints for recommendation lists.
2. What is Cold-Start Exploration vs Diversity? Using Thompson Sampling or Epsilon-Greedy exploration to inject new or un-popular items into user feeds to measure true item popularity without destroying user trust.