Design: Search Ranking

The Problem

Design a search system for a platform with 50 million documents. Users type short queries (2 to 4 words on average) and expect relevant results in under 200ms. The system must handle typos, synonyms, and ambiguous queries.

High-Level Architecture

  User Query: "cheep running shos nike"
              │
              ▼
  ┌───────────────────────────────┐
  │ STAGE 1: QUERY UNDERSTANDING  │  (~5ms)
  │ Spell Correct → Intent → NER  │
  │ Output: "cheap running shoes  │
  │          nike" + Brand=Nike    │
  └───────────────────────────────┘
              │
              ▼
  ┌───────────────────────────────┐
  │ STAGE 2: RETRIEVAL            │  (~10ms)
  │ BM25 + Dense Vector (Hybrid)  │
  │ Output: ~1000 Candidates      │
  └───────────────────────────────┘
              │
              ▼
  ┌───────────────────────────────┐
  │ STAGE 3: RANKING (L2R)        │  (~50ms)
  │ Cross-Encoder or GBDT Ranker  │
  │ Output: ~20 Scored Results    │
  └───────────────────────────────┘
              │
              ▼
  ┌───────────────────────────────┐
  │ STAGE 4: RE-RANKING           │  (~5ms)
  │ Freshness, Diversity, Ads     │
  │ Output: Final SERP Page       │
  └───────────────────────────────┘

Stage 1: Query Understanding

Before retrieving anything, clean up the raw query:

1. Spelling Correction: Fix typos using edit distance or a masked language model ("cheep" becomes "cheap", "shos" becomes "shoes").
2. Intent Classification: Is this navigational (go to a specific page), transactional (buy something), or informational (learn something)?
3. Entity Extraction (NER): Pull out structured attributes like Brand=Nike, Category=Shoes, Color=Red.
4. Query Expansion: Add synonyms ("shoes" also searches for "sneakers" and "footwear").

Stage 2: Retrieval

Run two retrieval methods in parallel and merge results:

1. Lexical (BM25): Fast inverted index lookup. Great at finding exact product names, SKUs, and specific phrases.
2. Semantic (Dense Vector): Encode the query with a bi-encoder, search the document vector index. Catches conceptual matches where words differ but meaning is the same.

Merge results using Reciprocal Rank Fusion (RRF): items that rank high in both lists get the highest combined score.

Stage 3: Ranking

A Learning-to-Rank model scores the 1000 candidates using rich features:

• Query-Document Features: BM25 score, semantic similarity score, query term coverage.
• Document Features: Page authority, freshness, click-through rate history.
• User Features: Past search history, location, device type.

Popular model choices: LambdaMART (gradient boosted trees optimizing NDCG), or a Cross-Encoder Transformer for higher accuracy at the cost of more compute.

Stage 4: Re-Ranking

Apply business logic on top of model scores:

• Remove duplicate or near-duplicate results.
• Boost fresh content for time-sensitive queries ("election results 2024").
• Insert sponsored results at designated positions.
• Enforce diversity so results are not all from the same domain.

Training Data Collection

Search ranking models need relevance labels. Two main sources:

1. Implicit Feedback: Click logs, dwell time (long clicks suggest relevance), query reformulations (user searching again suggests bad results).
2. Human Judgments: Trained raters score query-document pairs on a scale like Perfect (4), Good (3), Fair (2), Bad (1), Terrible (0).

Combine both: use human judgments for model training and implicit signals for online monitoring.

Say this out loud

A search ranking system has four stages. Query Understanding cleans and enriches the raw query. Hybrid Retrieval combines BM25 for exact matches with dense vectors for semantic matches. A Learning-to-Rank model scores candidates using query, document, and user features. Re-ranking applies business rules like freshness and diversity before displaying results.

Followups to expect

1. How do you handle zero-result queries? Progressively relax filters (drop Brand constraint, broaden category), fall back to semantic-only search, or show "did you mean" suggestions.
2. How do you measure search quality online? Track metrics like Mean Reciprocal Rank of clicks, abandonment rate (zero clicks), and reformulation rate. Run A/B tests comparing ranking model versions.