Hybrid Search & Reranking

Combining lexical BM25 keyword matching with dense vector embeddings and cross encoder reranking for production RAG.

🔴 advanced5 min readragretrieval

Hybrid Search and Reranking represent state of the art retrieval pipelines for production RAG systems. Hybrid Search combines sparse lexical keyword search (BM25) with dense semantic vector embeddings (Dense Retrieval) using Reciprocal Rank Fusion (RRF) to capture both exact domain terms and broad conceptual intent. A two stage retrieval pipeline uses fast hybrid search to retrieve top 100 candidate chunks, followed by a heavy Cross Encoder Reranker to compute deep pairwise attention scores, boosting top 5 RAG precision.

Why Single-Method Retrieval Fails

Production RAG pipelines relying on Dense Vector Search alone frequently fail in real-world enterprise environments:

Query: "Find invoice for part number SKU-8942-X" $\to$ Dense vector models fail because embeddings collapse rare serial numbers into generic domain concepts.
Query: "How do I fix error code ERR_CONN_REFUSED?" $\to$ Dense search returns general networking articles rather than exact error code documentation.

Conversely, BM25 Sparse Keyword Search fails on semantic queries lacking exact word overlap ("canine diet advice" misses "dog food manual").

Hybrid Search + Reranking combines sparse, dense, and neural cross-encoder models into a 2-Stage Retrieval Pipeline:

  USER QUERY
      │
      ├─► [ BM25 Sparse Keyword Search ]  ──► Top 100 Keyword Candidates  ──┐
      │                                                                     ├──► [ RECIPROCAL RANK FUSION (RRF) ]
      └─► [ Dense Vector Similarity ]     ──► Top 100 Vector Candidates   ──┘             │
                                                                                          ▼
                                                                           Top 50 Hybrid Candidates
                                                                                          │
                                                                                          ▼
                                                                           [ CROSS ENCODER RERANKER ]
                                                                                          │
                                                                                          ▼
                                                                           Top 5 Precise Context Chunks to LLM!

Stage 1: Hybrid Search (BM25 + Dense Vectors)

Hybrid search runs sparse lexical retrieval and dense vector retrieval in parallel:

BM25: Catches exact product codes, acronyms, names, and rare error strings.
Dense Vector Search: Catches synonyms, conceptual intent, and paraphrases.

Merging Results: Reciprocal Rank Fusion (RRF)

Raw BM25 scores (range $0 \to \infty$) and Cosine Similarity scores (range $-1 \to +1$) cannot be added directly.

Reciprocal Rank Fusion (RRF - Cormack et al., 2009) scores items based purely on Rank Positions:

$$\text{RRF Score}(d \in D) = \sum_{m \in M} \frac{1}{k + r_m(d)}$$

$r_m(d)$: Rank position of document $d$ in search method $m$ (e.g. Rank 1, Rank 2).
$k$: Smoothing constant (typically $k = 60$).

RRF prioritizes documents that rank near the top in both search lists without needing score normalization.

Stage 2: Cross-Encoder Reranking

Bi-encoder dense models compute query and passage embeddings independently for fast vector DB lookups ($O(1)$ speed).

However, independent bi-encoders cannot perform joint attention between query words and passage words.

A Cross-Encoder Reranker (Cohere Rerank, BGE-Reranker) takes the top 50 candidates from Stage 1 and passes each Query + Passage pair together through a heavy BERT-style Transformer:

$$\text{Input: } \text{[CLS] Query [SEP] Candidate Passage [SEP]}$$

The Cross-Encoder computes full cross-attention between every query word and every passage word, producing ultra-precise relevance scores to select the true Top 5 chunks for the final LLM prompt.

┌──────────────────────────┬──────────────────────────┐
│ STAGE 1: HYBRID RETRIEVAL│ STAGE 2: CROSS RERANKING │
├──────────────────────────┼──────────────────────────┤
│ Fast Dual-Encoder + BM25.│ Heavy Cross-Encoder.     │
│ Scans Millions of docs.  │ Rescores Top 50 docs.    │
│ Speed: <5ms              │ Speed: ~30ms             │
│ Goal: High RECALL        │ Goal: High PRECISION     │
└──────────────────────────┴──────────────────────────┘

Say this out loud

Hybrid search combines BM25 keyword matching with dense vector embeddings using Reciprocal Rank Fusion to catch both exact serial terms and broad conceptual intent. A two stage pipeline uses fast hybrid search to retrieve top 100 candidate chunks, followed by a heavy Cross Encoder Reranker that computes full joint self attention to pick the top 5 most precise context chunks for the LLM.

Followups to expect

What is ColBERT (Late Interaction Model)? A hybrid retrieval architecture that stores token-level vector embeddings for passages, performing MaxSim token alignment during search to achieve Cross-Encoder precision at near Bi-Encoder vector speeds.
How much latency does a Reranker add to RAG? Cross-Encoder reranking adds roughly 20ms to 50ms of GPU latency for 50 candidates, which is negligible compared to 1000ms+ LLM text generation latency.

Check yourself

Question 1 of 3

Why does combining BM25 Sparse Keyword Search and Dense Vector Search into a Hybrid Search pipeline outperform pure Dense Vector Search alone?

More in LLMs & GenAI

See all →

Pretraining → SFT → RLHF5 min Fine-Tune vs RAG vs Prompt: Choosing5 min Retrieval-Augmented Generation5 min