Semantic vs Lexical Search

The Two Foundations of Search

When a user types a query into a search bar, how should the database find relevant documents?

Two distinct paradigms exist:

┌──────────────────────────┬──────────────────────────┐
│ 1. LEXICAL SEARCH        │ 2. SEMANTIC SEARCH       │
├──────────────────────────┼──────────────────────────┤
│ Matches EXACT KEYWORD    │ Matches CONCEPTUAL INTENT│
│ tokens (BM25 / TF-IDF).  │ via Dense Embeddings.    │
│ Sparse Inverted Index.   │ Vector Database ANN.     │
│ Fast, exact SKU matching.│ Synonym & Paraphrase match│
└──────────────────────────┴──────────────────────────┘

1. Lexical Search (Sparse Inverted Index)

Lexical search algorithms (BM25, TF-IDF, ElasticSearch) treat text as a bag of sparse tokens:

$$\text{BM25}(D, Q) = \sum_{i=1}^n \text{IDF}(q_i) \cdot \frac{f(q_i, D) \cdot (k_1 + 1)}{f(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}$$

• How it works: Indexes documents using an Inverted Index (mapping words $\to$ document IDs).
• Strengths:
  • Exact Keyword Precision: Outstanding at finding exact serial numbers (SKU-8942-X), error codes (ERR_CONN_REFUSED), and rare proper names.
  • Ultra-Fast & Cheap: Highly optimized inverted indexes process millions of documents in sub-milliseconds with low RAM footprints.
• Weaknesses:
  • Vocabulary Mismatch: Fails when the query uses different words than the document ("car repair" misses "automobile maintenance").
  • Zero Context Awareness: Ignores word order and grammar semantics.

2. Semantic Search (Dense Vector Embedding)

Semantic search maps text into a continuous $d$-dimensional embedding space using deep neural networks (SBERT, OpenAI Embeddings):

$$\text{Score}(Q, D) = \text{CosineSim}(\text{Embed}(Q), \text{Embed}(D))$$

• How it works: Executes Approximate Nearest Neighbor (ANN) search inside a Vector Database (HNSW).
• Strengths:
  • Concept Matching: Understands synonyms, paraphrases, and intent across languages ("how to fix flat tire" matches "changing a punctured wheel").
  • Context-Aware: Captures overall passage meaning independent of specific keywords.
• Weaknesses:
  • Fails on Exact Strings: Collapses rare part numbers or obscure code names into broad domain vectors, missing exact matches.
  • Higher Resource Cost: Requires GPU embedding computation and large RAM footprints for vector indexes.

Comprehensive Head-to-Head Comparison

The Solution: Hybrid Search

Production search systems do not choose between Lexical and Semantic search.

They execute Hybrid Search: running BM25 and Dense Vector Search in parallel and combining ranks via Reciprocal Rank Fusion (RRF).

Say this out loud

Lexical Search matches exact sparse keyword tokens using BM25 and inverted indexes, excelling at exact part numbers and error codes. Semantic Search maps text into dense embedding spaces to match conceptual intent independent of word choice. Production systems combine both via Hybrid Search to cover exact keyword precision and broad semantic recall.

Followups to expect

1. What is SPLADE (Sparse Neural Search)? A neural model that predicts sparse token weights over full vocabulary dictionaries, combining neural semantic understanding with inverted index BM25 speed.
2. How does Chunking impact Lexical vs Semantic Search? Lexical search scales well on full documents due to TF-IDF length normalization, while Semantic vector search requires smaller text chunks (256 to 512 tokens) to maintain dense vector embedding quality.