Sentence Embeddings & Sentence-BERT

The Sentence Comparison Problem

Suppose you want to search a database of $N = 10,000$ customer support questions to find the most similar match for a new user query.

  Naive BERT Pairwise Cross-Encoder Approach:
  Pass: [CLS] Query [SEP] Candidate_1 [SEP] ──► Forward Pass 1
  Pass: [CLS] Query [SEP] Candidate_2 [SEP] ──► Forward Pass 2
  ...
  Total Forward Passes Required: 10,000 passes! (Takes ~5 seconds per query!)

If you want to find the top similar sentence pairs across all 10,000 documents:

$$\text{Required Pairwise Passes} = \frac{10,000 \times 9,999}{2} \approx 50,000,000 \text{ Forward Passes!} \quad (\text{Takes } \sim \mathbf{65 \text{ Hours!}})$$

Standard BERT cross-encoders cannot scale for real-time search!

The Sentence-BERT Solution (Siamese Architecture)

Sentence-BERT (SBERT - Reimers & Gurevych, 2019) introduced a Siamese / Dual-Encoder Architecture:

  OFFLINE PRE-COMPUTATION (Run ONCE):
  Sentence A ──► [ BERT Encoder ] ──► [ Mean Pooling ] ──► Embedding Vector u [1 x 768]
  Sentence B ──► [ BERT Encoder ] ──► [ Mean Pooling ] ──► Embedding Vector v [1 x 768]

  ONLINE REAL-TIME SEARCH (Sub-10ms):
  Cosine Similarity = ( u · v ) / ( ||u|| ||v|| )  ──► Sub-millisecond Matrix Math!

Pre-computing Vectors Offline

SBERT computes a single dense vector embedding $\mathbf{u} \in \mathbb{R}^{768}$ for every document offline and stores them in a Vector Database (Qdrant, Pinecone, FAISS).

When a user submits a query:

1. Compute query vector $\mathbf{q}$ in a single forward pass ($10\text{ms}$).
2. Execute vector search $\text{CosineSim}(\mathbf{q}, \mathbf{u}_i)$ across 10,000 pre-computed vectors in under 1 millisecond!

Training SBERT: Multiple Negatives Ranking Loss (MNRL)

SBERT is trained on Natural Language Inference (NLI) datasets containing (Premise, Positive, Negative) sentence triplets:

  Premise:   "A man is playing a guitar."
  Positive:  "A person is performing music." (Entailment)
  Negative:  "A man is sleeping on a couch." (Contradiction)

Multiple Negatives Ranking Loss (InfoNCE):

$$\mathcal{L} = -\log \frac{\exp(\cos(\mathbf{u}_i, \mathbf{v}_i^+) / \tau)}{\exp(\cos(\mathbf{u}_i, \mathbf{v}_i^+) / \tau) + \sum_j \exp(\cos(\mathbf{u}_i, \mathbf{v}_j^-) / \tau)}$$

This loss pulls semantically equivalent sentence vectors together while pushing contradictory or unrelated sentence vectors far apart.

Say this out loud

Standard BERT requires pairing sentences together, taking 65 hours to compare 10,000 sentence combinations. Sentence-BERT uses a Siamese network architecture with mean pooling to project sentences into single dense vectors offline. Sentence similarity reduces to fast cosine dot products in vector databases, enabling sub-10ms semantic search in RAG pipelines.

Followups to expect

• What is Mean Pooling vs CLS Pooling in SBERT? Mean Pooling averages all token output vectors across the sentence, outperforming the raw [CLS] token vector for sentence level semantic representation.
• What is a Two-Stage Search Pipeline (Retrieve and Rerank)? Stage 1 uses fast Sentence-BERT embeddings to retrieve top 100 candidate documents (sub-10ms). Stage 2 uses a heavy BERT Cross-Encoder to rerank the top 100 candidates for maximum precision.