Semantic Caching
Intercepting semantically equivalent LLM prompts to deliver sub-10ms responses and slash API token costs.
Semantic Caching (GPTCache, Redis Semantic Cache) stores prompt embeddings and their corresponding LLM responses in a fast Vector DB. Traditional exact-string caching fails when users paraphrase prompts ("What is the weather in NYC?" vs "How's the weather in New York City?"). Semantic Caching embeds incoming user prompts, computes cosine similarity against cached prompt vectors, and returns the pre-computed response instantly if similarity exceeds threshold τ (e.g. τ ≥ 0.95), bypassing LLM inference.
Exact-String Caching vs Semantic Caching
TRADITIONAL EXACT-STRING CACHE:
Cached Key: "What is the capital of France?" ──► Response: "Paris"
Query: "Tell me the capital of France" ──► MISS! (Different bytes, executes full LLM call)
SEMANTIC CACHE (Vector Embedding Search):
Cached Vector: Embed("What is the capital of France?")
Query Vector: Embed("Tell me the capital of France")
Cosine Similarity = 0.978 ≥ 0.95 Threshold ──► HIT! (Sub-10ms response, zero LLM API cost!)
The Semantic Cache Architecture
User Query ──► [ Text Embedding Model ] ──► Query Vector v_user
│
▼
[ VECTOR CACHE STORE (Redis) ]
CosSim(v_user, v_cached)
│
┌───────────────────────────────┴───────────────────────────────┐
▼ ▼
Similarity ≥ 0.95 (CACHE HIT) Similarity < 0.95 (CACHE MISS)
Return Cached LLM Response Execute Full LLM Generation
(Latency < 10ms, Cost = $0) Save (v_user, Response) to Cache!
Tuning Threshold $\tau$
Similarity Threshold τ
0.70 ───────────────► High Cache Hit Rate (Risk: Serves wrong answers to distinct questions!)
0.95 ───────────────► Optimal Sweet Spot (High hits for true paraphrases; zero false positives)
0.99 ───────────────► Low Cache Hit Rate (Behaves almost like exact-string matching)
Eviction & TTL Policies
- Time-To-Live (TTL): Cache entries expire after 24–72 hours to prevent stale responses for temporal queries.
- Metadata Filtering: Include tenant ID or user role in cache lookup to prevent cross-user data leakage.
- Invalidation Hooks: Clear specific cache clusters when underlying database facts or RAG documents are updated.
Say this out loud
"Semantic Caching stores prompt embeddings and pre-computed responses in a fast Vector DB. Unlike exact-string caching, it recognizes paraphrases: if an incoming query vector has cosine similarity ≥ 0.95 with a cached prompt, it serves the cached response in < 10ms with zero LLM API cost, while routing misses to the LLM."
Follow-ups to expect
- What is GPTCache? An open-source Python framework specifically designed to build semantic caches for LLM applications, integrating with Redis, Milvus, Qdrant, and LangChain.
- How do you prevent cache pollution from multi-turn chat contexts? Include conversation history hash or system prompt ID alongside prompt embeddings in the vector metadata payload, isolating context-dependent conversations.
Check yourself
Question 1 of 3
Why does traditional exact-string caching (e.g. Redis key-value cache on prompt string) fail for production conversational LLM applications?