Prompt Caching

Reusing pre-computed KV Caches across repeated prompt prefixes to reduce LLM latency and API costs by up to 90%.

🟡 intermediate4 min readinferencepractical

Prompt Caching avoids re-computing the Transformer prefill phase for shared or static prompt prefixes (e.g. system instructions, long document contexts, multi-turn chat history). By storing the computed KV Cache of a static prefix in server memory, subsequent user queries matching that prefix bypass prompt prefill computation. Prompt Caching reduces Time-To-First-Token (TTFT) latency from seconds to milliseconds and lowers API token input costs by up to 80-90%.

The Cost of Repeated Prefills

In multi-turn chat, RAG, or system-prompt applications:

Request 1: [ 10,000 Token System Prompt + Legal Document ] + [ User Query 1 ]
           └──► Must compute Prefill for 10,000 tokens! (TTFT = 2.5s)

Request 2: [ 10,000 Token System Prompt + Legal Document ] + [ User Query 2 ]
           └──► WITHOUT CACHING: Re-computes 10,000 token prefill again! (TTFT = 2.5s)
           └──► WITH PROMPT CACHING: Fetches KV Cache from RAM! (TTFT = 0.05s!)

                        PROMPT CACHING ARCHITECTURE
  Prefix Match: "System Prompt + Doc Context" (Indices 0..10000)
                              │
            ┌─────────────────┴─────────────────┐
            ▼                                   ▼
  [ KV CACHE HIT ]                     [ UN-CACHED SUFFIX ]
  Reuse pre-computed K, V tensors       Compute prefill for User Query only
  (TTFT = 50ms, 90% Cost Discount)      (Fast 50-token execution)

RadixAttention & Prefix Sharing (SGLang / vLLM)

Instead of manual caching rules, modern LLM serving engines maintain a Radix Tree mapping token sequences to physical KV Cache pages in VRAM/DRAM:

                       [ Root ]
                          │
             "System: You are an AI assistant..."
                          │
                          ▼
                   [ Node A (KV Cache) ]
                  /                     \
       "Doc 1: Policy PDF..."        "Doc 2: Legal Code..."
                │                               │
                ▼                               ▼
       [ Node B (KV Cache) ]           [ Node C (KV Cache) ]

When a new request arrives, SGLang traverses the Radix Tree, identifying the longest matching prefix node and reusing its cached KV pages instantly.

Primary Use Cases

Multi-Turn Conversational Chat: Retain KV cache of previous dialogue turns ($T_{1 \dots k}$) so turn $k+1$ evaluates only new user tokens.
RAG / Enterprise Document QA: Cache large reference documents (50k tokens) once; run hundreds of fast user queries against the cached document.
Agentic System Prompts & Tool Definitions: Cache heavy system instructions and JSON Schemas across all incoming user sessions.

Say this out loud

"Prompt Caching reuses pre-computed KV Cache tensors for static prompt prefixes like system instructions, RAG context, or chat history. By fetching cached KV tensors from memory, inference engines bypass the heavy O(N²) prefill pass, cutting Time-To-First-Token (TTFT) latency from seconds to milliseconds and reducing API token costs by up to 90%."

Follow-ups to expect

What is the minimum prefix length for Prompt Caching to be cost-effective? Cloud providers (Anthropic, OpenAI) typically enforce a minimum prefix threshold of 1,024 tokens. Below 1,024 tokens, KV cache lookup and memory management overhead exceed prefill compute savings.
How does cache eviction work in Prompt Caching? Serving engines use LRU (Least Recently Used) page eviction over the Radix Tree, flushing inactive KV cache pages to CPU host RAM or disk when GPU VRAM approaches capacity.

Check yourself

Question 1 of 3

Why does Prompt Caching dramatically reduce Time-To-First-Token (TTFT) for long-context prompts?

More in LLMs & GenAI

See all →

Pretraining → SFT → RLHF5 min Fine-Tune vs RAG vs Prompt: Choosing5 min Retrieval-Augmented Generation5 min