Context Windows & Long-Context Tricks

The Context Budget Challenge

Modern Large Language Models support massive context windows (32k, 128k, up to 1M+ tokens in Gemini 1.5 Pro).

However, filling maximum context windows on every API call introduces severe engineering drawbacks:

1. High Financial Cost: API billing scales linearly with input token count. Passing 100,000 tokens per call gets expensive fast.
2. High Latency (Time-to-First-Token / TTFT): Processing 100k input tokens through self-attention layers takes seconds before output generation starts.
3. Lost-in-the-Middle Degradation: Models attend strongly to context boundaries (start and end) while missing details buried in the middle of massive prompts.

Context Window Management optimizes prompt assembly to maximize accuracy while minimizing token overhead.

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. SLIDING WINDOW        │ 2. DYNAMIC SUMMARIZATION │ 3. PROMPT CACHING        │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Retains system prompt +  │ Summarizes old chat turns│ Caches pre-computed KV   │
│ most recent N turns.     │ into a compact summary   │ matrices for common      │
│ Drops oldest chat history│ block appended to prompt.│ system prompt prefixes.  │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Production Context Management Strategies

1. Sliding Window (FIFO Truncation)

Keep the static System Prompt at the top, and maintain a First-In-First-Out (FIFO) sliding window over recent conversation turns:

[ SYSTEM PROMPT (Fixed) ]
[ USER TURN 10 ] [ ASSISTANT TURN 10 ]
[ USER TURN 11 ] [ ASSISTANT TURN 11 ]  <-- Old turns 1 to 9 dropped!

2. Dynamic Conversation Summarization

When conversation history crosses a token threshold (e.g. 4,000 tokens):

1. Send older turns 1 to 8 to a fast small model (e.g. Claude Haiku or GPT-4o-mini).
2. Generate a compact 100-token summary paragraph of past background context.
3. Replace turns 1 to 8 with the Summary Block:

[ SYSTEM PROMPT ]
[ BACKGROUND SUMMARY: User is building a Python web app using FastAPI... ]
[ RECENT TURN 9 ] [ RECENT TURN 10 ]

3. Prompt Caching (KV Cache Prefixes)

Providers (Anthropic, OpenAI, DeepSeek) offer Prompt Caching:

If sequential API requests share an identical prompt prefix (e.g. a 20,000-token system prompt or reference PDF):

• The API provider caches pre-computed Key and Value tensors on GPU VRAM.
• Subsequent requests reuse the cached KV state, cutting TTFT latency by 80% and input token costs by 50% to 90%!

Evaluating Long-Context Performance: Needle in a Haystack

How do you test whether a model actually reads its entire 128k context window?

The Needle in a Haystack (NIAH) test places a specific random fact (the "needle", e.g. "The secret passcode is 8942") at various depth percentages (0% to 100%) inside a massive filler document (the "haystack", e.g. 100,000 tokens of essay text).

The test measures retrieval accuracy across different context lengths and needle depth locations, visualizing model retrieval limits as a heat map.

Say this out loud

Context Window Management optimizes prompt token budgets. Sliding window strategies retain recent turns while dynamic summarization compresses old chat history. Prompt Caching reuses pre-computed KV matrices for fixed system prompt prefixes, cutting API costs and latency. Needle in a Haystack tests evaluate retrieval performance across context depths.

Followups to expect

1. What is RAG-based Memory Management for Agents? Storing past conversation turns and user preferences as vector embeddings, retrieving only the top 3 relevant past memories when a user query requires historical context.
2. How does Prompt Caching impact system prompt design? System prompts and static context documents should be placed at the very top of the prompt string to maximize prefix matching efficiency for KV prompt caching.