Lost in the Middle

The U-Shaped Performance Curve

When AI labs announced 32k, 128k, and 1M token context windows, developers assumed models could read and recall information anywhere inside a massive document.

Nelson Liu et al. (Stanford, 2023) tested this assumption in "Lost in the Middle: How Language Models Use Long Contexts".

They discovered that LLM retrieval accuracy follows a stark U-Shaped Curve:

  Retrieval Accuracy (%)
   100 ┤  \                                                 /
       │   \                                               /
    50 ┤    \                                             /
       │     \________ LOST IN THE MIDDLE! __________/
     0 ┴─────┬──────────────┬──────────────┬─────────────┬──► Information Location in Prompt
           0% (Start)     30%            60%          100% (End)

1. Start of Prompt (0% - 10% Depth): High Accuracy ($\sim 90%$).
2. End of Prompt (90% - 100% Depth): High Accuracy ($\sim 90%$).
3. Middle of Prompt (30% - 70% Depth): Catastrophic Drop ($\sim 30%\text{--}50%$ Accuracy!).

Even state-of-the-art models (GPT-4, Claude, LLaMA) suffered severe performance drops when the key answer passage was buried in the middle of a 20-document context window.

Why "Lost in the Middle" Happens

1. Positional Encoding Bias

Traditional and Rotary Positional Embeddings encode absolute or relative positions. Models receive strong training signals for early tokens (starts of sequences) and late tokens (right before output generation). Middle token positions receive weaker positional gradients.

2. Causal Masking & Attention Accumulation

In Decoder-Only models, later tokens can attend to all preceding tokens. As sequence length grows to 32,000 tokens, Softmax attention probability gets diluted across thousands of middle tokens, while initial system prompt tokens (<|begin_of_text|>) act as Attention Sinks, absorbing disproportionate attention weight.

Mitigation Strategies for Engineers

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. BOUNDARY PLACEMENT    │ 2. CONTEXT TRIMMING      │ 3. RE-ORDER RERANKING    │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Place most important     │ Pass only top 3-5        │ Sort retrieved RAG       │
│ context chunks at the    │ retrieved chunks to the  │ chunks so top relevance  │
│ START or END of prompt.  │ LLM instead of 30.       │ lands at start/end.      │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Re-Order RAG Chunks (Top-at-Boundaries)

When retrieving $K=10$ document chunks from a vector database:

Instead of sorting chunks linearly ($1, 2, 3, 4, 5, 6, 7, 8, 9, 10$):

Sort them so highest-scoring chunks land at boundary positions:

$$\text{Chunk Order: } [\text{Rank 1}, \text{Rank 3}, \text{Rank 5}, \text{Rank 7}, \text{Rank 9}, \text{Rank 10}, \text{Rank 8}, \text{Rank 6}, \text{Rank 4}, \text{Rank 2}]$$

• Rank 1 chunk is placed at the very start of the context.
• Rank 2 chunk is placed at the very end of the context.

2. Limit Context Window Stuffing

Do not pass 30 chunks into an LLM just because the model supports a 128k context window!

Passing fewer, highly-focused chunks (e.g. Top 3 after Cross-Encoder reranking) yields significantly higher factual recall than stuffing 30 noisy chunks into context.

Say this out loud

Lost in the Middle describes how LLMs attend strongly to context boundaries at the start and end of a prompt while missing facts buried in the middle. Performance follows a U shaped curve due to attention sink effects and positional biases. Mitigations include re-ordering RAG chunks so top relevant passages land at start and end boundaries, and limiting prompt context stuffing.

Followups to expect

1. What are Attention Sinks (Xiao et al., 2023 - StreamingLLM)? Initial prompt tokens (like token 0) absorb massive attention scores regardless of semantic relevance, acting as numerical stabilization sinks for Softmax distributions.
2. Does Needle in a Haystack testing catch Lost in the Middle issues? Yes. Needle in a Haystack tests specifically measure retrieval accuracy at different depth percentages (0% to 100%) to verify whether fine-tuning or positional interpolation successfully eliminated Lost in the Middle degradation.