Why does stuffing more context into a long-context model sometimes make answers worse?
hardAnswer
- Attention is spread over everything you supply, so irrelevant passages actively compete with the relevant one.
- Models also attend unevenly across position: information in the middle of a very long context is used less reliably than material at the start or end, the 'lost in the middle' effect.
- More context raises cost and latency roughly with length, and it increases the chance of contradictory passages that the model must silently arbitrate.
- The practical consequence is that a tight, reranked top-5 usually beats an unfiltered top-50, and that retrieval quality still matters even with a million-token window.
Check yourself — multiple choice
- More context is always better
- Irrelevant passages compete for attention, middle positions are used less reliably, and cost rises — a reranked top-5 often beats an unfiltered top-50
- Long context models ignore prompts
- Only cost changes
Distraction and positional unreliability mean retrieval quality matters even with huge windows.
#long-context#rag
Practise LLMs & GenAI
214 interview questions in this topic.