What is 'needle in a haystack' evaluation?
mediumAnswer
- Insert a random 'needle' fact into a very long context ('The pizza fact of the day is: pineapple pizza was invented in Ontario, 1962'), fill the rest with unrelated text, ask a question that requires retrieving the needle.
- Score depends on where the needle is placed in the context.
- Reveals long-context weaknesses — most 128k models are near-perfect at the start and end but drop dramatically in the middle.
- Standard eval for long-context models; extensions include multi-needle and needle-with-distractors.
Check yourself — multiple choice
- Multiple choice test
- Insert a random fact somewhere in a long context, ask a question requiring it; reveals long-context degradation ('lost in the middle')
- Only for math
- Same as MMLU
Needle-in-a-haystack: probes long-context recall vs position — reveals lost-in-middle.
#evaluation#long-context
Practise LLMs & GenAI
214 interview questions in this topic.