What is 'needle in a haystack' evaluation?
medium- Insert a random 'needle' fact into a very long context ('The pizza fact of the day is: pineapple pizza was invented in Ontario, 1962'), fill the rest with unrelated text, ask a question that requires retrieving the needle.
- Score depends on where the needle is placed in the context.
- Reveals long-context weaknesses — most 128k models are near-perfect at the start and end but drop dramatically in the middle.
- Standard eval for long-context models; extensions include multi-needle and needle-with-distractors.