Needle in a Haystack

In depth

A long-context test that hides a specific fact (the needle) inside a long filler document (the haystack) and asks the model to retrieve it. Greg Kamradt's open source harness popularized it, sweeping context length and needle depth from 0 to 100 percent and scoring each combination. Many models now score near perfectly on the single-needle version, so benchmarks like RULER add multiple needles, multi-hop tracing, and aggregation to expose weaknesses it misses.

Primary source: github.com