Benchmark Saturation

In depth

The point at which top models score so close to a benchmark's ceiling that it can no longer tell them apart. MMLU is the standard example: frontier models passed 90% accuracy, which is why harder sets like Humanity's Last Exam were built. Near the ceiling, the remaining errors are often mislabeled or ambiguous questions rather than real gaps, so a flat leaderboard does not prove progress has stopped. The usual fixes are harder successors, regularly refreshed question pools, and task-based evals tied to real work.

Primary source: arxiv.org