SimpleQA

In depth

A factuality benchmark from OpenAI that tests whether a model can answer short, fact-seeking questions that each have one indisputable answer. The questions were collected adversarially against GPT-4, and every response is graded as correct, incorrect, or not attempted. The scoring rewards a model that knows what it knows: the ideal model answers the questions it is sure of and declines the rest instead of guessing. Model cards often report it as a hallucination signal.

Primary source: arxiv.org