TruthfulQA

In depth

A benchmark of 817 questions across 38 categories, including health, law, finance, and politics, written so that a common human misconception produces a confident wrong answer. It tests whether a model repeats false beliefs it absorbed from training text. In the original 2021 study, the best model was truthful on 58% of questions against 94% for humans, and larger models were generally less truthful.

Primary source: arxiv.org