Rubric-Based Grading

In depth

Scoring a model's output against an explicit, written rubric that defines what each score level means for each criterion, instead of asking for an open-ended quality rating. Rubrics make human and LLM-as-a-judge grading more consistent and let you score custom criteria like correctness, completeness, or readability separately. The Prometheus paper showed an open 13B evaluator given customized score rubrics reached a 0.897 Pearson correlation with human evaluators, on par with GPT-4.

Primary source: arxiv.org