Evals & Safety
MMMU
A multimodal benchmark of about 11.5K college-level questions drawn from exams, quizzes, and textbooks, where answering requires reading an image as well as the text.
In depth
A multimodal benchmark of about 11.5K college-level questions drawn from exams, quizzes, and textbooks, where answering requires reading an image as well as the text. It covers six disciplines, 30 subjects, and 183 subfields, with images ranging from charts and diagrams to music sheets and chemical structures. At launch in 2023, GPT-4V scored 56% and Gemini Ultra 59%. Vision-capable model releases still quote MMMU or its harder MMMU-Pro variant.
Primary source: arxiv.org
Example
In practice, developers reach for MMMU when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Evals & Safety.
FAQ
Common questions
What is MMMU?
A multimodal benchmark of about 11.5K college-level questions drawn from exams, quizzes, and textbooks, where answering requires reading an image as well as the text.
Why does MMMU matter for AI developers?
MMMU sits in the Evals & Safety part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about MMMU?
Developers Digest publishes tutorials and videos that cover Evals & Safety topics including MMMU. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Evals & Safety
SimpleQA
A factuality benchmark from OpenAI that tests whether a model can answer short, fact-seeking questions that each have one indisputable answer.
Evals & Safety
GDPval
An OpenAI benchmark that measures model performance on economically valuable work, with tasks covering 44 occupations across the 9 sectors that contribute most to U.S.
Evals & Safety
System Card
A system card is a detailed safety and capability report that a frontier lab publishes with a model release, extending the model card idea from the model to the whole deployed system.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever