Four-Factor Integrity Breakdown & Primary-Source Evidence
Access Model: Public Plaintext
"An enhanced MMMU benchmark that eliminates shortcuts and guessing strategies to more rigorously test multimodal models across 30 academic disciplines."
"MMMU-Pro introduces a vision-only input setting and increases the number of candidate options from 4 to 10, making it significantly harder for models to rely on guessing or exploiting shortcuts."
Originality
Weight 25%
4/10
Reuse Profile: Medium Reuse (25-75%)
"MMMU-Pro rigorously assesses multimodal models' true understanding and reasoning capabilities through a three-step process based on MMMU: (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images."
Freshness
Weight 20%
0/10
Initial Release: 2024-09Latest Refresh: 2024-09Rolling Updates: No
"arXiv:2409.02813 (cs) [Submitted on 4 Sep 2024"
Grading
Weight 15%
7.3/10
Integrity: Independent StandardizedSandbox: not applicable single turn
"Multiple Choice (10 options) Regex extraction, pass@1"
Frontier Model Standings on MMMU-Pro
Top 15 active models tracked
Benchmark maintainers can embed this live SVG badge in GitHub READMEs or documentation:
[](https://benchaudit.org/benchmarks/mmmu-pro/)