An Enhanced MultiModal ReAsoning Benchmark
Can MLLMs Reason in Multimodality?
*Equal contribution
*Equal contribution
🔥 [2025-03-02]: Exciting updates on the Leaderboard! We've added claude-3-7-sonnet-20250219 and kimi-k1.5-Preview performances. Currently, claude-3-7-sonnet-20250219🥇, Gemini-2.0-Flash-Thinking-exp-0121🥈, and o1🥉 on EMMA-Mini.
🔥 [2025-01-28]: We've added Gemini-2.0-Flash-Thinking-0121 and QVQ-72B-Preview performances on the Leaderboard!
🚀 [2025-01-09]: We released EMMA, a benchmark for advanced multimodal reasoning. 🥳The ability to organically reason over and with both text and images is a pillar of human intelligence, yet the ability of Multimodal Large Language Models (MLLMs) to perform such multimodal reasoning remains under-explored. Existing benchmarks often emphasize text-dominant reasoning or rely on shallow visual cues, failing to adequately assess integrated visual and textual reasoning.
We introduce , a benchmark targeting organic multimodal reasoning across mathematics, physics, chemistry, and coding. EMMA tasks demand advanced cross-modal reasoning that cannot be addressed by reasoning independently in each modality, offering an enhanced test suite for MLLMs' reasoning capabilities.
Our evaluation of state-of-the-art MLLMs on EMMA reveals significant limitations in handling complex multimodal and multi-step reasoning tasks, with even advanced techniques like Chain-of-Thought prompting and test-time compute scaling underperforming. These findings underscore the need for improved multimodal architectures and training paradigms to close the gap between human and model reasoning in multimodality.
| Reset | EMMA | EMMA-Mini | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Name | Size | CoT prompting | Overall | Math | Physics | Chemistry | Coding | Overall | Math | Physics | Chemistry | Coding |
| human expert | - | - | - | - | - | - | - | 77.75 | 75.00 | 64.50 | 86.00 | 85.50 |
| claude-3-7-sonnet-20250219 | - | - | - | - | - | - | - | 56.50 | 45.00 | 65.00 | 63.00 | 53.00 |
| gemini-2.0-flash-thinking-exp-01-21 | - | - | 42.50 | 37.11 | 60.26 | 41.58 | 48.05 | 48.00 | 34.00 | 63.00 | 47.00 | 48.00 |
| o1 | - | - | - | - | - | - | - | 45.75 | 41.00 | 49.00 | 40.00 | 53.00 |
| gemini-2.0-flash-thinking-exp-1219 | - | - | 38.06 | 31.61 | 56.41 | 37.93 | 43.44 | 43.50 | 35.00 | 57.00 | 41.00 | 41.00 |
| qwen2-vl-72b-instruct | 72B | ✗ | 33.46 | 33.07 | 42.31 | 32.06 | 34.57 | 37.25 | 38.00 | 40.00 | 34.00 | 37.00 |
| claude 3.5 sonnet | - | ✓ | 37.23 | 29.37 | 41.03 | 41.07 | 40.60 | 37.00 | 30.00 | 38.00 | 41.00 | 39.00 |
| gemini 2.0 flash | - | ✓ | 29.12 | 25.90 | 38.46 | 24.66 | 40.96 | 36.25 | 24.00 | 41.00 | 36.00 | 44.00 |
| gpt-4o | - | ✓ | 32.71 | 25.56 | 43.59 | 33.67 | 39.01 | 36.00 | 27.00 | 44.00 | 35.00 | 38.00 |
| gpt-4o | - | ✗ | 32.42 | 27.24 | 38.46 | 31.89 | 40.07 | 35.25 | 30.00 | 38.00 | 33.00 | 40.00 |
| internvl2.5-78b | 78B | ✗ | 33.50 | 31.39 | 38.46 | 35.20 | 31.91 | 35.25 | 30.00 | 40.00 | 38.00 | 33.00 |
| gemini 2.0 flash | - | ✗ | 33.61 | 23.88 | 38.46 | 36.31 | 42.02 | 34.25 | 20.00 | 40.00 | 36.00 | 41.00 |
| claude 3.5 sonnet | - | ✗ | 35.08 | 25.34 | 33.97 | 40.90 | 38.65 | 34.00 | 23.00 | 34.00 | 44.00 | 35.00 |
| kimi-k1.5-preview | - | - | - | - | - | - | - | 33.75 | 31.00 | 47.00 | 26.00 | 31.00 |
| QVQ-72B-Preview | 72B | - | - | - | - | - | - | 32.00 | 34.00 | 39.00 | 24.00 | 31.00 |
| qwen2-vl-72b-instruct | 72B | ✓ | 27.12 | 27.69 | 34.62 | 24.57 | 29.43 | 31.00 | 35.00 | 34.00 | 32.00 | 23.00 |
| internvl2-llama3-76b | 76B | ✓ | 23.35 | 22.20 | 32.05 | 19.73 | 30.32 | 28.25 | 27.00 | 33.00 | 21.00 | 32.00 |
| internvl2.5-78b | 78B | ✓ | 27.08 | 25.56 | 39.74 | 27.47 | 25.18 | 27.50 | 31.00 | 36.00 | 24.00 | 19.00 |
| llava-onevision-72b | 72B | ✗ | 27.33 | 27.69 | 35.90 | 25.26 | 28.72 | 27.25 | 25.00 | 32.00 | 24.00 | 28.00 |
| internvl2-llama3-76b | 76B | ✗ | 25.07 | 25.11 | 22.44 | 24.06 | 27.84 | 25.50 | 31.00 | 22.00 | 21.00 | 28.00 |
| llava-onevision-72b | 72B | ✓ | 23.82 | 22.42 | 15.38 | 22.70 | 30.67 | 25.25 | 23.00 | 26.00 | 23.00 | 29.00 |
| random choice | - | - | 18.08 | 14.01 | 25.64 | 16.50 | 25.71 | 22.75 | 13.00 | 23.00 | 27.00 | 28.00 |
Overall results of different models on the EMMA leaderboard. The best-performing model in each category is in-bold, and the second best is underlined.
EMMA is composed of 2,788 problems , of which 1,796 are newly constructed , across four domains: math, physics, chemistry, and coding. To provide fine-grained insights into how MLLMs might fail in multimodal reasoning, we assign labels to each problem in our benchmark. These labels are either created by domain experts or assigned by GPT-4o and subsequently verified by experts. As shown in Category Figure, questions in EMMA assess a wide array of multimodal reasoning skills. For example, the pattern inference problem in math challenges models to identify and generalize visual patterns; the visual decomposition simulation problem in physics requires graphically decomposing forces to determine resultant effects; the reaction simulation problem in chemistry demands precise interpretation and simulation of electron movement; the 3D visualization problem in coding.
We also try various test-time compute scaling strategies. While they tend to boost model performance, they are far from enough to close the gap to human-level performance. The best model and scaling strategy configuration we try still trails humans by 27%.
Using the abels for each question based on the multimodal skills it assesses, we find that CoT prompting hurts performance on visual-reasoning-heavy tasks, while it benefits closed-source models on tasks where textual CoT is theoretically useful.
Distribution of error types made by o1 on the math and coding portions of EMMA-mini. The majority of errors arise in visual reasoning.
Math Case 1
Math Case 2
Physics Case 1
Physics Case 2
Chemistry Case 1
Physics Case 3
Coding Case 1
Coding Case 2
Physics Case 4
Physics Case 5
Physics Case 6
Math Case 3
Math Case 4
Math Case 5
Math Case 6
Math Case 7
More examples detailed in the paper.
Click on the image to display the answer. Please check complete Coding questions here.
@article{hao2025can,
title={Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark},
author={Hao, Yunzhuo and Gu, Jiawei and Wang, Huichen Will and Li, Linjie and Yang, Zhengyuan and Wang, Lijuan and Cheng, Yu},
journal={arXiv preprint arXiv:2501.05444},
year={2025}
}
