Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
Provides rigorous benchmarking methodology and performance gaps that inform model selection and development for scientific applications.
AI Summary
This research paper introduces a new for evaluating LLMs' ability to understand scientific visualizations, testing six models on 49 assessment items based on 18 scientific visualizations. The study found significant performance variations, with Gemini exceeding human averages while open-source models lagged behind, particularly struggling with quantitative estimation and texture-based visualizations. The research establishes scientific visualization literacy as a critical dimension for evaluating multimodal AI systems.
Excerpt
Multimodal large language models (MLLMs) are increasingly used to interpret visualizations, yet current evaluations remain largely chart-centric and provide limited evidence of understanding of scientific visualization (SciVis). We benchmark six MLLMs on the scientific visualization literacy assessment test, a standardized SciVis literacy assessment comprising 49 items based on 18 scientific visualizations and illustrations, spanning 8 techniques and 11 task types. We evaluate three closed-sourc
