When you hand someone a picture and a written description of the same thing, the picture is supposed to help. That is the entire premise behind AI systems that can process both images and text: show the model what you are describing, and it should do better.
Two new studies published this month reveal that for the most advanced AI vision systems in the world, the opposite is often true. Show them the picture, and they get worse.
The finding strikes at the heart of one of the biggest claims in AI right now: that modern systems can genuinely reason about what they see. The research suggests they largely cannot. Instead, these systems are running their reasoning almost entirely through language, treating visual information as something to work around rather than use.
The first study, from researchers at Nanyang Technological University and Alibaba Group in Singapore and China, built a benchmark called CrossMath specifically designed to test this. The team took mathematical problems and presented each one in three formats: text only, image only, and a combined version with both the text and the image showing identical information.
Human reviewers checked every problem to confirm that all three versions contained exactly the same facts. Nothing was hidden or changed. The only difference was the format.
Then they ran a battery of leading vision-language AI models through all three versions of every problem.
The result was consistent across every model they tested. Adding the image to the text version of the problem made performance worse, not better. The AI systems were actively harmed by the presence of visual information they should have been able to use.
The researchers described this as a “substantial performance gap” between textual and visual reasoning, and concluded that current AI models do their reasoning primarily in language space, with limited genuine reliance on visual evidence.
The second study, from a separate research team, dug deeper into why this happens and found something more troubling. The team analysed the reasoning behaviour of 18 different vision-language models by tracking their confidence through each step of their reasoning process. What they found was a phenomenon they called “answer inertia”: the models form their conclusions very early in their reasoning process, before they have finished working through the problem, and then spend the rest of their visible reasoning reinforcing that early conclusion rather than revising it.
The team then introduced deliberately misleading text cues alongside clear visual evidence, planting wrong information in the text half of a question while making the correct answer obvious in the image. In case after case, the models followed the misleading text. They ignored what the image was clearly showing them. And critically, the detailed reasoning text the models generated, the step-by-step written explanation that is supposed to show how the AI reached its conclusion, often appeared confident and careful while the model was in the process of ignoring the visual evidence entirely.
This second finding is particularly significant for how AI systems are monitored and trusted. Chain-of-thought reasoning, the technique of asking AI to “show its work” before giving an answer, has become one of the primary tools used by researchers and developers to check whether an AI is actually doing what it appears to be doing. If a reasoning trace can look thorough and visually grounded while the system is actually ignoring the image, that monitoring technique has a serious blind spot.
Together, the two papers point to the same underlying problem. Vision-language AI systems are built from a large language model at their core, with a visual processing component added on top.
The language component is doing most of the cognitive heavy lifting. The visual processing adds noise rather than signal in many cases, because the system has not learned to deeply integrate visual and textual information. It has learned to process them separately and weight language far more heavily.
The implications extend well beyond mathematics problems. AI systems capable of processing both images and text are increasingly being deployed in medical imaging, scientific analysis, and document understanding. Radiology AI tools, pathology assistants, and clinical decision support systems all depend on the premise that when an AI is shown an image alongside written context, it is genuinely using both to reach its conclusion. The CrossMath results do not prove that medical AI systems fail in the same way, but they raise a direct question that anyone deploying them should be able to answer: when your system looks at an image and a text description of the same clinical finding, are you certain it is actually using both?
The researchers behind the CrossMath study say the pattern can be partially corrected by fine-tuning models on a training set specifically designed to force genuine cross-modal reasoning. Models trained this way showed improvements across visual reasoning tasks. But that solution requires deliberately building for it, and most AI vision systems currently in deployment were not.
Both papers are preprints posted to arXiv and have not yet completed peer review.
Sources
Xu, Y., Wang, Y., Wu, Z., Song, K., Lin, J. and Shen, Z. “Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap.” arXiv:2604.16256 [cs.CV / cs.CL]. Submitted 17 April 2026. Nanyang Technological University / Alibaba Group (Tongyi Lab). https://arxiv.org/abs/2604.16256
Sanchez Villegas, D. et al. “Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models.” arXiv:2604.14888 [cs.CL]. Submitted 16 April 2026. https://arxiv.org/abs/2604.14888
CrossMath benchmark code (referenced in the CrossMath paper)
The authors released code at GitHub as part of the submission. The repository is referenced in the paper abstract as available at: https://github.com/xuyige/CrossMath
Wei, J. et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” NeurIPS 2022. https://arxiv.org/abs/2201.11903
More Like This
Sam Brown holds a BSc in Electronic Systems Engineering from University of Regina, a Masters from the University of Alberta and a PhD from the University of Toronto in Electrical Engineering. His reporting interests include Physics and Artificial Intelligence.