Frontier models often recognize visual patterns correctly while failing at counting, spatial relationships, and state changes across video.
2
Many multimodal benchmarks make visual reasoning look stronger than it is because their images are tiny or unnecessary for answering the question.
3
Elorian is building visual reasoning models with new data, a training flywheel, transformer changes, and native visual chain of thought.
Summary
Andrew Dai argues that current frontier models are good at visual recognition but weak at visual reasoning. They identify a chessboard and infer that it must contain 32 white squares, miscount Catan roads by using pieces left beside the board, and lose track of a robot arm lifting a lid during a video. Dai separates understanding from reasoning with a simple test: if a person can answer after one second, the model can usually do it; questions that require sustained visual attention expose failures. He criticizes benchmarks such as ARC-AGI and MMU because their images can be too small or unnecessary. Elorian's proposed system uses visual reasoning data, a synthetic training loop involving evals and agents, changes to the transformer architecture, and visual chain of thought. Dai connects this work to robotics, construction safety, and mechanical design, where models must reason over video, policy text, blueprints, CAD, and physical constraints.
Pattern matching makes models guess the complete object
Dai shows a partial chessboard and asks how many white squares are visible. Frontier models answer 32 because they recognize a chessboard and supply the standard fact that a full board has 32 white squares. The model does not count the visible squares. Dai says the same pattern matching helps models identify plants, animals, and flowers, but hurts them when a question requires detailed inspection of the actual image.
Spatial counting exposes a lack of visual grounding
Dai uses board games to show that models struggle with simple spatial questions. He says frontier models would fail to report where randomly placed pieces are located on a chessboard. In a Catan example, a model counted blue roads that were off the board and inferred that five remained on the board. The correct count was seven. The model reasoned from a familiar pattern instead of counting the pieces in front of it.
Long videos cause models to lose the state of an interaction
In a video of a robot arm handling a cooker, current models miss that the arm lifted the lid and later miss that it turned on the stove. Dai calls this context amnesia. The models cannot maintain consistency across a long video, so they lose track of actions that changed the scene. This matters for robots because a useful visual system must follow what has happened, not only label what appears in one frame.
The one-second test separates recognition from reasoning
Dai proposes asking how long a person would need to answer a visual question. Identifying a game, naming a flower, or counting three chess pieces can take less than a second, and frontier models usually handle those tasks. Counting or answering a more complex question needs longer inspection. Dai associates that sustained process with visual reasoning, or visual thinking, and says this is where current frontier models fall apart.
Popular multimodal benchmarks can avoid the hard visual work
Dai questions what some visual reasoning evaluations measure. He points out that ARC-AGI uses images that are only 32 by 32 or 64 by 64 pixels, which makes it difficult to connect high scores with complex real-world visual tasks. He also says that many MMU questions can be answered without looking at the image, or with only a rough recognition of what it contains. He calls for benchmarks built around geometric alignment, spatial intelligence, and object terminology.
Visual AI needs thinking alongside generation and detection
Dai says image and video generators can produce high-fidelity scenes without physical grounding or causal logic. Explosions and collapsing buildings often look like Hollywood footage or game-engine output because those sources dominate the training data. Detection systems such as Google Lens, SAM 3, YOLO, and Mask R-CNN can map pixels to labels, but Dai describes them as passive. A thinking visual model would reason over space and time, then use that information for planning and physical action.
Elorian combines new data with native visual chain of thought
Elorian's approach has four parts. The team is collecting and generating visual reasoning data that Dai says cannot be found online. It uses a synthetic data flywheel involving evals, agents, supervised fine-tuning, and reinforcement learning. It is also changing the transformer architecture. The fourth part is visual chain of thought. For a question about red hotels, the model first draws boxes around hotels and other objects, then narrows the result to the red hotels in the image.
The proposed uses require language and vision to work together
Dai describes robotics, construction, and mechanical design as target areas. Robots need to act in changing physical environments, while construction systems must connect camera footage with zone-specific safety rules and OSHA policy language. Mechanical design requires detailed reasoning over blueprints, 3D CAD, and CAM files. Dai says companies have tried frontier models for these tasks, but the models produce errors because they struggle with visual context and spatial constraints.
Mechanical design needs simulation and geometric validation
Dai compares mechanical design with software development. Code can be checked with unit tests, while mechanical devices can be run through simulators to test whether a design works in the real world. He says this kind of validation could support multimodal reasoning over geometric logic. He also gives an example from a mechanical engineering company, where designing one small part of a robot testing platform took 100 to 200 hours, while the full platform took 2,000 to 3,000 hours of human time.
"So that is the distinction that we make between what is understanding, what is pattern recognition, versus what is reasoning."05:10
Who should watch
You are building a multimodal system and need a practical way to distinguish image recognition from visual reasoning.
You are choosing or interpreting multimodal benchmarks and want to know why high scores may not reflect detailed spatial reasoning.
You work with robots, construction footage, CAD, or mechanical design, where models must track state and connect visual information with language and physical constraints.