# Vision AI in 2025

Peter Robicheaux, Roboflow | AI Engineer World's Fair 2025 | 17:24

Source: https://www.youtube.com/watch?v=IQc05eCvNYE
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/vision-ai-in-2025
Published: 2025-08-03
Tags: benchmarks, edge, embeddings, multimodal

## TL;DR
- Computer vision has different engineering constraints from language models because useful systems need low latency, motion perception, and edge deployment.
- Common vision evaluations such as ImageNet and COCO measure pattern matching too easily, so they do not show whether a model has rich visual features.
- Roboflow's RF-DETR and RF100VL use stronger visual pretraining and harder, varied detection tasks to measure and improve visual generalization.

## Summary
Peter Robicheaux argues that computer vision still falls short of language models in visual detail. Current multimodal models can understand that an image contains a watch or a bus, yet fail at tasks such as reading the time or distinguishing the front from the back of a bus. He connects this weakness to caption-based vision-language pretraining, which often provides no training signal for fine visual differences. Pure vision pretraining, such as DINOv2, learns richer object and part features. Robicheaux also compares convolutional detectors with transformer-based detectors and shows that transformers gain more from large pretraining sets. Roboflow's answer is RF-DETR, a real-time detector using a DINOv2 backbone. He introduces RF100VL, a benchmark built from 100 difficult detection datasets across camera angles, imaging types, and specialized classes. Its few-shot track tests class names, instructions, and visual examples. The results show that small specialist detectors can outperform large vision-language models on visual-domain generalization.

## Key ideas
### Real-world vision needs low latency and edge computation
[00:39](https://www.youtube.com/watch?v=IQc05eCvNYE&t=39s)
Robicheaux says systems that interact with the physical world need vision because the built environment is organized around visual input. Vision also has constraints that differ from language. Motion requires processing multiple frames per second, and sending every frame to one central service introduces too much latency for timely decisions. Useful systems usually need to run at the edge. He places computer vision further behind human vision than computer speech is behind human speech, because models still struggle to extract basic visual details even when they have a broad conceptual understanding of the scene.

### Standard vision evaluations are too easy to measure visual intelligence
[01:50](https://www.youtube.com/watch?v=IQc05eCvNYE&t=110s)
Robicheaux argues that ImageNet and COCO mainly test pattern matching. Models can improve their scores by learning the regularities of those benchmarks without developing rich visual understanding. He uses Claude as an example: it can recognize the general concept of a watch, but it guesses the time instead of locating the hands and reading the numbers. Claude 4 still fails, even though 10:10 is a common default display on watches. For Robicheaux, this shows that strong language reasoning does not guarantee precise visual perception.

### Caption-based vision-language training misses visual differences
[04:15](https://www.youtube.com/watch?v=IQc05eCvNYE&t=255s)
The MMVP dataset pairs images that are close in CLIP space but far apart in DINOv2 space. A model may treat two images as equivalent because their captions would be the same, even when one dog faces the camera and the other faces away. Robicheaux explains that CLIP learns by matching images with captions after both have been shuffled. If the caption does not mention the distinctive pose, the training loss gives the model no reason to separate the images. The resulting visual features are too coarse for fine-grained perception.

### Pure vision pretraining learns object parts without captions
[05:51](https://www.youtube.com/watch?v=IQc05eCvNYE&t=351s)
Robicheaux uses DINOv2 feature visualizations to show what self-supervised vision pretraining can discover from internet images. The model finds the dog mask, its individual segments, and analogous parts across different subjects. Dog legs and human legs occupy related feature regions. He describes the open problem as finding visual features that align with language features while retaining enough visual fidelity for vision-language models. DINOv2 provides an example of the kind of detailed representation that caption-based training often misses.

### Transformer detectors gain more from large pretraining sets
[06:55](https://www.youtube.com/watch?v=IQc05eCvNYE&t=415s)
The comparison between YOLOv8 and LW-DETR illustrates a difference between convolutional and transformer-based detectors. Pretraining YOLOv8 on Objects365 produces almost no COCO improvement in the comparison Robicheaux shows. LW-DETR gains roughly five mean average precision points across the board, and sometimes seven, when pretrained. Objects365 has 1.6 million images, which Robicheaux calls large for image pretraining but small by language-model standards. He presents this as evidence that vision is only beginning to benefit from the kind of large pretraining already common in language.

### RF-DETR brings DINOv2 features to real-time detection
[08:35](https://www.youtube.com/watch?v=IQc05eCvNYE&t=515s)
Roboflow's RF-DETR replaces the LW-DETR backbone with a DINOv2 backbone and uses it for real-time object detection. Robicheaux reports a useful COCO improvement, while saying the model is second to the current leading system on that benchmark. The larger gain appears on RF100VL, where the DINOv2 features improve domain adaptability. His explanation is that COCO contains familiar classes and rewards precise bounding-box refinement, so it is a weak test of whether a detector learned broadly useful visual features.

### RF100VL tests detection across hard domains
[10:00](https://www.youtube.com/watch?v=IQc05eCvNYE&t=600s)
RF100VL combines 100 object-detection datasets selected from Roboflow Universe. The collection includes difficult domains, aerial viewpoints, microscopes, X-rays, and other imaging types that are unlike ordinary COCO images. It also uses specialized classes such as fibrosis and contextual meanings such as a volleyball block or a cable defect called a thunderbolt. A model must understand the class name in the setting where it appears. Robicheaux argues that this tests the richness of learned features more directly than asking only whether a model can find common objects such as dogs and cats.

### Small specialist detectors currently beat large VLMs on visual generalization
[12:45](https://www.youtube.com/watch?v=IQc05eCvNYE&t=765s)
Robicheaux says a COCO-trained or YOLOv8 model trained with about 10 examples per class performs better on RF100VL than Qwen2.5-VL 72B. Large vision-language models generalize well in the linguistic domain, but they remain weak when the distribution changes visually. In the few-shot track, RF100VL supplies a class name, instructions for annotators, and 10 visual examples per class. Grounding DINO scores about 19 mAP zero-shot, while a YOLOv8 nano trained from scratch on the 10-shot examples reaches about 25 mAP. Fine-tuned Grounding DINO is the best current result, but Robicheaux wants models to use all three information sources together.

## Notable quotes
- "My underlying conclusion is vision models aren't smart." (03:17)
- "If your loss function can't tell these two images apart, then why would your model be able to?" (05:35)
- "The vision language models are really good right now at generalizing out of distribution in the linguistic domain but absolutely hopeless when it comes to generalizing in the visual domain." (13:05)
- "You should be able to leverage these annotator instructions, the 10-shot examples, and the class names and come up with something more accurate." (16:54)

## Tools & references mentioned
- Roboflow
- ImageNet
- COCO
- Claude 3.5
- Claude 4
- MMVP
- CLIP
- DINOv2
- LW-DETR
- YOLOv8
- Objects365
- RF-DETR
- RF100VL
- Qwen2.5-VL 72B
- Grounding DINO
- Roboflow Universe
- Nature

## Who should watch
- You are building a vision system that must process video at the edge and need to understand why latency changes the model and deployment choices.
- Your detector performs well on COCO but fails on unusual viewpoints, medical images, industrial defects, or other specialized domains.
- You are evaluating a vision-language model and want a few-shot benchmark that tests visual generalization rather than language-only reasoning.

## Related talks

- [See, Hear, Speak, Draw](https://aietalks.com/talks/see-hear-speak-draw) (Logan Kilpatrick & Simón Fishman, OpenAI, 18:43)
- [How Transformers Finally Ate Vision](https://aietalks.com/talks/how-transformers-finally-ate-vision) (Isaac Robinson, Roboflow, 17:05)
- [FLUX, Open Research, and the Future of Visual AI](https://aietalks.com/talks/flux-open-research-and-the-future-of-visual-ai) (Stephen Batifol, Black Forest Labs, 22:32)
- [The 2025 AI Engineering Report](https://aietalks.com/talks/the-2025-ai-engineering-report) (Barr Yaron, Amplify Partners, 12:33)
- [Moondream: How Does a Tiny Vision Model Slap So Hard?](https://aietalks.com/talks/moondream-how-does-a-tiny-vision-model-slap-so-hard) (Vikhyat Korrapati, 19:26)