# How Transformers Finally Ate Vision

Isaac Robinson, Roboflow | AI Engineer Europe 2026 | 17:05

Source: https://www.youtube.com/watch?v=VhfAVA3BG2I
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-transformers-finally-ate-vision
Published: 2026-05-08
Tags: inference, multimodal

## TL;DR
- Vision transformers overcome their weak image inductive bias through large-scale, transformer-specific pretraining such as MAE and DINO.
- Infrastructure developed for language models, especially FlashAttention, removes much of the practical speed gap between transformer and convolutional vision models.
- Foundation models still need deployment flexibility, so Roboflow uses transfer evaluation and neural architecture search to produce models for different hardware and tasks.

## Summary
Isaac Robinson traces the competition between convolutional networks and vision transformers. Convolutions begin with a useful image prior: the same pattern can be detected wherever it appears. Vision transformers discard that prior, split images into patches, and have expensive attention. They still win because large-scale pretraining can learn much of the missing structure, while tools created for large language models improve transformer speed. Robinson follows this path through ViT, Swin, ConvNeXt, Hiera, and the SAM family. He argues that architecture alone does not decide performance. Pretraining and hardware-aware deployment matter just as much. Large foundation models such as SAM 3 are powerful but can be too expensive for edge use. Roboflow's RF-DETR work adds flexible model choices through transfer evaluation and neural architecture search. The talk ends with video models, JEPA, V-JEPA, and the open question of how these systems will support VLMs, world models, and physical AI.

## Key ideas
### Convolutional networks start with an image-specific prior
[00:36](https://www.youtube.com/watch?v=VhfAVA3BG2I&t=36s)
Robinson describes convolutional neural networks as having an inductive bias inspired by the eye. A filter is applied across an image, so an activation can respond to the same object or pattern wherever it appears. A person in the upper-left and a person in the bottom-right can produce the same kind of response. Networks such as ResNets build hierarchical structures from this operation. This locality and translation behavior made convolutional networks the standard approach to vision for a long time.

### Vision transformers trade image priors for flexible token mixing
[01:19](https://www.youtube.com/watch?v=VhfAVA3BG2I&t=79s)
A transformer begins with a set of tokens and applies a set-to-set operation without an inherent image-specific bias. For vision, ViT splits an image into patches, originally 16 by 16, adds learned positional encodings, and feeds the sequence into a transformer. Robinson contrasts the quadratic token operation with the much worse resolution scaling that results when attention is applied to image patches. A location in the upper-left does not have to behave like the same location in the bottom-right. This makes the architecture look like an unlikely choice for images.

### Swin restores locality with shifted attention windows
[03:33](https://www.youtube.com/watch?v=VhfAVA3BG2I&t=213s)
Swin Transformer limits attention to local windows instead of allowing every patch to interact at once. That reduces the scaling cost when the window size stays independent of image resolution. A problem remains: tokens in separate windows cannot communicate. Swin shifts the windows in the next layer, creating overlapping regions so information can cross the original boundaries. Robinson says this starts to resemble convolution, where filters operate locally and overlap across positions. Swin therefore adds a locality inductive bias while keeping the transformer structure.

### ConvNeXt shows that transformer design ideas can improve convolutional networks
[05:21](https://www.youtube.com/watch?v=VhfAVA3BG2I&t=321s)
ConvNeXt takes lessons from vision transformers and applies them to a convolutional architecture. It uses a patchify-style operation, replaces attention's spatial mixing with convolution, follows a mixer and feed-forward pattern, and keeps the hierarchical structure common in convolutional models. It also adopts layer normalization and other transformer-era changes. Robinson says ConvNeXt beats ViT and Swin on the standard ImageNet reference. Its weakness is speed, which leads to a closer examination of which inductive biases are actually needed.

### Pretraining lets ViTs learn image structure instead of building it into the architecture
[06:50](https://www.youtube.com/watch?v=VhfAVA3BG2I&t=410s)
The next step removes inductive biases from a well-designed model and uses pretraining to recover them. Robinson explains masked autoencoding as dropping image patches and asking the model to reconstruct them from context, similar to BERT's language objective. This technique is specific to ViTs because the dropped patches do not fit naturally into a convolution that is invariant across locations. At larger scale, methods such as DINOv2 and DINOv3 produce rich feature maps. Robinson gives examples where PCA visualizations separate cat paws and organize satellite imagery semantically, with frozen features and a linear probe approaching supervised results.

### Language-model infrastructure closes much of the transformer speed gap
[09:33](https://www.youtube.com/watch?v=VhfAVA3BG2I&t=573s)
The remaining concern is the cost of attention, especially its resolution scaling. Robinson points to FlashAttention as a major optimization borrowed from the large-language-model ecosystem. He discusses Hiera, which reported a speedup at similar accuracy compared with ViT, but also noted that its comparison did not use FlashAttention. Adding FlashAttention changes the comparison, reducing the practical importance of the architectural speedup. His conclusion is that a heavily pretrained ViT can retain its benefits while using infrastructure developed for language models.

### The SAM family follows the same return to a heavily pretrained ViT
[10:24](https://www.youtube.com/watch?v=VhfAVA3BG2I&t=624s)
Robinson uses the Segment Anything family as an application-level example. SAM uses a ViT backbone trained with MAE. MobileSAM replaces it with a specialized convolutional-transformer hybrid called TinyViT, and SAM 2 uses Hiera with MAE pretraining. SAM 3 returns to the simpler strategy: use a massively pretrained backbone rather than continue architecture ablations. Robinson presents this as the same pattern seen in backbone research, where increasingly strong pretraining makes the original transformer architecture difficult to beat.

### Foundation models need hardware and task flexibility to be useful in deployment
[11:33](https://www.youtube.com/watch?v=VhfAVA3BG2I&t=693s)
Large pretraining is expensive, and a foundation model that only comes in one size can be difficult to deploy. Robinson gives SAM 3 as an example, with 800 million parameters and a 300-millisecond runtime on a T4 GPU. That limits its use in settings built around low-power edge devices. Roboflow's RF100VL dataset measures how well foundation models transfer to diverse downstream object-detection tasks. Robinson says the RF-DETR approach achieves about a 40x speedup at the same accuracy as fine-tuning SAM 3, while a 15x speedup gives a meaningful improvement.

### Neural architecture search turns one foundation model into a deployable model family
[13:44](https://www.youtube.com/watch?v=VhfAVA3BG2I&t=824s)
Roboflow modifies a foundation model with neural architecture search to generate a family of high-performance models. The approach introduces flexible choices that remain compatible with existing foundation-model infrastructure. Those choices can be mixed according to the target data and target hardware. Robinson says this addresses the lack of deployment flexibility in large foundation models. He summarizes the recipe as massive ViT-specific pretraining, speedups from language-model infrastructure, and pretraining-compatible neural architecture search.

## Notable quotes
- "So, how is this possible? And I'm going to make the argument that it is because of massive VIT-specific pretraining, and then we get to borrow a lot of speedups and infrastructure from the fact that LLMs are blowing up." (03:05)
- "You do this at scale, and it turns out the model actually learns back the inductive biases." (07:47)
- "So, massive VIT specific pre-training plus speed ups from LLMs plus pre-training compatible neural architecture search, and that's it." (14:47)
- "SAM 3 is this very very powerful thing, but is also 800 million parameters." (11:58)

## Tools & references mentioned
- Roboflow
- ResNet
- Vision Transformer (ViT)
- Swin Transformer
- ConvNeXt
- Hiera
- MAE
- BERT
- DINOv2
- DINOv3
- FlashAttention
- Segment Anything (SAM)
- MobileSAM
- SAM 2
- SAM 3
- RF100VL
- RF-DETR
- JEPA
- V-JEPA

## Who should watch
- You are choosing between convolutional and transformer backbones and want the reasoning behind ViT's eventual advantage.
- Your vision model works in the lab but does not fit the latency, power, or hardware limits of its deployment target.
- You are building foundation-model transfer or neural architecture search systems for detection, segmentation, or video.

## Related talks

- [Vision AI in 2025](https://aietalks.com/talks/vision-ai-in-2025) (Peter Robicheaux, Roboflow, 17:24)
- [Moondream: How Does a Tiny Vision Model Slap So Hard?](https://aietalks.com/talks/moondream-how-does-a-tiny-vision-model-slap-so-hard) (Vikhyat Korrapati, 19:26)
- [See, Hear, Speak, Draw](https://aietalks.com/talks/see-hear-speak-draw) (Logan Kilpatrick & Simón Fishman, OpenAI, 18:43)
- [From VLM/VLAs to Embodied Agents](https://aietalks.com/talks/from-vlm-vlas-to-embodied-agents) (Armen Aghajanyan, Perceptron AI, 20:42)
- [Going beyond RAG: Extended Mind Transformers](https://aietalks.com/talks/going-beyond-rag-extended-mind-transformers) (Phoebe Klett, Normal Computing, 16:04)