# Fun Stories from Building OpenRouter and Where All This Is Going

Alex Atallah, OpenRouter | AI Engineer World's Fair 2025 | 18:47

Source: https://www.youtube.com/watch?v=84Vtz2IL1Ug
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/fun-stories-from-building-openrouter-and-where-all-this-is-going
Published: 2025-06-25
Tags: agents, caching, inference, mcp, multi-agent

## TL;DR
- OpenRouter began as an experiment to help developers discover and use the growing range of language models.
- The service became a marketplace by aggregating providers with different prices, capabilities, latency, and uptime.
- Atallah expects software to use multiple models, with routing and inference costs becoming major engineering concerns.

## Summary
Alex Atallah describes how OpenRouter grew from a question about whether AI inference would become a winner-take-all market. The arrival of Llama 1 and Alpaca convinced him that many specialized models would appear, creating a need for discovery and comparison. He first tested this idea with Window AI, a Chrome extension that let users choose a model inside other web applications. OpenRouter then launched as a model collection and became a marketplace for providers. Its API normalizes differences in tool calls, caching, pricing, performance, and stream behavior. Atallah argues that the market will be multimodel because developers use different models for different jobs. He also discusses an AI-focused middleware system that can add web search and transform streamed outputs. Future work includes more modalities, geographic routing, model discovery, prompt observability, and lower prices.

## Key ideas
### Moderation differences created early demand for alternative models
[01:32](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=92s)
In January 2023, Atallah saw users looking beyond OpenAI because they wanted to understand model moderation policies. Some users were writing stories that included violence, and OpenAI sometimes refused to generate parts of them or treated them as questionable under its terms of service. Roleplay created another gray area around what models would produce. This gave Atallah early evidence that developers and users cared about more than a single model's general quality. They also cared about what a model allowed and how its provider handled moderation.

### Llama 1 made open weights models seem competitive, even though they were hard to use
[03:14](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=194s)
The open-source race accelerated with Meta's Llama 1. Its abstract claimed that the 13-billion-parameter model outperformed GPT-3 on many benchmarks, which surprised the community because it was a smaller model that could potentially run on a laptop. Atallah says Llama 1 was still mainly a text-completion model, difficult to run locally, and supported by immature infrastructure. Even so, its benchmark results changed expectations about what open models could do and prompted people to search for useful applications.

### Alpaca showed that useful model behavior could be distilled cheaply
[04:27](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=267s)
In March 2023, Stanford researchers generated outputs from GPT-3, fine-tuned Llama 1 on them, and created Alpaca for less than $600 in total. Atallah calls this the first successful distillation that he saw transfer both style and knowledge from a large model to a smaller one. The result suggested that teams did not need a $10 million training budget to create their own models. It also suggested that unique data could be made available as a language model. Atallah then expected tens or hundreds of thousands of models that would need to be discovered and understood.

### Window AI tested whether users could bring model choice into ordinary software
[06:21](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=381s)
Before OpenRouter, Atallah launched Window AI, an open-source Chrome extension. It let users choose a language model while a web application simply consumed the user's selected model. The example shown in the talk used Together's open-source deployment of GPT-NeoX inside a web page. This experiment addressed a practical problem: a developer might not know which model a user wants, so the user needs a way to bring that choice into the software. OpenRouter was later co-founded with Lewis, whose Plasmo framework supported Window AI.

### Provider differences turned a model directory into a marketplace
[08:49](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=529s)
OpenRouter initially collected models and showed usage data. Each model often had only one or two providers, so the system used a primary provider and a fallback without naming providers. That changed as more companies hosted the same models at different prices and performance levels. Providers also differed in support for features such as the min-p sampler, caching, tool calling, and structured outputs. OpenRouter aggregated those providers and exposed their different price points. It became a marketplace that covered both open-source and closed-source models, with latency and throughput data based on real usage.

### A single API reduces the cost of changing models and providers
[08:03](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=483s)
Atallah describes OpenRouter as an API for accessing more than 400 models from more than 60 active providers. Customers pay through one API and can switch models with near-zero switching costs. OpenRouter normalizes tool calls and handles caching so developers do not need to track every provider's supported features. The service also offers model comparisons, usage visibility, privacy controls, and public rankings showing which models people use. Aggregating providers can improve uptime because a request can be routed across multiple deployments of the same model.

### The market is likely to use many models, with inference treated as a commodity
[11:47](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=707s)
Atallah says OpenRouter's data led him to believe that intelligence will not be winner-take-all. Customers use different models for different purposes, and Google Gemini's share on the platform grew from roughly 2 or 3 percent to 34 or 35 percent over the period he shows. He expects the future to be multimodel. Inference should look more like a commodity, where the same model can be delivered by different providers at different rates and performance levels. He also expects inference to become a dominant operating expense, making model selection and routing important.

### OpenRouter is building AI-native middleware around inference
[13:50](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=830s)
OpenRouter does not have an MCP marketplace, but Atallah describes a need to add features such as web search and PDF parsing to every model. Standard MCP pre-flight behavior is not enough because OpenRouter also needs to transform outputs as they return to the user. The team designed an AI-focused middleware and plugin system. A web search plugin can give any language model web-search ability and add web annotations to streamed results. Plugins can call MCPs internally and augment the response while tokens are being delivered, rather than waiting for the entire output.

### Provider normalization requires handling latency, cancellation, and billing differences
[15:47](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=947s)
The team worked to reduce latency to about 30 milliseconds through custom caching. It also had to make streams cancellable across providers with different policies. Some providers bill for the entire generation when a stream is dropped, while others bill only for later tokens or stop billing immediately. OpenRouter had to study these edge cases and standardize behavior across models and providers. Atallah presents this as a substantial architecture problem because developers need predictable behavior even when the underlying inference services handle streams differently.

## Notable quotes
- "Will this market be winner take all?" (00:48)
- "This is knowledge finally being distilled into software." (05:36)
- "The future is going to be multimodel." (12:40)
- "Inference is also a commodity." (12:40)
- "We really wanted to get extremely low latency." (15:47)

## Tools & references mentioned
- OpenRouter
- OpenAI
- ChatGPT
- Bloom 176B
- OPT
- Meta
- Llama 1
- GPT-3
- Alpaca
- Stanford
- Window AI
- Plasmo
- Together
- Google Gemini
- Anthropic
- Claude
- Amazon Bedrock
- Vertex AI
- MCP
- Next.js
- a16z

## Who should watch
- You are building an application that may need to switch between models or providers without rewriting its integration.
- Your team needs to compare model prices, latency, throughput, uptime, or provider-specific features using one API.
- You are designing middleware, routing, web search, or other features that must work across many language models.

## Related talks

- [Making Open Models 10x Faster and Better for Modern Application Innovation](https://aietalks.com/talks/making-open-models-10x-faster-and-better-for-modern-application-innovation) (Dmytro (Dima) Dzhulgakov, Fireworks AI, 18:55)
- [Preferences Over Benchmarks: Model Routing](https://aietalks.com/talks/preferences-over-benchmarks-model-routing) (Archana Kamath & Tyler Gillam, DigitalOcean, 15:54)
- [The Rise of Open Models in the Enterprise](https://aietalks.com/talks/the-rise-of-open-models-in-the-enterprise) (Amir Haghighat, Baseten, 16:50)
- [Local Models: Trust, Control, Optimization](https://aietalks.com/talks/local-models-trust-control-optimization) (Carter Abdallah, NVIDIA & Vincent Weisser, Prime Intellect & Lucas Atkins, Arcee AI & Chris Alexiuk, NVIDIA, 43:21)
- [The State of Model Routing](https://aietalks.com/talks/the-state-of-model-routing) (Nader Khalil, NVIDIA & Walden Yan, Cognition & Alex Atallah, OpenRouter & Tanay Varshney & Carter Abdallah, NVIDIA, 48:17)