# Preferences Over Benchmarks: Model Routing Archana Kamath & Tyler Gillam, DigitalOcean | AI Engineer | 15:54 Source: https://www.youtube.com/watch?v=FvxY8oPoI8o Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/preferences-over-benchmarks-model-routing Published: 2026-08-22 Tags: cost, evals, inference, latency ## TL;DR - There is no single best model because the right choice depends on the request, its tools and prompts, required latency, acceptable cost, and user preferences. - A configurable router can select models per task, apply ranking or failover rules, and be tested against a team's own evaluations. - In the coding-agent demo, routing produced similar results at much lower session cost, while the router made decisions in under 200 milliseconds and required no application code changes. ## Summary Archana Kamath and Tyler Gillam argue that choosing one model by its position on a public benchmark misses the conditions that determine whether it fits a real request. Classification may work well with a small open model, code completion may need a faster model, and code review or security work may justify a frontier model. Their Inference Light Router lets teams describe preferences such as cost, latency, task, preferred models, and hardware. Teams can add decision-tree rules, rank models manually, configure failover, and use their own evaluations to adjust routing. In the demo, a software engineering router selected different models for code generation, optimization, testing, and documentation. A coding-agent comparison showed similar output quality while reducing session cost from 25 cents to 8 cents, then from 44 cents to 14 cents after more prompts. The speakers present routing as a base for evaluation, caching, and personalization. ## Key ideas ### The one-model habit creates cost and availability problems [01:17](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=77s) Archana Kamath gives three reasons to stop sending every request to one model. Inference spending is growing, and she names Walmart, Uber, and Microsoft as companies capping usage to control bills. A single frontier model can also be unnecessary for tasks that a smaller model handles well. Finally, relying on one model leaves a product without failover when that model goes down or degrades. She compares model orchestration with cloud cost optimization, saying that orchestration is arriving in months rather than taking years to become a mature discipline. ### The right model depends on the request [02:29](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=149s) Kamath says there is no single best model. A small open model may handle classification and labeling, while inline code completion may need a faster, larger model. Code generation and bug fixing can often use a mid-sized open-weight model, while accuracy-sensitive work such as code review and security may call for a frontier model. The choice also depends on the task, system prompt, surrounding tools, acceptable cost, required latency, and what the end user wants. A public leaderboard cannot encode all of those conditions for a particular application. ### Routing becomes useful when teams can inspect and change its choices [04:38](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=278s) Tyler Gillam describes a router built from an open proxy planner and a purpose-built routing model. Teams state what matters for their workflow, including cost, latency, quality, preferred models, or hardware. They can describe a task in natural language, add decision-tree rules, start from presets, and change configuration in a line of code. Gillam's main distinction from earlier auto-routing systems is that teams can evaluate and improve the choice instead of treating it as a black box. His loop is to route, evaluate, adjust, and feed the results back into the configuration. ### Model pools can use fixed preferences or recent speed [06:51](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=411s) In the console demo, Gillam customizes a software-engineering preset with tasks such as bug fixing, code generation, and test writing. A task can have more than one model. For code generation, he ranks one model first and uses another as failover if the preferred model is unavailable. For bug fixing, he uses a fastest selection policy, which chooses from the model pool based on recent speed. This lets different tasks follow different rules rather than forcing the whole workflow through one global model choice. ### The playground shows task-based routing with similar outputs and lower cost [07:48](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=468s) Gillam sends a basic Fibonacci-function prompt, an optimization request, and a unit-test request through the playground. The router maps each request to a configured task and selects different models for the work. He observes that the routed responses are faster and cheaper than sending the requests to Opus. He calls this comparison a vibe check because it does not establish performance rigorously. The router's value in this demonstration comes from matching the model to the task instead of paying premium-model rates for every request. ### An evaluation measures the tradeoff better than a visual comparison [09:29](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=569s) Gillam compares the router with Opus on an evaluation. The router scores 90% correctness, while Opus scores 95%. He says the difference is within the judge's margin of error. The router also uses significantly fewer tokens and returns faster. This gives the team a way to judge whether the cost and latency savings are acceptable for its own workload, rather than relying on a general leaderboard or on whether one manually inspected answer looks better. ### Coding-agent sessions show the cost gap accumulating across tasks [09:37](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=577s) The live workflow runs the same spinning-wheel app request in two terminals. One sends every request to a premium model, while the other uses the software-engineering router. The router selects models as requests match tasks such as code generation, then produces a result that Gillam considers comparable and possibly better in that example. The router session costs 8 cents versus 25 cents after the first request sequence. After adding unit tests and README documentation, the totals are 14 cents for the router and 44 cents for the direct Opus setup. Latency is selected per step rather than fixed at the premium model's level. ### Routing is a base layer for evaluation, caching, and personalization [13:56](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=836s) Kamath says routing decisions take under 200 milliseconds per request and run on a custom mixture-of-experts model built for routing. The router is free and requires no application code changes, and the routing model is open sourced. She describes routing as a foundation for other features. Evaluations test whether the selected model works for the use case, caching avoids paying again for the same answer, and personalization lets the router learn what works for a team over time. The intended loop improves as a workload is routed and evaluated repeatedly. ## Notable quotes - "There is no single best model. The right one depends on the actual request." (02:29) - "You validate with your own evaluations, not someone else's leaderboard." (06:08) - "The software engineer router has only spent 8 cents on the session, while Opus directly has spent 25 cents." (11:46) - "There is no single best model. There's only the right model for the request." (15:04) ## Tools & references mentioned - DigitalOcean - Inference Light Router - OpenCode - GPT-5 - Opus - Claude - Gemini - Llama - Walmart - Uber - Microsoft ## Who should watch - You run several model types in production and need a way to trade response quality against cost and latency per task. - Your application depends on one model and has no fallback when that provider becomes slow or unavailable. - You want to replace informal model comparisons with evaluations tied to your own prompts, tools, and user preferences.