# How a Trading Firm Ships New AI Models the Same Day

Noyan Tokgozoglu, Optiver | AI Engineer | 23:23

Source: https://www.youtube.com/watch?v=aGIUg8zKI8A
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-a-trading-firm-ships-new-ai-models-the-same-day
Published: 2026-10-27
Tags: cost, deployment, evals, harness-engineering

## TL;DR
- Optiver measures the effect of changes on trading within hours, so its AI platform must support fast decisions, internal evaluation, and quick deployment.
- An inference gateway, internal golden sets, and a model- and harness-agnostic meta harness let Optiver adopt new frontier models without depending on one provider.
- Optiver reduced waste by controlling default context, improving caching, exposing team-level spend, using open-source models where suitable, and routing work between models.

## Summary
Noyan Tokgozoglu explains why Optiver built its own on-prem AI platform. The firm makes markets with its own money, and changes rolled out to traders can affect P&L within hours. That creates pressure to test new models quickly, measure them on internal workloads, and keep systems available across global regions. Optiver uses an inference gateway to route requests, track costs, set budgets, and switch providers. New models run against customer trading teams' golden sets before wider adoption. A meta harness gives teams portability across model and agent frameworks. Tokgozoglu also describes how spending rose after major model releases and how the company brought it under control by reducing oversized context, improving MCP and cache use, exposing budgets, testing open-source models, and routing tasks to cheaper models. The platform is built around self-service, but AI engineers work directly with traders and improve the shared tools based on their workflows.

## Key ideas
### Optiver sees trading feedback within hours
[00:27](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=27s)
Optiver is a market maker that trades with its own money and has no clients. Its systems process more than 10 million trades per day, price more than a million instruments, and handle petabytes of data daily. Unlike a typical technology company that may wait months or quarters for revenue data, Optiver can roll a change out to traders and quants and see its P&L impact by the end of the day. That short feedback loop makes missed opportunities expensive. Tokgozoglu says the firm therefore needs to decide quickly, measure changes efficiently, and keep a bias for action.

### The platform is built for speed, measurement, and self-service
[02:53](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=173s)
Optiver's platform principles are to move fast, stay modular, and measure each layer. New models and new patterns such as loops, swarms, or graphs can be tested and adopted when they add value. External benchmarks are insufficient because Optiver's internal data distribution differs from public benchmark data, so the company uses internal evaluations. Traders get self-service tools with sensible defaults. AI engineers also work directly with trading teams, learn their workflows, and bring useful patterns back into the platform. Because Optiver is regulated and protects its intellectual property, its infrastructure is on-premises, with limited external calls for third-party inference.

### The inference gateway gives Optiver model choice and operational control
[05:13](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=313s)
Optiver runs models from third-party providers, self-hosted models, and open-source models through an inference gateway. The gateway handles routing, records model spend, and supports budgets. It also gives the firm a fallback when a provider is unavailable, which matters because downtime can reduce trading activity. Above the model layer, Optiver has an agent platform with skills, plugins, tools, memory, and knowledge systems. An agent gateway provides a shared ledger for agents, communication, orchestration, and identity management. The company also provides web interfaces, an agent workbench, and a personal assistant so people can use the platform through familiar surfaces.

### New frontier models can reach production on the day they appear
[07:10](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=430s)
When a new frontier model becomes public, Optiver aims to use it in production flows the same day if its internal tests show value. The company partners with trading teams to collect golden sets and runs new models against those workloads immediately. The platform team then adds the model to the gateway, prepares its APIs and cost controls, updates harnesses, and arranges global support. Results and recommendations are shared with employees so they know how well the model works and where to use it. The gateway also protects the firm from regional availability problems, cloud-provider differences, and workloads that must stay in isolated on-premises zones.

### A meta harness keeps teams portable across models and frameworks
[08:31](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=511s)
Optiver wants to avoid coupling its workflows to one model provider or agent framework. Third-party models, on-premises models, and different harnesses pass through the inference gateway, where interactions are logged and sent to evaluation. A meta harness sits above the supported harnesses. Teams can define their own configurations there while keeping some portability between implementations. The meta harness connects to team knowledge bases, and the company uses traces to derive additional knowledge over time. Teams can deploy agents on the platform or access them through the web portal, IDE, and other interfaces.

### Spend control starts with reducing unnecessary context
[09:54](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=594s)
Optiver's AI spend rose sharply during major frontier model releases, then came back under control through operational changes. Tokgozoglu says fresh sessions sometimes carried at least 200,000 tokens of default context because teams were putting too much information into prompts. The company introduced MCP tool search, allowed agents to use a CLI that wrapped multiple MCPs, and examined cache invalidation and cache duration. Engineers also moved information into knowledge bases instead of keeping it in very large system prompts. These changes reduce token use while preserving access to information, and the savings increase when tokens become more expensive.

### Visibility and model routing help teams spend with more discipline
[12:32](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=752s)
Optiver gives teams visibility into their spending, alerts, and self-managed limits. Tokgozoglu says teams often set reasonable limits when they can see the cost and control it themselves. The company also tests open-source models where they offer good value and explores routing prompts to cheaper models. Another pattern is to use a frontier model for planning and smaller models for implementation. Better knowledge bases can reduce retrieval volume by returning only the needed context. The goal is to get more useful work from the same token budget, rather than simply making more models available.

### Optiver builds interfaces around the tools traders already make
[13:54](https://www.youtube.com/watch?v=aGIUg8zKI8A&t=834s)
Teams had built different harnesses, developer-box setups, and small web interfaces to fit their workflows. Optiver responded by building a shared web portal with session management, chat windows, and artifacts. Traders now contribute code through pull requests when they find useful additions, including collaborative document editing. The company is also working toward rooms where a user can bring together their own sessions, several agents, and other deployed agents, with access through Slack as well. Tokgozoglu's point is practical: users with high agency will build missing tools themselves, so the platform team should provide a better supported version.

## Notable quotes
- "At Optiver, we implement something. We roll out that change to our traders and our quants, and literally within hours you can see the P&L impact." (01:16)
- "If a new frontier model is made available publicly, we want to be able to leverage it in production flows the same day because there's immense value in that if it is valuable." (07:10)
- "We need to be able to be agnostic to the model and harness." (08:44)
- "On a fresh session, we were seeing that, you know, sometimes people have like 200k tokens worth of context like by default minimum, sometimes even more." (10:43)
- "Our users are very high agency. They will go and build something if we don't build it for them. So, we just built a better version of it." (15:21)

## Tools & references mentioned
- Optiver
- Apple
- SpaceX
- Hugging Face
- Claude Code
- MCP
- Codex
- Slack

## Who should watch
- You run AI infrastructure for a regulated company and need to support both hosted and self-hosted models without locking teams to one provider.
- Your model releases are fast, but internal evaluation, cost controls, harness updates, or regional support slow adoption.
- Your teams are building agent tools independently and you need to decide which capabilities belong in a shared platform.

## Related talks

- [Trends Across the AI Frontier](https://aietalks.com/talks/trends-across-the-ai-frontier) (George Cameron, Artificial Analysis, 17:52)
- [Local Models: Trust, Control, Optimization](https://aietalks.com/talks/local-models-trust-control-optimization) (Carter Abdallah, NVIDIA & Vincent Weisser, Prime Intellect & Lucas Atkins, Arcee AI & Chris Alexiuk, NVIDIA, 43:21)
- [Making Open Models 10x Faster and Better for Modern Application Innovation](https://aietalks.com/talks/making-open-models-10x-faster-and-better-for-modern-application-innovation) (Dmytro (Dima) Dzhulgakov, Fireworks AI, 18:55)
- [Finetuning: 500M AI Agents in Production with 2 Engineers](https://aietalks.com/talks/finetuning-500m-ai-agents-in-production-with-2-engineers) (Mustafa Ali, Method Financial & Kyle Corbitt, OpenPipe, 18:44)
- [The Rise of Open Models in the Enterprise](https://aietalks.com/talks/the-rise-of-open-models-in-the-enterprise) (Amir Haghighat, Baseten, 16:50)