# How to Actually Tokenmaxx and Save Millions

Alvin Pane, OutRival | AI Engineer NYC 2026 | 11:12

Source: https://www.youtube.com/watch?v=8g1N_a3h9H0
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-to-actually-tokenmaxx-and-save-millions
Published: 2026-10-14
Tags: cost, evals, observability, prompt-engineering

## TL;DR
- Token usage measures the cost of producing work, so teams should reduce redundant calls and unnecessary prompt content.
- A cheap candidate that disagrees with production 18% of the time may still be acceptable when production disagrees with itself 7% of the time.
- Prompt and model changes should ship only after paired shadow evaluation, human review, confidence intervals, and checks on critical categories.

## Summary
Alvin Pane presents token optimization as a measurement and release process. Teams should first trace model calls, retries, prompts, token counts, latency, and cost. They can then measure how often production disagrees with itself before judging a cheaper candidate. At OutRival, a candidate disagreed with production on 18% of examples, while repeated production runs disagreed 7% of the time. Pane recommends prefix caching, automated prompt search with DSPy and GEPA, and model downsizing after the prompt has been tightened. A HarborBite synthetic example cuts instruction tokens by 51%, but that reduction is only useful if behavior holds. Shadow evaluation compares production with itself, the candidate with itself, and the two systems with each other. Human review classifies disagreements as regressions, improvements, or acceptable alternatives. A paired confidence interval must place the lower bound above zero, with critical customer categories checked separately before release.

## Key ideas
### Token reduction creates room for more useful work
[00:00](https://www.youtube.com/watch?v=8g1N_a3h9H0&t=0s)
Pane argues that tokens are a cost of doing work rather than a useful success measure. OutRival's digital workers handle phone, text, and email tasks, and the company processes billions of tokens each day at the application layer. Repeated model calls therefore add latency and compute cost. His HarborBite example uses a synthetic prompt with about 20 edge cases, detailed definitions, and anti-hallucination instructions to show how prompt bloat accumulates. The point is to extract the required behavior from a smaller token payload, rather than reward raw token consumption.

### Optimization starts with traces of the existing system
[01:39](https://www.youtube.com/watch?v=8g1N_a3h9H0&t=99s)
Before changing prompts or models, Pane says teams should instrument their shared model client so each attempt belongs to a trace for one task. The trace should connect model calls, tool calls, and retries. It should record the prompt version, model settings, token usage, latency, and cost. He names OpenTelemetry as a framework for this work. Those records provide the production benchmark needed to compare later changes against actual behavior instead of relying on a few hand-picked examples.

### Candidate disagreement must be compared with production variance
[02:26](https://www.youtube.com/watch?v=8g1N_a3h9H0&t=146s)
A cheaper candidate disagreed with the expensive production baseline on 18% of historical examples. Pane initially rejected it, then measured whether production agreed with itself. In an AA ceiling test, the production model ran three times on the same inputs and disagreed with itself roughly 7% of the time, giving about 93% agreement. This shows why raw candidate-versus-production disagreement is incomplete. A replacement should be judged against the variation already present in the production system.

### Caching and prompt search reduce repeated and unnecessary work
[04:00](https://www.youtube.com/watch?v=8g1N_a3h9H0&t=240s)
Pane recommends three levers in sequence, starting with prompt structure that improves prefix cache rates. The second is shrinking the prompt itself. His HarborBite example is almost 5,000 tokens because developers added rules and hard-coded worked examples over time. He proposes turning prompting into a search problem: define the interface, input, output schema, and scoring function, then let DSPy and GEPA revise instructions against a benchmark. The example removes most long examples and cuts instruction tokens by 51% while keeping the application contract.

### Smaller models need tests against unfamiliar behavior
[06:00](https://www.youtube.com/watch?v=8g1N_a3h9H0&t=360s)
After tightening the prompt, teams can downsize the model. Pane describes splitting work so a cheaper model handles routing or extraction while a larger model handles heavy reasoning. Another approach uses the same DSPy search loop with a smaller model, searching for instructions that match benchmark answers from the larger production model. He warns that compression introduces overfitting. A prompt can perform well on a narrow benchmark and still fail on inputs outside it, so many downsizing experiments were not ready to release.

### Shadow evaluation needs three separate comparisons
[07:27](https://www.youtube.com/watch?v=8g1N_a3h9H0&t=447s)
Pane's shadow deployment sends the same inputs through repeated production and candidate runs. It measures production against production, candidate against candidate, and candidate against production. The within-system comparisons establish how much each model varies without a system change. The cross-system difference is then adjusted by the average internal disagreement. This gives a more useful estimate of changed decision behavior than treating every difference as a regression.

### Human review separates errors from acceptable differences
[08:32](https://www.youtube.com/watch?v=8g1N_a3h9H0&t=512s)
Raw disagreement does not determine which system is better. Pane says reviewers should inspect sampled differences and decide whether the candidate was better, production was better, or both answers were acceptable for an ambiguous input. In the HarborBite example, review found 443 regressions and 2,463 outright improvements, nearly six improvements per regression. He uses this review to assess the direction of change rather than assuming that matching production is the only goal.

### Release decisions need paired confidence intervals and category checks
[09:16](https://www.youtube.com/watch?v=8g1N_a3h9H0&t=556s)
Pane says teams should not ship based on a point estimate from one batch. They should calculate a confidence interval around the net gain, keep both systems paired on the exact same requests, and group repeated runs of the same conversations. The candidate is better only when the lower confidence bound is above zero. The calculation must also run separately on critical customer categories, because a strong overall average can hide damage to an important intent. Measurement continues after release.

## Notable quotes
- "Tokens aren't the work. They're what you pay to get the work done." (00:20)
- "Does prod even agree with prod?" (03:20)
- "We aren't necessarily trying to build a perfect clone of a production system that may already have some flaws baked into it." (08:32)
- "To say the candidate is better, we want the lower confidence bound above zero." (09:38)

## Tools & references mentioned
- OutRival
- OpenTelemetry
- DSPy
- GEPA
- Meta
- HarborBite

## Who should watch
- You run expensive, repeated model calls and need a way to find prompt or caching waste before adding more compute.
- You are evaluating a cheaper model and need to distinguish real regressions from disagreement caused by production's own variability.
- Your team has a promising prompt rewrite but lacks a release rule based on paired tests, human review, and confidence intervals.

## Editor's note

Alvin Pane warns that a prompt can perform well on a narrow benchmark and still fail on inputs outside it. Kitaru records real agent runs, including model calls and tool results, so a failure seen in production can be replayed after a prompt, model, or code change. Those recorded sessions can become cohorts for repeatable evaluation.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Stop Rationing Tokens: Let the Harness Pick the Model](https://aietalks.com/talks/stop-rationing-tokens-let-the-harness-pick-the-model) (Laurent Gil & Žilvinas Urbonas, Cast AI, 18:08)
- [From Tokenmaxxing to Trusted Throughput](https://aietalks.com/talks/from-tokenmaxxing-to-trusted-throughput) (Mingsheng Hong, Ironclad, 23:04)
- [Your Agent Is Wasting Tokens and You Don't Know It](https://aietalks.com/talks/your-agent-is-wasting-tokens-and-you-dont-know-it) (Erik Hanchett, Amazon Web Services, 05:55)
- [Tokens Should Have Jobs](https://aietalks.com/talks/tokens-should-have-jobs) (Katelyn Lesse & Angela Jiang, Anthropic, 13:21)
- [Notion's Token Town](https://aietalks.com/talks/notions-token-town) (Sarah Sachs, Notion, 23:55)