Token usage measures the cost of producing work, so teams should reduce redundant calls and unnecessary prompt content.
2
A cheap candidate that disagrees with production 18% of the time may still be acceptable when production disagrees with itself 7% of the time.
3
Prompt and model changes should ship only after paired shadow evaluation, human review, confidence intervals, and checks on critical categories.
Summary
Alvin Pane presents token optimization as a measurement and release process. Teams should first trace model calls, retries, prompts, token counts, latency, and cost. They can then measure how often production disagrees with itself before judging a cheaper candidate. At OutRival, a candidate disagreed with production on 18% of examples, while repeated production runs disagreed 7% of the time. Pane recommends prefix caching, automated prompt search with DSPy and GEPA, and model downsizing after the prompt has been tightened. A HarborBite synthetic example cuts instruction tokens by 51%, but that reduction is only useful if behavior holds. Shadow evaluation compares production with itself, the candidate with itself, and the two systems with each other. Human review classifies disagreements as regressions, improvements, or acceptable alternatives. A paired confidence interval must place the lower bound above zero, with critical customer categories checked separately before release.
Pane argues that tokens are a cost of doing work rather than a useful success measure. OutRival's digital workers handle phone, text, and email tasks, and the company processes billions of tokens each day at the application layer. Repeated model calls therefore add latency and compute cost. His HarborBite example uses a synthetic prompt with about 20 edge cases, detailed definitions, and anti-hallucination instructions to show how prompt bloat accumulates. The point is to extract the required behavior from a smaller token payload, rather than reward raw token consumption.
Optimization starts with traces of the existing system
Before changing prompts or models, Pane says teams should instrument their shared model client so each attempt belongs to a trace for one task. The trace should connect model calls, tool calls, and retries. It should record the prompt version, model settings, token usage, latency, and cost. He names OpenTelemetry as a framework for this work. Those records provide the production benchmark needed to compare later changes against actual behavior instead of relying on a few hand-picked examples.
Candidate disagreement must be compared with production variance
A cheaper candidate disagreed with the expensive production baseline on 18% of historical examples. Pane initially rejected it, then measured whether production agreed with itself. In an AA ceiling test, the production model ran three times on the same inputs and disagreed with itself roughly 7% of the time, giving about 93% agreement. This shows why raw candidate-versus-production disagreement is incomplete. A replacement should be judged against the variation already present in the production system.
Caching and prompt search reduce repeated and unnecessary work
Pane recommends three levers in sequence, starting with prompt structure that improves prefix cache rates. The second is shrinking the prompt itself. His HarborBite example is almost 5,000 tokens because developers added rules and hard-coded worked examples over time. He proposes turning prompting into a search problem: define the interface, input, output schema, and scoring function, then let DSPy and GEPA revise instructions against a benchmark. The example removes most long examples and cuts instruction tokens by 51% while keeping the application contract.
Smaller models need tests against unfamiliar behavior
After tightening the prompt, teams can downsize the model. Pane describes splitting work so a cheaper model handles routing or extraction while a larger model handles heavy reasoning. Another approach uses the same DSPy search loop with a smaller model, searching for instructions that match benchmark answers from the larger production model. He warns that compression introduces overfitting. A prompt can perform well on a narrow benchmark and still fail on inputs outside it, so many downsizing experiments were not ready to release.
Shadow evaluation needs three separate comparisons
Pane's shadow deployment sends the same inputs through repeated production and candidate runs. It measures production against production, candidate against candidate, and candidate against production. The within-system comparisons establish how much each model varies without a system change. The cross-system difference is then adjusted by the average internal disagreement. This gives a more useful estimate of changed decision behavior than treating every difference as a regression.
Human review separates errors from acceptable differences
Raw disagreement does not determine which system is better. Pane says reviewers should inspect sampled differences and decide whether the candidate was better, production was better, or both answers were acceptable for an ambiguous input. In the HarborBite example, review found 443 regressions and 2,463 outright improvements, nearly six improvements per regression. He uses this review to assess the direction of change rather than assuming that matching production is the only goal.
Release decisions need paired confidence intervals and category checks
Pane says teams should not ship based on a point estimate from one batch. They should calculate a confidence interval around the net gain, keep both systems paired on the exact same requests, and group repeated runs of the same conversations. The candidate is better only when the lower confidence bound is above zero. The calculation must also run separately on critical customer categories, because a strong overall average can hide damage to an important intent. Measurement continues after release.
