Caching system prompts, tool prompts, and messages reduces what the agent sends after the first call.
2
Routing simple tasks to cheaper models avoids using an expensive model for every inference call.
3
Offloading large tool results, capping tool loops, and trimming conversation history keeps repeated calls from inflating the context.
Summary
Erik Hanchett describes five ways to reduce token costs in production agents without changing prompts or switching away from the models already in use. First, cache the system prompt so the full version is sent only on the first call. He then recommends routing tasks by difficulty, using cheaper models for simple work and more capable models for harder tasks. Large tool results can be stored and summarized instead of being added to the context on every loop. Tool loops should have a maximum iteration count, and observability should show which tools run too often or for too long. For multi-turn agents, a sliding window can send only recent messages, with a summary preserving older context when needed. The advice is practical and code-oriented, using AWS Strands agents as the example.
Hanchett begins with system prompt caching. In his pseudo code for AWS Strands agents, setting cache prompt to default sends the full system prompt on the first agent call. Later calls send a much smaller cached version. He says the same approach can apply to tool prompts and messages. The point is to avoid paying for identical prompt content each time the agent runs. He presents this as a small configuration change rather than a prompt rewrite or a model change.
The agent does not need the most expensive model for every task. Hanchett suggests using a frontier model for difficult work and a cheaper model such as Claude Haiku for simple inference, with Claude Sonnet for something more difficult. An if statement can route the request, and another inexpensive model can decide which model should handle it. His recommendation is to use multiple models according to the use case and perform that routing inside the agent.
Large tool results should stay out of repeated context
A tool can return a result that is much larger than the agent needs on every subsequent call. Hanchett shows a manual approach in Strands agents and mentions that additional APIs are available. The result can be stored locally or in the cloud, then summarized. The agent can use that summary instead of inserting the entire result into the context every time the tool loop runs. This prevents the same large payload from being sent back to the language model repeatedly.
Agents can call the same tool over and over when a loop has no limit. Hanchett says a tool might run 10 or 20 times, or enter an infinite loop, which creates unnecessary token usage. He recommends setting a maximum number of iterations for every agent loop. Before deployment, observability tools can show how often each tool is called and how long those calls run. That information gives developers a way to find inefficient tools and adjust the agent.
In a multi-turn conversation, the full history is sent to the language model on every new call. Hanchett says this can consume hundreds or thousands of tokens as the conversation grows. Strands agents include a sliding window conversation manager that sends only the last 10 messages by default, with the window size configurable. The trade-off is losing the earliest messages from the active history. A summary of older messages can be added to the context when the window is reached.
Hanchett closes by grouping the advice into five actions: cache prompts, route requests by difficulty, offload large tool results, cap tool loops, and trim long histories. He also pairs loop limits with observability, so developers can inspect tool duration and iteration counts before refining the agent. None of these recommendations requires changing the agent's prompts. They target repeated input and unnecessary work around the model call.
"If you can find any way that where you have this tool result that you don't necessarily send it on every single call back to the large language model, that will save a lot of tokens for you."02:18
Who should watch
You are running an agent in production and repeated prompts, tool results, or conversation history are making each call more expensive.
Your agent sometimes repeats tool calls and you need a simple way to limit loops and inspect which tools are inefficient.
You are using multi-turn agents and want to reduce history sent to the model without throwing away older information entirely.
