Without Caching, Your AI Agent Costs 10x More

George Cameron, Artificial Analysis19:43 · Nov 2026 · 7,980 views
Thumbnail for Without Caching, Your AI Agent Costs 10x More Watch on YouTube
TL;DR
  1. 1

    Agent costs now depend heavily on input tokens, cache hit rate, and the number of steps in a task.

  2. 2

    In one Artificial Analysis benchmark, caching reduced the cost of an agent task from over $200 to about $21.

  3. 3

    The cheapest model is often sufficient for tasks with a fixed answer, while open-ended tasks may benefit from a more intelligent model.

Summary

George Cameron explains how developers should measure and control the cost of AI agents. Artificial Analysis evaluates models across intelligence, speed, pricing, cache hit rate, and other parts of the AI stack. Cameron argues that model choice now matters more because comparable tasks can cost a few cents or several dollars. Agent costs are difficult to estimate because a task may involve many turns, reasoning tokens, input tokens, and repeated context. In the AA Briefcase agent benchmark, an Opus task averaged about $21 across 165 steps. With no cache hits, the same task cost more than $200. Cameron also separates tasks where more intelligence can improve the result from tasks with a fixed answer, where a cheaper model is enough. He recommends using general benchmarks to create a shortlist, then testing those models on a team's own agents and workloads. For production systems, he recommends building a custom benchmark and comparing each model's cost against its performance on the actual task.

Key ideas
00:01

Artificial Analysis compares the tradeoffs across the AI stack

George Cameron introduces Artificial Analysis as an independent AI benchmarking company. It evaluates agent systems, language models, image, video, speech, and music models, as well as hardware and inference layers. A developer choosing a model is also choosing how to access it, what it costs, how fast it runs, and how often its inputs are cache hits. The company's Intelligence Index combines 10 evaluations covering areas such as knowledge work, scientific reasoning, coding, terminal use, and hallucination. Cameron says the index is intended to help developers choose models for current and near-term applications, rather than measure progress toward AGI.

05:25

Model progress is moving fast, so model selection becomes a recurring task

Cameron shows a chart with many model releases from leading labs and says the pace of progress is accelerating. He acknowledges that keeping up can feel exhausting, but argues that tracking releases matters for anyone who wants to know what AI can do. He also says open-weights models remain within three to nine months of leading proprietary models. That gap makes them a practical option in some applications, especially when their cost or other deployment characteristics are better.

09:01

The same level of intelligence becomes about ten times cheaper within a year

Artificial Analysis groups models by intelligence level and tracks the cheapest way to reach each level over time. Cameron gives an example in which reaching an Intelligence Index score of 30 initially cost a little over one dollar, then fell to around 10 cents a year later. The current cost is below 10 cents. He says the time dimension matters when planning AI spending because a capability that costs one million dollars today may cost around $100,000 later and potentially around $10,000 after another two years.

10:18

Agent cost is driven by turns, input tokens, and cache hits

Cameron says agent pricing is more complicated than the earlier focus on output and reasoning tokens. In the AA Briefcase agent benchmark, agents work through thousands of files to create reports, PowerPoints, and marketing videos. An Opus task costs about $21 on average and takes 165 steps. Each language model call includes answer tokens, reasoning tokens, input tokens, and cache hits. Cameron identifies the number of input tokens, the proportion that are cache hits, and the number of turns per task as the metrics developers should track most closely.

12:52

Caching can change a long-horizon task from about $21 to more than $200

The benchmark's average Opus task costs about $21 when caching is working. Cameron says that with a zero percent cache hit rate, the same task costs over $200. He gives this as a reason to examine inference providers and deployment settings, since prompt caching may not be enabled in every setup. Artificial Analysis has started publishing cache hit percentage benchmarks so developers can compare this part of the system rather than looking only at model token prices.

13:38

Reasoning effort raises cost through later turns and repeated history

Reasoning effort affects more than the number of reasoning tokens in one model call. Cameron says it can also change the number of turns in an agent trajectory. Those turns add input tokens because message histories are carried through the task. In his example, the same Opus model costs about one dollar per task at low reasoning effort and about $21 at maximum effort. The setting therefore changes the whole trajectory and its input-token cost, not just one line item.

15:30

Model choice should follow the task's intelligence ceiling

Cameron recommends using an intelligence-versus-cost Pareto view to choose a model. A team that wants the highest available intelligence can look toward the expensive end, while a team with a tighter budget can seek the best intelligence available at that price. He says the harder problem is understanding the tradeoff curve for the team's own tasks. General model rankings can provide a starting point, but the final choice depends on what the application actually needs.

17:02

Some tasks need more intelligence, while others have a fixed answer

Cameron divides tasks into unlimited-intelligence and limited-intelligence categories. For an open-ended task such as analyzing business costs and suggesting ways to reduce them, a smarter model may produce a better result. For a question such as how much was spent on subscriptions last month, more intelligence does not help once the system has the right data and tools. He gives the example that both a cheaper model and a more capable model could return $80. In that case, the team should find the cheapest model that reaches the required level.

18:06

Teams should benchmark shortlisted models on their own agents

Cameron recommends understanding the model market, comparing models against the team's priorities, and creating a shortlist. Developers can test the shortlist on small prototypes in their own context. For production deployments, he recommends building a custom benchmark and comparing five shortlisted models on the actual workload. Artificial Analysis's Optima product lets users upload context and generate a benchmark with tradeoff curves for their task, showing model performance alongside cost.

"If we don't have that then, and we don't have any cache hits, right? Cache hit percentage is zero, caching isn't working, maybe we've got our own setup with VLMs, we haven't enabled prompt caching, then instead of $21 per task, this costs over $200."12:53
Who should watch
  • You are building an agent that makes many model calls and need to understand why its costs are rising.
  • You are deciding between a leading proprietary model and a cheaper or open-weights model for a real workload.
  • Your application has repeated context and you need to measure cache hit rate, reasoning effort, and task length before production.