The most intelligent model is not always the right choice because reasoning models can use far more tokens, cost more, and take longer to respond.
2
The intelligence gap between open-weights and proprietary models has narrowed sharply, with China-based labs producing several leading open-weights models.
3
Model costs and speeds have improved quickly, but demand for compute is still expected to rise as models become larger and agents make more sequential requests.
Summary
George Cameron presents Artificial Analysis's view of AI progress across several frontiers. The company benchmarks model intelligence, API speed and cost, hardware, and multiple modalities. Cameron argues that model intelligence is only one factor in choosing a system. Reasoning models can score higher while using an order of magnitude more output tokens, increasing both cost and latency. That matters especially for agents, where many sequential calls can turn a small per-request delay into minutes of waiting. He also describes the shrinking gap between open-weights and proprietary models, with DeepSeek and Alibaba among the strongest contributors. Model costs have fallen sharply across intelligence levels, while output speeds have improved through model design, inference software, and hardware. Cameron's final view is that compute demand will continue to grow because larger models, reasoning, and agent workflows consume more inference capacity.
Model intelligence is only one frontier developers need to measure
Cameron says AI progress should be viewed across several frontiers rather than through a single ranking of intelligence. Artificial Analysis's intelligence index combines seven evaluations into a generalist measure, and its current ranking places o3 first, followed by o4-mini with high reasoning, DeepSeek R1, Grok 3 Mini, Gemini 2.5 Pro, and Claude 4 Opus Thinking. He then shifts the focus to trade-offs around that intelligence. The talk examines reasoning models, open weights, cost, and speed. The practical point is that developers should measure the properties that affect their application instead of automatically selecting the highest-ranked model.
Reasoning models gain intelligence by producing far more output tokens
Artificial Analysis plotted intelligence against the output tokens used in its seven evaluations and found two distinct groups. Non-reasoning models use fewer output tokens and provide less intelligence, while reasoning models use more tokens and provide greater intelligence. Cameron gives a large difference in verbosity: GPT-4.1 used 7 million tokens for the evaluations, o4-mini high used 72 million, and Gemini 2.5 Pro used 130 million. Those extra tokens affect both cost and response time. The measurement therefore needs to include the tokens generated during reasoning, rather than treating the model's intelligence score as an isolated result.
Latency can determine whether an agent workflow is usable
Cameron compares median API response times and says GPT-4.1 took 4.7 seconds to return a full response, while o4-mini high took more than 40 seconds. A single request may tolerate that difference in some settings, but agent workflows multiply it. He says 30 sequential queries is normal for an agent. At 10 seconds per reasoning-model request, the total wait can reach 300 seconds, or five minutes. At one second per request, the same workflow takes 30 seconds. He applies this to a contact-center application, where 30 seconds might be acceptable but five minutes is not. Developers should measure latency in the actual workflow.
Open-weights models have moved much closer to proprietary systems
Around the GPT-4 period, Cameron says the intelligence gap between open weights and proprietary models was large. Models such as Llama 65B and Llama 2 70B were not close to GPT-4. The gap narrowed through models including Mixtral 8x7B and Llama 3 405B, although o1 pulled ahead again in late 2024. DeepSeek V3 and then DeepSeek R1 changed the comparison, with R1 coming within a few points of the leading models on the Artificial Analysis intelligence index. Cameron says the gap is now smaller than it has ever been, making open-weights options more relevant to application decisions.
China-based labs lead much of the open-weights frontier
Cameron says the leading open-weights models in both reasoning and non-reasoning categories come from China-based AI labs. DeepSeek leads both categories in his comparison. Alibaba's Qwen 3 series ranks second in reasoning, while Meta and Nvidia also appear close with Neotron fine-tunes of Llama. The observation is about model performance rather than a general claim about every open model. For developers evaluating open weights, the comparison includes systems from several regions and companies, with recent Chinese releases making a large contribution to the frontier.
The cost of accessing a given intelligence level has fallen sharply
The cost comparison shows large differences between models. Cameron says running the Artificial Analysis intelligence index cost $2,000 with o3. GPT-4.1 was roughly 30 times cheaper, and GPT-4.1 nano was more than 500 times cheaper than o3. He also says the cost of accessing GPT-4-level intelligence has fallen by more than 100 times since the middle of 2023. New intelligence bands become cheaper soon after they appear. The implication for application planning is that a cost structure that does not work today may become feasible within six months, so teams should consider how falling model prices could change their design.
Per-token prices hide the cost of verbose reasoning
Cameron warns that developers should not compare only the advertised price per million tokens. Reasoning models output thinking tokens, and those tokens are billed as output tokens even when a provider hides them from the user. The actual cost depends on both the token price and how many tokens the model generates. He also points out that non-reasoning models vary in response verbosity. Artificial Analysis therefore measures the cost of running its full intelligence index, which captures the amount of output needed to reach a model's measured result. Application teams should benchmark their own workloads in the same way.
Faster models and hardware have changed the speed frontier
Output speed has risen across intelligence levels. Cameron says GPT-4-level intelligence was available at around 40 output tokens per second in 2023, while the same level can now be accessed at more than 300 output tokens per second. He attributes the improvement to mixture-of-experts models, distillation, inference optimizations such as FlashAttention and speculative decoding, and faster hardware. Artificial Analysis's B200 benchmarks reach more than 1,000 output tokens per second, compared with roughly 40 for GPT-4 in 2023. He also names specialized accelerators from Cerebras, SambaNova, and Groq.
Improved efficiency will not stop compute demand from growing
Cameron expects total demand for compute to keep increasing even as models become cheaper and faster. Larger models require more capacity, and he cites DeepSeek as having more than 600 billion total parameters. Demand for higher intelligence also continues to rise. Reasoning models use more inference-time compute because they generate longer responses, while agent systems can make 20, 30, or more than 100 sequential model requests. These factors multiply demand rather than cancelling out the efficiency gains from better models, inference software, and hardware.
"You need to think about this and measure it in your application and benchmark not just by the cost per million tokens but also considering how many reasoning tokens there are and how verbose these models are."12:03
Who should watch
You are choosing between reasoning and non-reasoning models and need to understand the latency and token costs behind their benchmark scores.
Your agent makes many sequential API calls, so a per-request delay could turn into several minutes of application latency.
You are evaluating open-weights models, planning around falling inference costs, or estimating future hardware and compute needs.
