When Running Your Own AI Models Beats Paying for APIs

Steven Latré, imec18:30 · Oct 2026 · 1,812 views
Thumbnail for When Running Your Own AI Models Beats Paying for APIs Watch on YouTube
TL;DR
  1. 1

    Moving from APIs to self-hosted models adds hardware, concurrency, model, harness, and inference-engine choices to the cost calculation.

  2. 2

    Hardware costs rise in steps, so a model with similar accuracy can require much more expensive infrastructure than another model.

  3. 3

    For long-running coding tasks, the right harness can use half as many tokens and cut GPU wall time in half without changing task accuracy much.

Summary

Steven Latré presents imec's study of the cost of moving long-running coding agents from hosted APIs to owned infrastructure. The team ran more than a thousand benchmark runs on SWE-bench Pro, testing open-weight models, coding harnesses, and hardware configurations. Model accuracy matters, but it does not determine the practical cost by itself. Hardware requirements rise in steps, concurrency can fall sharply for larger models, and GPU utilization determines whether renting or buying infrastructure makes financial sense. Latré says smaller systems may need about 89% utilization to beat API pricing, while an 8x B200 system can break even at about 15% utilization. The study also found that harness choice changes token consumption and GPU wall time even when task accuracy stays similar. imec plans to publish a broader benchmark dataset and bring software and hardware teams together to study these trade-offs.

Key ideas
00:30

AI infrastructure choices become much harder once a team owns the stack

Steven Latré explains that imec works from semiconductor technology up through systems and workloads, which led his team to study AI infrastructure. He describes a progression from using closed APIs, to optimizing token costs, to hosting open-weight models, leasing infrastructure, and potentially buying it. APIs mainly require choosing a model for a task. Owned infrastructure adds decisions about model trust, coding harnesses, hardware architecture, CPU-to-GPU balance, KV-cache optimization, and inference engines. Latré says this larger design space is where imec wants to connect model and harness decisions with infrastructure choices.

05:28

Long-horizon coding tasks are where owning infrastructure may first make sense

The study focuses on long-horizon tasks because Latré considers them the first place where taking control of the AI stack can become relevant. The team used SWE-bench Pro, a coding-assistance benchmark, and tested three coding harnesses: Codex, Claude Code, and another harness discussed in the study. They compared a standard API environment with open-weight models of different complexities and parameter sizes. The runs also mapped those models to hardware architectures that could support multiple concurrent users. The team measured task resolve rate, latency, and cost across more than a thousand runs.

07:28

Model accuracy does not predict infrastructure cost in a smooth way

The smaller models in the study, including Qwen and DeepSeek, resolved roughly 32.5% to 40% of tasks. GLM reached an Opus-level range, while Kimi K3 reached an 86% resolve rate. Latré says the hardware comparison is more revealing than accuracy alone. Qwen and DeepSeek had similar application performance, yet Qwen required hardware that was four times more expensive to rent. Moving from GLM to Kimi K3 added about 20% accuracy and only about 15% to GPU rental cost in the tested setup. Hardware costs therefore behave like steps, because a small model change can force a different hardware configuration.

10:05

Concurrency can matter more than accuracy for agent workloads

With an API, users mainly think about token latency and throughput. With owned infrastructure, they must also ask how many users can share a machine before latency deteriorates. The model parameters must fit in memory, and each user's KV cache adds to the memory requirement. Latré says comfortable concurrency dropped sharply for larger models, even when the hardware was also larger. Kimi K3 could support about four concurrent users reliably in the tested coding-assistance setup, while its latency was still three times slower than the baseline Claude Code setup. His conclusion is that concurrency can matter more than accuracy for long-horizon coding tasks.

12:37

Large GPU systems can reach break-even at much lower utilization

The study compares staying with an API, renting GPUs, and buying infrastructure through a total-cost-of-ownership analysis. Smaller hardware systems are difficult to justify because their GPUs must remain busy almost all the time. Latré gives an estimated break-even utilization of about 89% for lower-tier systems. For an 8x B200 system, the threshold falls to about 15% GPU utilization. He says GLM-class applications were around the point where moving from API usage to infrastructure became more attractive in the tested coding-assistance case. The analysis found roughly 24.5% cost savings in that infrastructure scenario.

14:35

Harness choice changes GPU cost even when task results look similar

The harnesses produced broadly similar application accuracy across the tested models and tasks, so a harness could appear unimportant when the only concern is whether the task finishes. That changes when the team pays for GPU time. Different harnesses generated very different numbers of tokens. For GLM, Codex used about half as many tokens as the other harnesses in the study. Its GPU wall time was also about half as long. Latré's point is that the harness-to-model pairing can leave task accuracy mostly unchanged while substantially changing total infrastructure cost.

16:20

The benchmark must expand because model, hardware, and application mixes keep changing

Latré describes the results as an initial study rather than a final pricing rule. The current work covers coding assistance, while enterprise workloads usually combine several application types. Quantization and optimizations such as speculative decoding add more variables, and new models and hardware keep changing the comparison. imec is extending its benchmarking pipeline into a public dataset that can track these changes. Latré also describes an alliance intended to bring software teams interested in hardware optimization together with chip and infrastructure providers. He argues that software and hardware work needs to be studied more closely together.

"So often it's just a step wise function where you where it might be that you're choosing a specific model and that just taking a slightly better or slightly less performant model would have a huge impact on the hardware architecture of choice as well."09:46
Who should watch
  • You are deciding whether long-running coding agents should stay on hosted APIs or move to rented or owned GPUs.
  • You need to compare open-weight models and coding harnesses while accounting for concurrency, latency, and hardware cost.
  • Your team is building an internal AI platform and wants benchmark measurements that include GPU utilization and total cost of ownership.