Open-weight models such as GLM 5.2 now compete with leading proprietary models on coding, reasoning, and agent benchmarks.
2
The same model and coding agent can feel much faster on one provider because caching, routing, scheduling, quantization, and GPU configuration affect the full inference stack.
3
Reliable inference is part of speed for agents, since one failed model call can invalidate an entire multi-call workflow.
Summary
Yunmo Koo argues that open-weight models have become good enough for many production workloads, so model choice is only one part of the decision. In his tower-defense experiment, GLM 5.2 and Claude Opus 4.8 produced similar working games from one prompt, while GLM 5.2 cost more than 5.6 times less. A second demonstration used the same model and coding agent on different providers. FriendliAI built a Candy Crush-style game in about two minutes, while another provider took more than three. Koo explains the difference through the inference stack: cache-aware routing, a GPU, CPU, and NVMe cache hierarchy, and hybrid speculative decoding. He also treats error rates, timeouts, and uptime as performance concerns because agents make many calls. His advice is to test a provider against the prompts, traffic, token lengths, and latency needs of the application itself.
Open-weight models have crossed the threshold for many workloads
Koo says he is moving more of his work to open-weight models because they now perform in the same tier as leading proprietary models on coding, reasoning, and agentic benchmarks. He points to GLM 5.2 ranking above Claude Opus 4.8 on the Code Arena leaderboard, with Claude Opus 4.8 below Claude Fable 5. He also names MiniMax M3 as a model with strong results across coding, terminal, and tool-use benchmarks. The practical question has changed from whether these models are good enough to how fast and reliably a provider can run them in production.
A cheaper open-weight model can produce a similar result
Koo asked GLM 5.2 and Claude Opus 4.8 to build a tower defense game from a single prompt. Both produced working implementations with a map, turrets, and enemies. He describes the structure and quality as similar, then compares their cost. GLM 5.2 was more than 5.6 times cheaper for the task. Koo says the difference compounds in agent-style workflows that require repeated iterations, so the cost of running a model matters alongside its quality when a workload must operate at scale.
The provider changes the experience even when the model stays the same
Koo compares the same GLM 5.2 model and the same coding-agent workflow on two providers. The task was to build a simple Candy Crush-style mobile game called Moto Crush. FriendliAI generated the game in about two minutes, while another provider took more than three minutes. The model, agent application, and task were unchanged. Koo says users experience the combination of the model, developer tools, and coding agent, so provider performance can make the same underlying model feel substantially different.
Inference performance comes from the whole serving stack
Koo attributes the provider gap to the full inference technology stack. He names model architecture, the cache system, routing policy, quantization strategy, and GPU configuration as factors that shape what developers experience. He describes FriendliAI as a full-stack inference engine rather than only a model API. Batching, scheduling, quantization, speculative decoding, and hardware configuration must work together. His point is that there is no single optimization that explains production speed.
Prefix caching removes repeated work from long agent inputs
Modern agent requests can exceed 10,000 tokens, while their outputs are relatively short. Koo says that 50% to 90% of the input tokens can be identical across requests in real workloads. Most of the compute cost then sits in prefill, when the model processes the input and builds the KV cache. If a later request shares a prefix with an earlier one, the provider can reuse the cached tensors instead of recomputing that prefix. This reduces time to first token and avoids repeated GPU work.
Cache-aware routing and tiered memory keep prefixes available
KV caches are held by individual replicas in a distributed system, so a router needs to send a request to a replica that already has the relevant prefix. Otherwise, the system loses the reuse benefit. Koo pairs this routing policy with a cache hierarchy because KV caches for large models and long contexts can exceed the model weights in size. Frequently used data stays in GPU memory, less active data moves to CPU RAM, and colder data can be offloaded to NVMe or disk. The system brings blocks back when needed.
Hybrid speculative decoding adapts to the workload
Autoregressive decoding generates one token per forward pass, which makes decoding sequential and can leave the GPU underused. Speculative decoding lets a draft process propose several future tokens, then has the target model validate them in one pass. Accepted tokens avoid separate generation work. Koo describes model-based speculation, which uses a small draft model and can have higher acceptance at extra compute cost, and model-free speculation, which uses patterns in the prompt or prior output. FriendliAI dynamically chooses between them based on request characteristics and adapts its draft model to live serving traffic.
Reliability affects the speed an agent actually experiences
A fast average response does not help if requests often hit timeouts, 5xx errors, 429 rate limits, or other failures. Koo says this matters more for agents because one task can require many model calls, and a single failed call in the middle can invalidate the workflow. FriendliAI therefore measures latency and throughput together with error rate and uptime. He cites an OpenAI snapshot for GLM 5.1 Pro that shows FriendliAI with over 99.9% uptime, then advises teams to test their own prompts, token lengths, traffic patterns, and latency expectations.