Agent workloads should be measured by end-to-end task completion time rather than the latency of an individual model call.
2
Open weight models can complete useful coding-agent tasks at much lower cost than closed frontier models.
3
FriendliAI uses prefix caching, hierarchical KV-cache management, cache-aware routing, and agent-aware scheduling to reduce agent task latency.
Summary
Byung-Gon Chun argues that agent inference has different infrastructure needs from chat. A chat interaction is usually measured as one request and one response. An agent runs a task through repeated cycles of planning, tool use, observation, and more model calls. The context grows, steps share long prefixes, and sub agents may run in parallel. The useful measure is therefore the time needed to finish the task. Chun shows the same tower-defense coding task completed at a usable level by an open weight model and a closed model, while the open weight run costs much less. FriendliAI's serving stack is built around this workload. It caches shared prefixes, stores KV caches across GPU, host memory, and disk, routes requests to replicas that already hold the needed prefix, and schedules calls with awareness of the larger agent program. Chun also presents a customer split test in which FriendliAI was seven times faster with a lower error rate.
FriendliAI's research background shaped its inference focus
Chun introduces FriendliAI as an inference cloud for agents and describes its roots in research at Seoul National University. He says the team invented continuous batching, which is now standard across the industry, and that its Orca work inspired vLLM, a widely used open source serving framework. The company operates globally, with headquarters in San Francisco and a team in Seoul. Chun frames this background as the reason the company approaches agents through serving and inference optimization rather than only through model selection.
Open weight models can make useful agents much cheaper
Chun compares the same tower-defense game task run by a coding agent with GLM 5.2 on FriendliAI and Claude Opus 4.8. He says both outputs reached a clearly usable level, although they were not identical. The cost difference was large in his example: Opus 4.8 cost about $1.50, while GLM 5.2 on FriendliAI cost 27 cents. Chun presents this as evidence that open weight models have crossed a quality threshold for many agent workflows, while warning that model cost is only part of the total inference problem.
An agent task is a loop rather than a single request
Chun contrasts chat with agent workloads. In chat, a person asks a question, receives a response, and cares about the latency of that response. An agent runs a task through repeated planning, tool execution, observation, and context updates. The user cares when the whole task finishes. A task can involve many model calls and tool calls, and an agent can create sub agents that run in parallel. This means the serving system must optimize the program's end-to-end completion time rather than treating every model call as an independent request.
Agent contexts grow while consecutive steps reuse a large prefix
Chun shows internal coding-agent runs using GLM 5.2 and says their prompts and completions are much longer than typical chat inputs. Each observation is added to the context, so the context grows as the task continues. Consecutive agent steps usually share a huge prefix. Recomputing that prefix on every call wastes compute on work that has already been done. In longer tasks such as deep research, multiple stages can contain sub agents, tens or hundreds of inference steps, and tool calls spread over minutes or hours.
Prefix caching and KV-cache management reduce repeated work
FriendliAI computes the key-value state for a shared prefix once and reuses it on later steps, processing only the new suffix. Chun says reading from the cache is cheaper than recomputing the prefill, which improves time to first token and reduces compute as the task continues. The cache must also fit and move efficiently. FriendliAI uses memory management to pack more active context onto each GPU, KV-cache quantization to reduce memory use, hierarchical caching across GPU memory, host memory, and disk, and distributed caching so a prefix can be served across replicas.
Cache-aware routing preserves locality across replicas
A naive load balancer can distribute requests evenly across GPU clusters while losing the prefix cache that a later request needs. Chun says a cache-aware router instead sends a request to a replica that already holds the relevant prefix. That can turn a full prefill into a cache hit. The router still has to balance load so that one replica does not become a hotspot. In his example, two requests belonging to task A go to the same replica to preserve cache locality.
Agent-aware scheduling uses the larger program as context
Chun says most systems schedule each model call as if it were independent. An agent-aware optimizer can recognize that a call belongs to a longer program and make different decisions. Examples include preempting the right work, speculatively prefilling context for an unlikely next step, and choosing cache evictions using information about the agent-level task. The aim is to reduce total agent latency rather than make one isolated call appear fast.
The serving stack affects the economics of open models
Chun compares the same mobile-game task, using GLM 5.2 with Kilo Code, through FriendliAI and another inference provider. He says FriendliAI completed the task end to end faster because of its agent-centric cloud design. He then describes three deployment options: a serverless model API, dedicated endpoints with isolated deployments and production SLAs, and bringing your own GPU to run the stack on private infrastructure. He cites a Kilo Code split test in which FriendliAI was seven times faster than other providers and direct model-lab usage, with a significantly lower error rate.
