Agentic workloads have long, variable sessions, high KV cache hit rates, and input-to-output ratios that make cache locality more valuable than steady-state throughput alone.
2
KV cache-aware routing can keep multi-turn requests on a pod that already holds the needed prefix, reducing the demonstrated request time from about three seconds to about one second.
3
Prefill and decode disaggregation reduces P99 inter-token latency when long prompts interfere with token generation, but it needs fast KV-cache transfer networking and is not the best choice for every workload.
Summary
Yuchen Fama and Ashish Kamra describe two ways to run inference for agentic workloads on Kubernetes. The first is KV cache-aware routing. Agent sessions can run for thousands of turns, reuse system prompts, and produce cache hit rates above 90 percent. Since cached and uncached tokens can have a 10x cost difference, llm-d routes requests toward pods with the right prefix and available capacity. A demo shows the same cached session taking about one second instead of three. The second technique is prefill and decode disaggregation. Prefill is compute-heavy and bursty, while decode is memory-bandwidth-heavy and sensitive to streaming latency. Running them in separate worker pools prevents long prompts from stalling token generation. Red Hat reports lower P99 inter-token latency on GPT-OSS with this design, especially in the middle concurrency range. The speakers also describe when aggregated serving is preferable, then apply both techniques to GLM 5.2 on H200 hardware.
Agentic sessions break the assumptions behind steady-state benchmarks
Ashish Kamra says public inference benchmarks usually report steady-state results from isolated and sanitized runs. Agentic workloads behave differently because they contain multi-turn interactions and large changes in context size. Yuchen Fama cites sessions ranging from a few turns to 3,000 turns. Agents often reuse system prompts and tool definitions, so cache hit rates frequently exceed 90 percent. Input-to-output ratios can exceed 100 to 1. Capacity planning therefore needs distributions and P90 values instead of simple averages. Fama also mentions subagent patterns that make scheduling harder.
Agentic inference makes KV-cache locality an economic concern
Fama describes KV-cache management as volatile because the client determines the context. The system may need frequent cache evictions and rewrites, while the inference engine has to coordinate with routing and scheduling layers. Prefix routing becomes important when latency is a primary scheduling measure. Fama also separates cached-token throughput from uncached-token throughput because the pricing difference can be large. Using Anthropic API pricing as an example, he says cached and non-cached tokens can differ by 10x. That makes sending a request to a pod with the right prefix more than a small performance optimization.
llm-d scores pods by both load and likely cache hits
The llm-d router has endpoint picker plugins that choose among inference pods. The endpoint picker probes pod metrics, including running and waiting requests, KV-cache utilization, and prefix-cache availability. It can then favor a pod with lower load and a higher chance of a cache hit. Fama also describes cache tiers for agentic sessions, including hot, warm, and cold data. The work includes offloading to NVMe SSD and file systems, KV-centric stores such as Mooncake, and eviction policies based on priority or session pinning. The aim is to keep session context available where the next request needs it.
A cached prefix can cut the demonstrated request time from three seconds to one
The demo sends a first request that populates the KV cache and takes roughly three seconds. The second turn keeps the same system prompt, routes to the same pod, reuses the cache, and takes about one second. A third request changes the system prompt, lands on a different pod, finds no matching KV cache, and again takes about three seconds. When the user prompt changes but the system prompt stays the same, the following turn reuses the cache and returns to about one second. Fama says this type of routing helps time to first token and throughput, but agentic workloads also need attention to inter-token latency.
Prefill and decode interfere because they use the GPU differently
Ashish Kamra explains that llm-d is a Kubernetes-native distributed inference framework with workload APIs and autoscalers for independently managing pods. In aggregated serving, one pod handles both prefill and decode. Prefill processes the initial prompt, builds the KV cache, and needs high compute and large-batch parallelism. Decode generates one token at a time, depends more on memory bandwidth and cache residency, and is sensitive to latency. A long incoming prompt can therefore stall ongoing token generation when both phases share a GPU. Disaggregation puts prefill and decode on separate, independently scalable inference workers.
Disaggregation transfers the prompt cache from prefill workers to decode workers
In llm-d, the gateway router evaluates cluster state and selects a prefill worker and a decode worker. The prefill worker processes the prompt and constructs its initial KV cache. It then produces metadata for the cache transfer. The selected decode worker pulls the computed KV cache across the network and continues token generation. This design separates the bursty, compute-heavy prompt phase from the latency-sensitive generation phase. The boundary also adds a systems requirement: the cluster needs a high-speed network fabric for moving KV data between workers. Without that fabric, the cost of disaggregation can outweigh its benefit.
PD disaggregation improves inter-token latency most in the middle concurrency range
For an internal GPT-OSS 12B result using 16 H100s, Kamra reports that aggregated serving has P99 inter-token latency around 900 milliseconds, with visible fluctuations. The disaggregated configuration reaches about 100 milliseconds and produces a smoother curve. In another comparison, KV-cache-aware routing improves an aggregated deployment before disaggregation is added. The concurrency graphs show that PD is similar to aggregated serving at lower and higher concurrency in one setup, while its largest advantage appears in the middle range. The speakers present this as a workload-dependent tradeoff rather than a universal replacement for aggregated serving.
The useful PD boundary depends on workload shape and network hardware
Kamra recommends considering PD for long contexts, high input-to-output ratios, large models that use model parallelism, middle concurrency levels, and strict inter-token streaming requirements. He recommends staying aggregated for short or moderate contexts, low concurrency, strict time-to-first-token requirements that aggregated serving can already meet, or clusters without the network fabric needed for KV transfers. RDMA or RoCE is named as the type of fabric required. He also says the prefill-to-decode ratio should change with traffic, so static worker ratios and independent autoscaling are insufficient for changing workloads.
A GLM 5.2 case study uses separate H200 pools for prefill and decode
Fama describes an ongoing GLM 5.2 deployment aimed at H200 clusters, since customers may not have access to B200 systems. The design uses up to three prefill workers for throughput and one dedicated decode worker for latency. NIXL handles KV transfer between the pools. The workers combine tensor, data, and expert parallelism, with a leader-worker set group. In an agentic workload with a 45 to 1 input-to-output ratio, the team reports four times faster time to first token and 60 percent more requests with the tested prefill and decode setup. Fama says the work is continuing with more prefill replicas and upper-layer tuning.
"What those benchmarks actually don't show you is the chaotic reality of multi-turn interactions, massive context fluctuations which are very typical of agentic workloads."00:32
Who should watch
You are operating multi-turn or agentic sessions where requests often reuse long prefixes and cache locality affects latency or cost.
You are deciding whether to add KV-cache-aware routing before buying more GPUs, and want to see the routing behavior in a concrete demo.
You are evaluating prefill and decode disaggregation and need to understand its concurrency trade-offs and networking requirements.
