Inference decoding is mostly limited by memory bandwidth because each generated token requires reading model weights and the KV cache.
2
Splitting prefill and decode across different workers lets teams scale compute-heavy and memory-heavy workloads independently.
3
Disaggregating work across different silicon, then using agents to optimize kernels, can improve speed for specific models and hardware.
Summary
Daniel Kim and Michelle explain how inference systems have evolved to handle growing token demand. Daniel starts with runtime techniques. Autoregressive decoding spends much of its time waiting for memory, so systems use model parallelism, batching, continuous batching, paged KV caches, prompt caching, and chunked prefill. Michelle then moves to system-level scheduling, where workloads differ by their mix of prompt processing and token generation. Prefill and decode can run in separate pools, and speculative decoding can reduce the number of full-model passes. Gimlet Labs extends this approach across Nvidia, AMD, Trainium, Cerebras, D-Matrix, and Intel Gaudi hardware. The speakers also describe attention and FFN disaggregation, which maps different parts of a model to hardware suited to their resource needs. They finish with agents that generate optimized kernels, tune software for new hardware, and potentially help design custom silicon. The talk is practical about tradeoffs: there is no universally best hardware or disaggregation strategy.
Autoregressive decoding spends most of its time waiting for memory
Daniel Kim explains that LLMs generate tokens one at a time in an autoregressive loop. To produce one more token, the system must read the model weights from memory into compute. He says the computer can spend up to 90% of its time waiting for memory to stream in the weights. Splitting a model across four GPUs can reduce the amount of model each GPU reads, but the GPUs must repeatedly combine their partial results with all-reduce operations. Daniel says this can turn a theoretical four-times speedup into only a two-times speedup because communication adds overhead. Adding GPUs improves speed, but it also changes the energy and throughput tradeoff.
Batching improves throughput until KV cache memory becomes the limit
Batching reuses a weight read across several user requests. Continuous batching goes further by inserting a new request into idle slots when another request has finished or is waiting. The cost is that every generated token adds information to the KV cache. Daniel gives the example of a 128K-context coding agent needing 42 GB of KV cache for one request. For a 70-billion-parameter model, his rule of thumb is 1 GB of additional KV cache for every 3,000 tokens. Several concurrent requests can therefore exhaust memory even when the compute hardware has capacity.
Traditional KV-cache implementations reserve physical memory for each request, including space that may never be used. Daniel compares the alternative to virtual memory in an operating system. A paged KV cache provisions virtual blocks and places each request's KV data wherever free space exists. He says this reduces unused reserved space and allows a system to serve three times more requests than the earlier approach. Prompt caching addresses repeated work in long system prompts. A coding agent or chat application can reuse a previously calculated KV cache instead of running the same prefill again.
Prefill and decode have different hardware requirements
Prefill processes the incoming prompt and produces the first output token. It is highly parallel and compute-bound. Decode then produces one token at a time while reusing the KV cache, which makes it more dependent on memory. Long-context coding agents can make this distinction a scheduling problem. A very long prefill can block other prefill and decode requests, so chunked prefill divides it into smaller pieces. The system can then interleave those pieces with other requests instead of allowing one long prompt to clog the entire pipeline.
Disaggregating prefill and decode lets systems scale each phase separately
Michelle describes prefill-decode disaggregation as a system-level way to separate compute-heavy and memory-heavy work. A prefill pool handles prompt processing, while a decode pool generates output tokens. A summarization workload with a long prompt and short answer can use more prefill capacity and less decode capacity. A workload that generates a long story has the opposite balance. Separate pools prevent one phase from stalling the other and allow each resource group to be scaled according to the workload.
Speculative decoding uses a smaller draft model to propose several candidate tokens. The full model then verifies those candidates and keeps the ones that match its output. Michelle gives an example where standard decoding needs nine full-model forward passes for nine tokens, while speculative decoding needs only three full-model passes, assuming the draft model runs quickly. The technique moves the speed-throughput curve because the expensive full model does less sequential work. The choice of hardware still matters, since some systems can run small draft models particularly quickly.
Different model operations can be mapped to different silicon
Attention-FFN disaggregation separates the attention layer from the feedforward network inside a model. Michelle uses very large models as the motivating case, where even one layer can consume substantial resources. Attention performs many KV-cache accesses and is more memory-bound. The FFN performs dense matrix multiplications and benefits from hardware that handles those operations quickly. A system can send each part to a suitable machine, even though inference then moves across several machines within a model layer. Michelle is direct that no hardware or disaggregation strategy is best for every workload.
Agents can search for kernels tuned to a model, workload, and device
Gimlet Labs uses KForge, an agent that generates optimized kernels. Kernel performance depends on the model, hardware, hardware generation, and workload shape, so hand-tuning every combination does not scale easily. KForge takes hardware and workload profiles and treats optimization as a brute-force search. Michelle says that, in less than a week, it improved GPT-OSS-120B end-to-end performance on AMD hardware by 33%, building on kernels that had already been optimized. The speakers extend the idea to custom silicon, compilers, model bring-up, operations, procurement, and other work around inference.
"The idea here is taking a very long piece of prefill, chopping it up into smaller bits, so you can interleave in different decode and prefill requests."09:18
Who should watch
You are choosing between GPUs or other accelerators for inference and need to understand which parts of a workload are compute-bound or memory-bound.
You are operating long-context agents and want practical techniques for KV-cache use, prompt reuse, batching, and prefill scheduling.
You are building an inference platform across several kinds of silicon and want to evaluate disaggregation, speculative decoding, or automated kernel tuning.