# Hacking the Inference Pareto Frontier

Kyle Kranen, NVIDIA | AI Engineer World's Fair 2025 | 20:25

Source: https://www.youtube.com/watch?v=Y2qc0UhDSnc
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/hacking-the-inference-pareto-frontier
Published: 2025-08-01
Tags: cost, deployment, inference, latency

## TL;DR
- Inference deployment is constrained by quality, latency, and cost, and the right operating point depends on the application.
- Disaggregating prefill and decode lets teams match different workloads to different GPU pools, with Kyle Kranen reporting up to two times more tokens per second per GPU at fixed latency in one Llama 7B example.
- Routing, inference-time scaling, KV manipulation, worker specialization, and dynamic load balancing can change speed and cost while keeping model quality fixed.

## Summary
Kyle Kranen frames an inference system as a token factory with a Pareto frontier across quality, latency, and cost. The useful operating point depends on the application: cancer research may accept high cost and latency, while code completion needs very fast responses. He then explains how NVIDIA Dynamo supports techniques that reshape this frontier. Disaggregating prefill and decode allows separate GPU pools for compute-heavy and memory-heavy phases. KV-aware routing avoids repeating work, while inference-time scaling can let a smaller model approach the quality of a larger one through repeated queries. Agent structure also creates opportunities to retain KV data across tool calls and reduce round trips. Kranen closes with worker specialization and dynamic load balancing, since traffic patterns change and a fixed prefill/decode split can become inefficient. The talk is practical about configuration costs and workload limits, especially for disaggregation.

## Key ideas
### Deployment has three constraints, and the application decides their priority
[01:31](https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=91s)
Kranen evaluates a deployable system by quality, latency, and cost. Quality asks whether the application can complete its task accurately enough. Latency covers user experience and safety limits, such as robotics. Cost is the price per request relative to the application's margin requirements. The balance changes by use case. A personal cancer cure could justify millions of dollars and high latency if it works. Tab completion in an IDE must feel fast when the user presses Tab. An asynchronous coding agent can tolerate more latency, while quality and cost remain important. The target is usually one operating point on the frontier: the required quality and latency at the lowest cost.

### Common inference techniques move different parts of the frontier
[04:24](https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=264s)
Quantization can reduce latency and cost because smaller representations allow larger batch sizes. Retrieval-augmented generation usually raises quality while increasing latency and cost. Reasoning produces more tokens, which also changes the trade-off. Model configuration can shift speed, cost, and quality in different directions, including through parallelism choices. These techniques can be combined. Kranen gives the example of adding retrieval to improve quality, then adding quantization to recover some latency. The practical lesson is to treat deployment methods as a toolbox whose effects interact rather than as isolated switches.

### Disaggregation separates prefill and decode so each phase can use suitable resources
[06:04](https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=364s)
KV caching creates two phases in autoregressive generation. Prefill fills the cache for the input, while decode generates new cache entries and output tokens. Disaggregation places these phases on separate workers and GPU sets. Prefill is generally compute-bound, while decode can be memory-bound, so the deployment can use fewer GPUs with smaller batches for prefill and more GPUs with larger batches for decode. Separating them also avoids scheduling conflicts between new requests and requests already decoding. In a Llama 7B example using 16 H100 GPUs, Kranen reports up to two times the tokens per second per GPU at a fixed latency. The benefit depends on input length, latency, throughput, and configuration.

### Disaggregated deployments need careful prefill and decode balancing
[10:08](https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=608s)
The split between prefill and decode workers determines whether disaggregation helps. Too many decode workers can leave them waiting for work. Too many prefill workers can keep pushing work into the decode queue. The configuration space is wide because the balance also depends on each phase's parallelism settings. Kranen says disaggregation gives little speedup for low-input-length workloads because those requests are mostly decode. It is often useful for interactive applications in the range of roughly 20 to 200 tokens per second, while aggregated and disaggregated systems can converge at very low latency and throughput or at very high-latency, high-throughput points.

### KV-aware routing avoids repeating prefix work while controlling queueing
[11:05](https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=665s)
After disaggregation, KV data must move between machines, and previous work may be stored on a GPU, host memory, or external storage. Random routing ignores this locality. Routing only for the best KV prefix match can also send work to an overloaded worker. Kranen describes a smarter cost function that seeks a large prefix match while accounting for the worker's current load. As a deployment grows, more KV state is represented across the machines, which can raise the cache hit rate and reduce repeated prefill work. Routing changes speed and cost without changing quality because the model still performs the same computation.

### Inference-time scaling lets smaller models trade extra queries for quality
[13:00](https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=780s)
Kranen describes inference-time scaling as querying a model again and asking it to reconsider or reason further. In the example, an 8B model approaches the quality of a 49B model after about three or four re-queries, and the 49B model approaches the quality of a 235B model. Re-querying costs more speed and money, but it can be cheaper than using the larger model. If quality is held constant, a smaller model with repeated queries can provide lower latency and cost. Agent workloads make this easier to schedule because their usage patterns are moderately predictable. Making the router and scheduler aware of repeated work can also reduce round trips and raise throughput.

### Agent structure allows KV state to survive predictable tool calls
[16:16](https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=976s)
Tool calls create a period when a request may not need GPU execution. If a tool call takes around 30 seconds, its KV state may be evicted from high-bandwidth GPU memory before the next model call. When the system knows the state will be needed again, it can move the KV data to host memory and restore it shortly before the tool completes. That avoids doing the same prefill again. Kranen presents this as a use of workload structure: inference-time scaling and tool calling reveal when KV state will be reused. The technique can increase speed and reduce cost while keeping output quality unchanged.

### Specialized workers and dynamic balancing adapt the deployment to changing traffic
[17:43](https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=1063s)
Different input and output sequence lengths favor different worker designs. Shorter inputs with longer outputs may fit aggregated workers with higher tensor parallelism. Middle-range workloads may favor disaggregation, while long-context workloads may use disaggregation with context parallelism. The right mix varies by model. A fixed configuration can fail when the user distribution changes. If input lengths grow, the deployment may need more prefill capacity relative to decode capacity. Kranen therefore argues for autoscaling across the two worker types in real time. Dynamic load balancing is mainly what lets a disaggregated system keep working near its potential as demand changes.

## Notable quotes
- "The three things that I like to think about when I'm thinking about whether or not something can actually be deployed and used is really simple. It's quality, latency, and cost." (01:31)
- "The point I'm trying to make here is that you really have this toolbox of large sets of tools that you can use together." (05:36)
- "Disaggregation as a technique basically allows you to have these two phases which were typically used on the same set of GPUs onto multiple different workers and sets of GPUs." (06:58)
- "If we know that it's going to be used again, why not just offload it?" (16:47)
- "You actually have to do autoscaling across these two types of instances in real time to account for changes in user usage distribution of your platform." (19:26)

## Tools & references mentioned
- NVIDIA Dynamo
- NVIDIA
- Llama 7B
- DeepSeek
- Cursor
- Natural Plan dataset
- H100

## Who should watch
- You are choosing an inference architecture and need to reason about quality, latency, and per-request cost for a specific application.
- Your workload mixes long prompts, generation, agent steps, or tool calls, and you want concrete reasons to separate workers or preserve KV state.
- You run a disaggregated deployment whose traffic changes over time and need to decide how routing, worker specialization, and autoscaling should work.

## Related talks

- [Mastering LLM Inference Optimization From Theory to Cost Effective Deployment](https://aietalks.com/talks/mastering-llm-inference-optimization-from-theory-to-cost-effective-deployment) (Mark Moyou, NVIDIA, 33:39)
- [How fast are LLM inference engines anyway?](https://aietalks.com/talks/how-fast-are-llm-inference-engines-anyway) (Charles Frye, Modal, 16:07)
- [AI Engineering 201: Inference](https://aietalks.com/talks/ai-engineering-201-inference) (Charles Frye, Full Stack Deep Learning, 1:43:16)
- [GPUs & Inference Track](https://aietalks.com/talks/gpus-inference-track) (Santos, Covalent & Dylan Patel, SemiAnalysis & S Madra, Groq & Yen Vinko, Crusoe & Kait, Google DeepMind & Dima, Fireworks AI & Scott Wu, Cognition AI, 4:08:41)
- [System Design for Next-Gen Frontier Models](https://aietalks.com/talks/system-design-for-next-gen-frontier-models) (Dylan Patel, SemiAnalysis, 18:29)