Frontier-model inference has two different workloads: prefill is compute-heavy, while decode repeatedly loads model weights and is limited by memory bandwidth.
2
Serving large models at a reasonable cost requires continuous batching, disaggregated prefill, and context caching rather than simple batch-size-one inference.
3
Future training clusters will contain hundreds of thousands of GPUs, creating difficult problems with power, networking, optical failures, and uneven chip performance.
Summary
Dylan Patel explains why running the next generation of frontier models will be a systems problem as much as a model problem. Prefill processes the prompt and demands substantial compute. Decode generates tokens one at a time and repeatedly reads the model weights, making memory bandwidth the main constraint. A large model such as Llama 3 405B therefore needs serving software that supports continuous batching and separates prefill from decode. Patel also describes context caching as a way to avoid paying to process the same long document repeatedly. The talk then moves to future training infrastructure. Labs are building 100,000-GPU clusters and planning even larger systems, with power use, cross-building networks, optical link failures, and slow GPUs all reducing useful throughput. Patel is direct about the gap between today's open-source tooling and the systems already used by major labs.
Frontier models have not yet reached the next scale of capability
Patel argues that apparent model stagnation comes partly from comparing current systems with models trained in 2022. GPT-4, GPT-4 Turbo, and GPT-4o are described as smaller models trained for longer, while Claude 3.5 Sonnet is smaller than Claude 3 Opus but performs better after longer training. He expects a much larger model to arrive soon. GPT-4 is described as having roughly 1.8 trillion parameters and as being very expensive to run. Patel says that, within about a year, a model of that size may already be considered from the previous generation. His focus is inference because most builders will need to run these models rather than train them.
Prefill and decode place different demands on hardware
Inference has two stages. Prefill processes the user's prompt, and decode generates each output token iteratively, feeding each new token back into the model. Patel gives the example that a 2,000-token prompt takes about a petaflop of processing, while a 32,000-token prompt takes about 20 petaflops. Decode has a different bottleneck because the system must load the model's weights for every generated token. For a 1.8-trillion-parameter model, serving 64 users at 30 tokens per second would require about 60 terabytes per second of memory bandwidth. An H100 has about 3 terabytes per second, which shows why this is a difficult hardware and systems problem.
Patel says decode uses relatively little compute compared with prefill, which helps explain the price difference between input and output tokens in commercial APIs. He gives contemporary pricing examples of $5 per million input tokens and $15 per million output tokens. Running at batch size one in the cloud is wasteful because the accelerator is not kept busy. He estimates that it can cost 10 times more, and possibly 10 to 100 times more, than running with a high batch size. Since requests arrive at different times, a server needs continuous batching. It must add new users while existing requests are still generating tokens.
Open-source serving needs systems support that simple local runtimes lack
Patel uses Llama 3 405B as an example of a model that will expand what open-source builders can run. He says Llama.cpp is not enough to serve it effectively. Builders may need libraries such as TensorRT-LLM, which works on Nvidia hardware, or vLLM, an open-source library that supports AMD and Intel and may support other chips. Continuous batching is one missing capability he calls out for some software. Builders can implement it themselves or contribute to an open-source project. Without these serving features, the cost of renting GPUs makes large-model inference impractical.
Disaggregated prefill protects latency and service quality
Because prefill and decode stress hardware differently, high-volume services can place them on separate accelerator pools. One pool handles the compute-intensive prefill, then passes the request to another pool for bandwidth-intensive decode. Patel says Google has publicly described doing this, and he believes OpenAI and Anthropic are also doing it. The design also limits the effect of noisy neighbors. A customer who submits 10,000 database rows for an LLM query could otherwise damage the performance of other users. Patel connects this to time to first token, variable tokens per second, service-level agreements, and reliability. Rate limits alone are an annoying and incomplete answer once a service is open to arbitrary workloads.
Context caching can reduce the cost of long enterprise prompts
Patel presents Google's context caching for Gemini 1.5 Pro as an alternative to fine-tuning when the model is expensive or unavailable for fine-tuning. A long document can be placed into the model's context, and the resulting KV cache can be reused instead of recomputed for every query. He says Google offered a context length of about 2 million tokens at the time. This matters for legal, contract-review, and similar enterprise applications, where the document sent as input can dominate API cost while the answer is short. The cache needs a great deal of memory, so it is stored in CPU memory or storage rather than GPU memory. He says vLLM was building open-source support for this approach.
Very large training clusters create failures that cannot be treated as exceptions
Patel says labs are building clusters with 100,000 chips, and that a cluster of this size could train GPT-4 in three days. He describes possible future models with tens of trillions of parameters and says such systems could require hundreds of terabytes per second of memory bandwidth. Microsoft's Arizona data center is described as having about 100,000 GPUs and using 150 megawatts. Elon Musk has discussed a 300,000-GPU cluster, whose electricity could cost about $500 million per year. At this scale, optical links are a serious operational problem. Patel estimates that a 100,000- or 500,000-GPU cluster could experience a component failure about every five minutes.
A single slow GPU can reduce the speed of a synchronous training job
Training is synchronous: the GPUs process data, update weights, and exchange gradients before continuing. Hardware is not identical, even when it has the same product name. Patel calls the variation the Silicon Lottery. He cites a ByteDance paper that found a 25 percent speed decrease caused by one GPU that technically worked but ran more slowly than expected in a 20,000-GPU cluster. Removing that GPU produced a large performance improvement. The example shows why future systems need to detect and manage stragglers, not only replace components that fail completely.
"When you connect these chips together, there's a lot of optics."14:42
Who should watch
You are deciding how to serve a large open-source model and need to understand why batch size, memory bandwidth, and request scheduling affect cost.
You are building an API for long documents, such as contracts or legal records, and want to know why context caching may matter more than fine-tuning.
You work on training infrastructure and need concrete examples of the failures and bottlenecks that appear in clusters with tens or hundreds of thousands of GPUs.
