LLM inference becomes expensive because model weights stay fixed while the KV cache grows with context length and concurrent users.
2
Prefill is compute-bound and determines time to first token, while decode is memory-bound because each new token repeatedly reads model weights and prior KV vectors.
3
Serving techniques such as paged attention, continuous batching, prefix caching, and KV quantization improve memory use and throughput, but engine choice depends on the workload.
Summary
Harshul Jain and Tanmay Sah explain LLM inference from the request pipeline down to GPU memory movement. They show why longer contexts increase memory use and time to first token, while naive sequential serving limits throughput. The central calculation is the KV cache: for Mistral 7B, one token uses about 131 KB, so context length and concurrency quickly consume available GPU memory after model weights are loaded. The workshop then separates model changes from serving changes. Tanmay covers quantization, attention variants, and FlashAttention. Harshul covers paged attention, continuous batching, prefix caching, and KV quantization. Their benchmarks show large gains over a basic Hugging Face baseline from a default vLLM setup. They report no statistical difference between vLLM and SGLang on standard workloads, while SGLang performed three to four times better in their agentic branching test. The talk is practical, with slides, notebooks, and a capacity calculator in the repository.
Jain frames inference around memory, time to first token, and throughput. Memory rises as the input context grows. Time to first token, or TTFT, also gets slower with longer context. A basic local implementation handles requests sequentially, so five requests wait behind one another instead of running together. Throughput measures both tokens served per second and users served per second. These symptoms provide the starting point for understanding the rest of the inference pipeline.
Attention projects tokens into query, key, and value vectors. For Mistral 7B, Jain calculates a KV size of about 131 KB per token. A 4K context therefore needs about half a gigabyte of KV storage, while a 16K context needs about 2.1 GB for one request. With 80 users and a 4K context, the cache can need about 42 GB. GPU memory must first hold model weights and fixed overhead, then uses the remaining space for KV vectors, so context length and concurrency directly compete for capacity.
Prefill determines first-token latency, while decode is memory-bound
During prefill, the system processes the input tokens, builds their key and value vectors, and computes attention. This is a compute-heavy phase, and its duration becomes TTFT. Decode then generates tokens one at a time. Each step computes attention for the new token while reading the model weights and the previous tokens' KV vectors. The work has lower arithmetic intensity, so data movement from high-bandwidth memory limits the rate. The speaker calls this phase memory-bound.
Serving capacity is a tradeoff between quality, latency, and throughput
With a fixed model and GPU, KV size per token is fixed. Increasing concurrent users therefore requires reducing the context length, which can affect quality. More users and larger batches also increase inter-token latency and TTFT. Jain describes a tradeoff triangle between quality, latency, and throughput. A premium chat service may favor quality and low latency, while an asynchronous agent workload may favor quality and serving more concurrent tasks.
Model compression trades memory for possible quality and speed changes
Tanmay Sah uses quantization to fit larger models into limited GPU memory. He gives a 120-billion-parameter model with 240 GB of BF16 weights as an example, then reduces it to about 120 GB with FP8 and about 65 GB with MXFP4. For Mistral 7B, the workshop compares FP16 with INT8 and INT4-style compression. In their demo, INT8 reduces model memory from about 15 GB to about 7.5 GB, leaving more room for KV cache and therefore more context or concurrent users. Sah says quality and throughput still need to be tested.
Attention variants reduce the amount of KV state that must be stored
Sah presents multi-head attention, multi-query attention, grouped-query attention, and multi-head latent attention as different ways to reduce attention cost or KV storage. Multi-head attention keeps separate blocks, while multi-query attention shares one block across queries. Grouped-query attention takes a middle position by sharing groups of heads. Latent attention compresses the key and value information into a latent vector and reconstructs it. He also mentions sparse attention, linear attention, Mamba, and diffusion-style generation as alternative directions.
FlashAttention improves data movement by tiling the calculation
The workshop describes FlashAttention as dividing large matrix operations into smaller tiles. Instead of repeatedly moving entire matrices between high-bandwidth memory and shared memory, the system places smaller tiles in faster memory, performs the calculation, and tracks the values needed for the online softmax. The goal is to reduce memory movement while retaining the attention calculation. Sah also corrects an earlier estimate during the workshop, saying their multi-head latent attention comparison showed about 14x savings rather than 50x or 56x.
Serving optimizations manage KV memory and keep the GPU busy
Paged attention avoids allocating one large contiguous memory region for every request. It divides KV storage into blocks and maps logical token positions to physical blocks, following the idea of operating-system paging. Continuous batching admits new requests as earlier requests finish instead of waiting for an entire batch to complete. Prefix caching reuses shared prompt computation across requests. KV quantization stores the cache with fewer bits, leaving space for more tokens. Jain presents these as separate serving-side controls that can be combined in an inference engine.
Engine choice changes when requests branch agentically
In the workshop's standard workload test on an H100, vLLM and SGLang showed no statistical difference in request rate, TTFT, and latency. The agentic test repeatedly asked a model to produce an answer and then review and rate it. In that branching workload, the speakers report that SGLang was three to four times better in their setup. Their practical recommendation is to use vLLM as the default for standard API traffic, then test SGLang when agentic branching is central to the application.
