A model can answer correctly and still be unusable once real prompts, concurrent users and a GPU bill arrive. Token generation first has to be connected to memory bandwidth, batching and KV cache. Those mechanics become the measurements that matter under load: time to first token, inter-token latency and completed requests. SGLang shows why a promising optimization can fail when its batch settings or prompts do not match production. TensorRT-LLM makes the other bargain, compiling an engine for a known GPU and workload. With the single-server choices clear, an agent request can be followed across a cluster: keep useful cache near the next turn, split prefill from decode when the traffic justifies it, and route work by the cost of finishing the task rather than the token price.
Why first: Frye supplies the vocabulary for the rest of the pack: tensor operations, memory bandwidth, batching, cold starts and the choice between an API and self-hosting. His broad map reaches the GPU, where the next talk looks more closely at why some inference workloads keep the hardware busy and others leave it waiting.
Why second: Frye first names memory movement as the constraint. Here he explains the hardware behind it: GPUs favor bandwidth, concurrency and matrix-matrix work over one low-latency operation at a time. Moyou follows by turning those preferences into concrete measurements for a serving workload.
Why here: Frye shows what the GPU wants. Moyou shows what an operator must record: input and output lengths, KV cache pressure, time to first token and the delay between later tokens. Those measurements give Frye's next talk the workload definition it needs to compare engines honestly.
Why here: Moyou says to size an engine around observed traffic. Frye demonstrates the benchmark: expose enough parallel work to find maximum throughput, then sweep back toward low load to see what latency costs. Kiely and Zhang then take that method into SGLang, where a single flag can decide whether decode uses the fast path.
Why here: Frye establishes what an engine benchmark must preserve. Kiely and Zhang show why runtime settings cannot be copied blindly: CUDA graph limits and speculative decoding depend on batch size and representative prompts. Gupta follows with the more committed option of compiling an engine for a known GPU and workload envelope.
Why here: SGLang exposes a runtime that can be adjusted while it serves. Kiely and Gupta show the other end of that choice: TensorRT-LLM builds around a specific GPU, batch range and sequence length, trading portability for optimization. Dzhulgakov then widens the view from one compiled endpoint to the production service around it.
Why here: Gupta gets model weights behind an API. Dzhulgakov accounts for the work that remains: sourcing GPUs, matching settings to prompt length, sharing adapters, monitoring the service and minimizing cost within a latency target. Kranen takes that target and makes the quality, latency and cost trade explicit.
Why here: Dzhulgakov tunes one serving stack to a workload. Kranen shows how the operating point moves when prefill and decode use separate workers, requests follow their cached prefixes, or a smaller model gets another attempt. Fama and Kamra follow with the Kubernetes machinery and workload limits behind those choices.
Why here: Kranen introduces cache-aware routing and prefill/decode separation. Fama and Kamra show the request path, the cache transfer and the latency evidence, then draw the boundary where an aggregated server is still better. The routing panel closes the pack by moving the same cost question from workers to models.
Why last: Fama and Kamra route a request to the worker that already holds useful KV state. Khalil, Yan, Atallah, Varshney and Abdallah ask which model should receive the work at all. Their answer ends the pack at the product boundary: preserve useful context, notice when a cheap model is thrashing, and price the completed task rather than the token.