A model can answer correctly and still be unusable once real prompts, concurrent users and a GPU bill arrive. Token generation first has to be connected to memory bandwidth, batching and KV cache. Those mechanics become the measurements that matter under load: time to first token, inter-token latency and completed requests. SGLang shows why a promising optimization can fail when its batch settings or prompts do not match production. TensorRT-LLM makes the other bargain, compiling an engine for a known GPU and workload. With the single-server choices clear, an agent request can be followed across a cluster: keep useful cache near the next turn, split prefill from decode when the traffic justifies it, and route work by the cost of finishing the task rather than the token price.
Why first: Frye supplies the vocabulary for the rest of the pack: tensor operations, memory bandwidth, batching, cold starts and the choice between an API and self-hosting. His broad map reaches the GPU, where the next talk looks more closely at why some inference workloads keep the hardware busy and others leave it waiting.
Why second: Frye first names memory movement as the constraint. Here he explains the hardware behind it: GPUs favor bandwidth, concurrency and matrix-matrix work over one low-latency operation at a time. Moyou follows by turning those preferences into concrete measurements for a serving workload.
Why here: Frye shows what the GPU wants. Moyou shows what an operator must record: input and output lengths, KV cache pressure, time to first token and the delay between later tokens. Those measurements give Frye's next talk the workload definition it needs to compare engines honestly.
Why here: Moyou says to size an engine around observed traffic. Frye demonstrates the benchmark: expose enough parallel work to find maximum throughput, then sweep back toward low load to see what latency costs. Chandrasekar and Kramberger then check whether the benchmark client actually delivered the load it reports.
Why here: Frye sweeps load to find maximum throughput. Chandrasekar and Kramberger show that the harness running the sweep can fail quietly: one asked for 200 QPS and delivered 38, another added 58 seconds of client-side latency, and a temperature of zero produced a 20 percent throughput gain. Kiely and Zhang follow with the same lesson inside SGLang, where settings and prompts must match production.
Why here: Chandrasekar and Kramberger make the benchmark itself trustworthy. Kiely and Zhang show why runtime settings cannot be copied blindly: CUDA graph limits and speculative decoding depend on batch size and representative prompts. Gupta follows with the more committed option of compiling an engine for a known GPU and workload envelope.
Why here: SGLang exposes a runtime that can be adjusted while it serves. Kiely and Gupta show the other end of that choice: TensorRT-LLM builds around a specific GPU, batch range and sequence length, trading portability for optimization. Dzhulgakov then widens the view from one compiled endpoint to the production service around it.
Why here: Gupta gets model weights behind an API. Dzhulgakov accounts for the work that remains: sourcing GPUs, matching settings to prompt length, sharing adapters, monitoring the service and minimizing cost within a latency target. Kranen takes that target and makes the quality, latency and cost trade explicit.
Why here: Dzhulgakov tunes one serving stack to a workload. Kranen shows how the operating point moves when prefill and decode use separate workers, requests follow their cached prefixes, or a smaller model gets another attempt. Fama and Kamra follow with the Kubernetes machinery and workload limits behind those choices.
Why here: Kranen introduces cache-aware routing and prefill/decode separation. Fama and Kamra show the request path, the cache transfer and the latency evidence, then draw the boundary where an aggregated server is still better. Lao and Zhang then take cache-aware routing from one cluster to engines spread across regions.
Why here: Fama and Kamra send a turn to the pod that holds its cache. Lao and Zhang show what breaks at OpenAI's scale: a feedback loop that adjusted routing weights could swing traffic back and forth between engines and destroy the cache locality it was meant to protect. Their replacement computes routing policy from a global view, and can send work to a farther engine with spare capacity. The routing panel closes the pack by moving the same cost question from engines to models.
Why last: Lao and Zhang choose which engine should receive a request. Khalil, Yan, Atallah, Varshney and Abdallah ask which model should receive the work at all. Their answer ends the pack at the product boundary: preserve useful context, notice when a cheap model is thrashing, and price the completed task rather than the token.