# Inference: serving models efficiently A pack of 10 talks from the AI Engineer YouTube channel, in the order to watch them. 7h 06m of video. Page: https://aietalks.com/packs/inference A model can answer correctly and still be unusable once real prompts, concurrent users and a GPU bill arrive. Token generation first has to be connected to memory bandwidth, batching and KV cache. Those mechanics become the measurements that matter under load: time to first token, inter-token latency and completed requests. SGLang shows why a promising optimization can fail when its batch settings or prompts do not match production. TensorRT-LLM makes the other bargain, compiling an engine for a known GPU and workload. With the single-server choices clear, an agent request can be followed across a cluster: keep useful cache near the next turn, split prefill from decode when the traffic justifies it, and route work by the cost of finishing the task rather than the token price. ## This pack is for you if - Your model works in a demo, but latency or GPU cost makes the production numbers uncomfortable. - You are choosing an inference engine and need to benchmark the request shapes your users actually send. - Long prompts, agent turns or uneven traffic are causing cache misses, queueing or unpredictable response times. ## The talks, in order ### 1. AI Engineering 201: Inference Charles Frye, Full Stack Deep Learning | 1:43:16 | AI Engineer Summit 2023 Video: https://www.youtube.com/watch?v=N7lJY5IKVLE Summary: https://aietalks.com/talks/ai-engineering-201-inference.md Why first: Frye supplies the vocabulary for the rest of the pack: tensor operations, memory bandwidth, batching, cold starts and the choice between an API and self-hosting. His broad map reaches the GPU, where the next talk looks more closely at why some inference workloads keep the hardware busy and others leave it waiting. ### 2. What Every AI Engineer Needs to Know About GPUs Charles Frye, Modal | 19:52 | AI Engineer World's Fair 2025 Video: https://www.youtube.com/watch?v=y-UGrYbJsJk Summary: https://aietalks.com/talks/what-every-ai-engineer-needs-to-know-about-gpus.md Why second: Frye first names memory movement as the constraint. Here he explains the hardware behind it: GPUs favor bandwidth, concurrency and matrix-matrix work over one low-latency operation at a time. Moyou follows by turning those preferences into concrete measurements for a serving workload. ### 3. Mastering LLM Inference Optimization From Theory to Cost Effective Deployment Mark Moyou, NVIDIA | 33:39 | AI Engineer World's Fair 2024 Video: https://www.youtube.com/watch?v=9tvJ_GYJA-o Summary: https://aietalks.com/talks/mastering-llm-inference-optimization-from-theory-to-cost-effective-deployment.md Why here: Frye shows what the GPU wants. Moyou shows what an operator must record: input and output lengths, KV cache pressure, time to first token and the delay between later tokens. Those measurements give Frye's next talk the workload definition it needs to compare engines honestly. ### 4. How fast are LLM inference engines anyway? Charles Frye, Modal | 16:07 | AI Engineer World's Fair 2025 Video: https://www.youtube.com/watch?v=DeFF3J8T5Pk Summary: https://aietalks.com/talks/how-fast-are-llm-inference-engines-anyway.md Why here: Moyou says to size an engine around observed traffic. Frye demonstrates the benchmark: expose enough parallel work to find maximum throughput, then sweep back toward low load to see what latency costs. Kiely and Zhang then take that method into SGLang, where a single flag can decide whether decode uses the fast path. ### 5. Introduction to LLM serving with SGLang Philip Kiely & Yineng Zhang, Baseten | 43:42 | AI Engineer World's Fair 2025 Video: https://www.youtube.com/watch?v=Ahtaha9fEM0 Summary: https://aietalks.com/talks/introduction-to-llm-serving-with-sglang.md Why here: Frye establishes what an engine benchmark must preserve. Kiely and Zhang show why runtime settings cannot be copied blindly: CUDA graph limits and speculative decoding depend on batch size and representative prompts. Gupta follows with the more committed option of compiling an engine for a known GPU and workload envelope. ### 6. From Model Weights to API Endpoint with TensorRT-LLM Philip Kiely & Pankaj Gupta, Baseten | 1:40:01 | AI Engineer World's Fair 2024 Video: https://www.youtube.com/watch?v=Lko9lTGD_9U Summary: https://aietalks.com/talks/from-model-weights-to-api-endpoint-with-tensorrt-llm.md Why here: SGLang exposes a runtime that can be adjusted while it serves. Kiely and Gupta show the other end of that choice: TensorRT-LLM builds around a specific GPU, batch range and sequence length, trading portability for optimization. Dzhulgakov then widens the view from one compiled endpoint to the production service around it. ### 7. Customized, production-ready inference with open source models Dmytro (Dima) Dzhulgakov, Fireworks.ai | 18:55 | AI Engineer World's Fair 2024 Video: https://www.youtube.com/watch?v=ePMvfa8vgL8 Summary: https://aietalks.com/talks/customized-production-ready-inference-with-open-source-models.md Why here: Gupta gets model weights behind an API. Dzhulgakov accounts for the work that remains: sourcing GPUs, matching settings to prompt length, sharing adapters, monitoring the service and minimizing cost within a latency target. Kranen takes that target and makes the quality, latency and cost trade explicit. ### 8. Hacking the Inference Pareto Frontier Kyle Kranen, NVIDIA | 20:25 | AI Engineer World's Fair 2025 Video: https://www.youtube.com/watch?v=Y2qc0UhDSnc Summary: https://aietalks.com/talks/hacking-the-inference-pareto-frontier.md Why here: Dzhulgakov tunes one serving stack to a workload. Kranen shows how the operating point moves when prefill and decode use separate workers, requests follow their cached prefixes, or a smaller model gets another attempt. Fama and Kamra follow with the Kubernetes machinery and workload limits behind those choices. ### 9. KV Cache-Aware Routing and P/D Disaggregation on Kubernetes Yuchen Fama & Ashish Kamra, Red Hat | 21:48 | AI Engineer World's Fair 2026 Video: https://www.youtube.com/watch?v=YXowceUKYJI Summary: https://aietalks.com/talks/kv-cache-aware-routing-and-p-d-disaggregation-on-kubernetes.md Why here: Kranen introduces cache-aware routing and prefill/decode separation. Fama and Kamra show the request path, the cache transfer and the latency evidence, then draw the boundary where an aggregated server is still better. The routing panel closes the pack by moving the same cost question from workers to models. ### 10. The State of Model Routing Nader Khalil, NVIDIA & Walden Yan, Cognition & Alex Atallah, OpenRouter & Tanay Varshney & Carter Abdallah, NVIDIA | 48:17 | AI Engineer World's Fair 2026 Video: https://www.youtube.com/watch?v=QHBjufYK8TA Summary: https://aietalks.com/talks/the-state-of-model-routing.md Why last: Fama and Kamra route a request to the worker that already holds useful KV state. Khalil, Yan, Atallah, Varshney and Abdallah ask which model should receive the work at all. Their answer ends the pack at the product boundary: preserve useful context, notice when a cheap model is thrashing, and price the completed task rather than the token. ## Editor's note These talks make inference tuning a comparison between configurations, not a hunt for one universal benchmark. Kitaru can replay the same recorded model workload against two runtime setups so a team can compare outputs alongside time to first token, completion latency and token use. The replay holds the application inputs steady; it does not reproduce GPU scheduling, queue pressure or cache placement from the original deployment, so those still need measurements from the serving system itself. Written by the AIE Talks editors (the Kitaru team), not by any of the speakers.