# Mastering LLM Inference Optimization From Theory to Cost Effective Deployment

Mark Moyou, NVIDIA | AI Engineer World's Fair 2024 | 33:39

Source: https://www.youtube.com/watch?v=9tvJ_GYJA-o
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/mastering-llm-inference-optimization-from-theory-to-cost-effective-deployment
Published: 2025-01-01
Tags: cost, deployment, gpus, inference

## TL;DR
- LLM inference keeps the model weights and generated-token history on the GPU, so KV cache capacity directly affects throughput and cost.
- Production sizing depends on measuring actual input and output token patterns, including time to first token, token-to-token latency, and total completion time.
- Compiling an engine for a specific GPU with TensorRT-LLM, then hosting it with NVIDIA Triton, allows teams to tune memory use, batching, precision, and parallelism for their workload.

## Summary
Mark Moyou explains why LLM inference needs different deployment thinking from ordinary deep learning models. Prompts go through a prefill stage, then the model generates one token at a time. The KV cache stores the attention information needed to continue generation, and it competes with model weights for GPU memory. That makes input length, output length, batching, and model precision direct cost and capacity concerns. Moyou recommends measuring real production traffic instead of relying on cherry-picked benchmarks. Teams should track time to first token, token-to-token latency, total completion time, and the distribution of input and output lengths. Those measurements can guide engine configuration and even different engines for different traffic periods. He covers TensorRT-LLM for compiling optimized NVIDIA GPU engines, Triton for serving them, FP8 quantization, inflight batching, KV-cache quantization, and tensor parallelism.

## Key ideas
### LLM inference generates one token at a time and keeps its history on the GPU
[02:50](https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=170s)
Moyou frames inference around two facts. A prompt is placed on the GPU, and the model generates the answer one token at a time. Every generated token is added back to the GPU because the model needs the earlier context to produce a coherent continuation. This persistent history is the KV cache, which stores key and value matrices from attention. The same basic mechanism applies whether a request goes to a large hosted API or to a model running on one GPU. The cache therefore affects both the memory available for concurrent requests and the cost of serving them.

### Prefill and generation put different pressure on the deployment
[04:28](https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=268s)
Before generation begins, the system tokenizes the input and computes attention across the whole prompt. Moyou calls this stage prefill. A long input makes prefill take longer and occupies more memory. After prefill, generation computes the next token repeatedly. With a KV cache, the system only needs to compute attention for the newest token rather than reprocessing the full prompt each time. Moyou describes this as changing from matrix-matrix work during the initial prompt processing to vector-matrix work during generation. The distinction explains why input length mainly affects time to first token, while long outputs keep consuming cache memory during decoding.

### Model weights and KV cache compete for the same GPU memory
[14:44](https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=884s)
Moyou gives a practical memory rule: multiply the parameter count by two to estimate the model's FP16 memory in gigabytes. An 8B Llama model would therefore occupy about 16 GB in FP16, leaving the rest of an L4's memory for the KV cache and generated tokens. He presents the GPU as holding the model weights or the token history, with both competing for capacity. A larger model, longer prompts, and longer outputs reduce the number of requests that fit at once. Shrinking the model through quantization can free memory for more tokens and requests, which can improve throughput and lower deployment cost.

### Production traffic has distinct input and output patterns that must be measured
[17:25](https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=1045s)
Moyou divides requests into long-input/short-output, long-input/long-output, short-input/long-output, and short-input/short-output patterns. Long inputs extend prefill and use more memory before generation starts. Long outputs consume cache space for longer and can keep a request active. A deployment may receive all of these patterns, so a benchmark using only one or two patterns can mislead capacity planning. He recommends recording input sequence length and output sequence length across users, then plotting their distribution. Those observations can determine the engine's configured limits and show when users are sending prompts that fill the GPU.

### Latency measurements reveal different failures under load
[20:38](https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=1238s)
The first measure is time to first token, which reflects prompt processing and attention performance. Token-to-token latency measures the spacing between later generated tokens. Moyou says this latency can drift upward as sequences grow because memory use and system load increase. Total generation time measures the complete request from prompt arrival to the finished response. He also recommends plotting time to first token against input-token count. A rising slope means longer prompts are taking increasingly longer to process. Wider completion-time distributions can point to batching or scheduling problems. These measures describe behavior under load more accurately than one total latency number.

### Inference engines should be built around observed workload bounds
[19:29](https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=1169s)
Moyou argues that the aim of engine optimization is to shrink the model while preserving accuracy, then use the recovered memory for more tokens. A two-dimensional histogram of input sequence length and output sequence length helps choose the engine's bounds. Instead of configuring only for a stated maximum, a team can inspect what users actually sent during a measured period. He also proposes seasonal engines. Traffic during the day may have different request shapes from traffic at another time, so an operator could scale with configurations built for those patterns. The deployment team still has to test whether a different engine satisfies each workload at lower cost or latency.

### TensorRT-LLM, Triton, and lower precision target different parts of serving
[22:45](https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=1365s)
Moyou describes TensorRT-LLM as the model compilation package for LLMs on NVIDIA GPUs. The resulting engine is compiled for a specific GPU and cannot simply move to another GPU. Triton hosts the engine, accepts requests, and handles serving and batching alongside other model types. He points to FP8 as a way to reduce model memory substantially while retaining similar accuracy, creating more room for tokens and potentially increasing speed. Inflight batching lets a new request enter as soon as another finishes instead of waiting for every request in a batch. He also mentions quantized KV cache and tensor parallelism, which splits a model across GPUs to reduce latency, generally within one node.

### Future scaling depends on attention efficiency and faster GPU interconnects
[32:00](https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=1920s)
In the closing discussion, Moyou explains why very large context lengths are expensive. A longer prompt requires more memory for attention, so scaling that behavior across a large deployment can require far more GPUs. FlashAttention is one approach that keeps more of the operation in fast GPU cache. He also describes NVLink as the connection that lets GPUs move data quickly between one another, mentioning a Blackwell configuration connecting 72 GPUs. Tensor parallelism and pipeline parallelism distribute model work differently, while mixture-of-experts models add another pattern for handling large systems. He says the industry is still working on alternatives to attention because its scaling cost remains a serious constraint.

## Notable quotes
- "Every token that I generate gets locked onto the GPU." (03:47)
- "The more that I shrink the faster it runs the more GPU memory I have for what tokens." (19:29)
- "The first thing that we measure is time to First token." (15:26)
- "An engine that's built to a specific GPU cannot be moved to another GPU." (27:27)
- "In Flight batching it just means I don't have to wait for all the requests to finish to start a new request." (25:08)

## Tools & references mentioned
- NVIDIA
- TensorRT-LLM
- NVIDIA Triton
- NVIDIA Inference Microservices
- Llama
- L4
- A100
- H100
- Blackwell
- Hopper
- Ada Lovelace
- FP8
- FP4
- FlashAttention
- NVLink
- GenPerf
- Hugging Face
- Triton
- TensorRT
- PyTorch
- TensorFlow
- Flask
- Mixture of Experts

## Who should watch
- You are sizing an LLM deployment and need to understand how prompt lengths, response lengths, and KV-cache memory affect GPU capacity.
- Your benchmarks look good at low load, but production latency changes as requests get longer or arrive concurrently.
- You are choosing between custom engine work and NVIDIA serving software, and need to understand what TensorRT-LLM and Triton each do.

## Editor's note

From the pack [Inference: serving models efficiently](https://aietalks.com/packs/inference):

These talks make inference tuning a comparison between configurations, not a hunt for one universal benchmark. Kitaru can replay the same recorded model workload against two runtime setups so a team can compare outputs alongside time to first token, completion latency and token use. The replay holds the application inputs steady; it does not reproduce GPU scheduling, queue pressure or cache placement from the original deployment, so those still need measurements from the serving system itself.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Hacking the Inference Pareto Frontier](https://aietalks.com/talks/hacking-the-inference-pareto-frontier) (Kyle Kranen, NVIDIA, 20:25)
- [How fast are LLM inference engines anyway?](https://aietalks.com/talks/how-fast-are-llm-inference-engines-anyway) (Charles Frye, Modal, 16:07)
- [From Model Weights to API Endpoint with TensorRT-LLM](https://aietalks.com/talks/from-model-weights-to-api-endpoint-with-tensorrt-llm) (Philip Kiely & Pankaj Gupta, Baseten, 1:40:01)
- [Running LLMs Locally: Practical LLM Performance on DGX Spark](https://aietalks.com/talks/running-llms-locally-practical-llm-performance-on-dgx-spark) (Mozhgan Kabiri chimeh, NVIDIA, 10:16)
- [The LLM Triangle: Engineering Principles for Robust AI Applications](https://aietalks.com/talks/the-llm-triangle-engineering-principles-for-robust-ai-applications) (Almog Baku, 26:19)