# How fast are LLM inference engines anyway?

Charles Frye, Modal | AI Engineer World's Fair 2025 | 16:07

Source: https://www.youtube.com/watch?v=DeFF3J8T5Pk
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-fast-are-llm-inference-engines-anyway
Published: 2025-06-27
Tags: benchmarks, gpus, inference, latency

## TL;DR
- Open-weight models and open-source inference engines have become good enough that self-hosting now makes sense for many workloads.
- Prefill-heavy workloads can reach much higher throughput than decode-heavy workloads, with one benchmark showing about four times as many requests per second.
- Benchmarking should expose the engine's maximum parallelism, then sweep down to measure the tradeoff between throughput and latency.

## Summary
Charles Frye describes how open-weight models and inference software have changed since 2023. Models such as Llama, Qwen, and DeepSeek now make many applications possible, while engines such as vLLM, SGLang, and TensorRT-LLM handle techniques including KV caching, paged attention, multi-token prediction, and speculative decoding. Modal ran benchmarks across models, engines, context lengths, and hardware, then published the results in the LLM Almanac. Frye shows that the balance between input and generated tokens has a large effect on throughput. In one comparison, changing from 128 input and 1,024 output tokens to 1,024 input and 128 output tokens raised throughput from about one request per second to about four, while time to first token stayed almost the same. He explains how the benchmark estimates maximum throughput and why total capacity comes from scaling across replicas. The talk is practical, though Frye is clear that many configurations still need further optimization.

## Key ideas
### Open-weight models now support applications that previously required proprietary models
[01:14](https://www.youtube.com/watch?v=DeFF3J8T5Pk&t=74s)
Frye says the Llama, Qwen, and DeepSeek model series have changed what engineers can do with open weights. In 2023, he saw open models as too weak for some of the interesting applications he wanted to build. He now thinks they are catching up with frontier labs in some areas, and are at least capable enough for many applications discussed at AI Engineer events. He connects this to a broader pattern from operating systems, databases, and programming languages: when a workload only needs a good-enough capability level, collaborative projects can catch up and offer better properties.

### Inference engines package optimizations that are difficult to implement by hand
[01:30](https://www.youtube.com/watch?v=DeFF3J8T5Pk&t=90s)
Frye recalls initially thinking that running a language model would mainly require wrapping a class around a torch.nn.Module. Transformer inference quickly becomes more involved than that. KV caching is only the starting point, followed by paged attention, multi-token prediction, and speculative decoding. Engines such as vLLM, SGLang, and TensorRT-LLM provide these mechanisms. Their availability changes the calculation for teams deciding whether to self-host. Frye names situations such as government use, air-gapped systems, or a strong preference for open-source models as reasons that can still justify running the models directly.

### Modal turned repeated customer questions into a public benchmark database
[05:07](https://www.youtube.com/watch?v=DeFF3J8T5Pk&t=307s)
Teams asked Modal questions such as how quickly an eight-billion-parameter Llama model could run with SGLang for a specified input and output length. Frye says the team first needed days, then an hour or two, and eventually 15 to 20 minutes to produce trustworthy answers. They then ran a larger benchmark across roughly ten models, the inference engines vLLM, SGLang, and TensorRT-LLM, and about ten context lengths. The results, methodology, source code, and an executive summary are available through the LLM Almanac at modal.com/llm-almanac.

### The published results begin with out-of-the-box performance rather than full optimization
[07:48](https://www.youtube.com/watch?v=DeFF3J8T5Pk&t=468s)
Frye says the benchmark interface does not yet contain every model, engine, hardware, and configuration combination. Some combinations are impossible to run or are difficult to compare. The initial results use out-of-the-box engine performance because carefully optimizing one hundred configurations would take substantial time. He invites contributions, especially optimized TensorRT-LLM implementations, which expose many tuning options. He also mentions an issue where SGLang's Qwen 3 support was buggy for the eight-bit quantization used in one test, so that model initially had results only for vLLM.

### A one-second first-token target is common, while 300 milliseconds is more interactive
[08:26](https://www.youtube.com/watch?v=DeFF3J8T5Pk&t=506s)
Frye uses time to first token as an important filter in the benchmark interface. He calls one second a common service-level objective because it is a round number that users are willing to wait for. He describes 300 milliseconds as a tighter target for interactive systems, while noting that the threshold is a made-up number that became familiar through repetition. The benchmark interface lets users inspect throughput under a chosen first-token limit, model, engine, quantization, and token workload.

### Prefill-heavy workloads can deliver much higher throughput than decode-heavy workloads
[10:47](https://www.youtube.com/watch?v=DeFF3J8T5Pk&t=647s)
Frye compares 128 input tokens and 1,024 output tokens with 1,024 input tokens and 128 output tokens. He describes the first as closer to a reasoning workload and the second as closer to a retrieval-augmented generation workload, while stressing that the comparison is about the balance between prefill and decode. In the example, vLLM reaches about one request per second for the decode-heavy case and about four requests per second for the prefill-heavy case. Time to first token remains almost identical even though the input contains ten times as many tokens in the second setup.

### Shorter numerical formats can improve throughput because they make multiplication faster
[12:22](https://www.youtube.com/watch?v=DeFF3J8T5Pk&t=742s)
Frye links the prefill advantage to the transformer computation. Prefill can use large matrix-matrix multiplications, load the weights once, and reuse them across many tokens. He says the example's fourfold improvement uses BF16, which has tensor-core support but is slower than FP8 or FP4 on Hopper and Blackwell hardware. He expects FP4 results on Blackwell GPUs to create an even larger gap. He also gives a product decision: when a team wants better quality without breaking latency targets, adding context may be cheaper and easier to serve than immediately adding more reasoning tokens.

### Maximum throughput comes from exposing enough parallelism, then measuring lower-load latency
[14:43](https://www.youtube.com/watch?v=DeFF3J8T5Pk&t=883s)
Frye explains that the reported throughput is per replica, with one GPU in the example. A service needing much more total capacity should scale out across replicas. To estimate a maximum, the benchmark sends about one thousand requests and divides the request count by the time until they all finish. This exposes substantial parallelism to the engine. Queueing theory predicts that sending more work than this maximum will cause latency to grow sharply. The benchmark also sends one request, waits for its response, and repeats, then sweeps between those two extremes to produce the displayed throughput and latency results.

## Notable quotes
- "It sort of finally makes sense to self-host." (02:57)
- "So the combination of those two things has kind of flipped the playing field around." (02:19)
- "You get much higher throughput if you have more tokens in the context as opposed to tokens being generated." (11:27)
- "Don't immediately reach for reasoning. Reach for context instead because it's going to be cheaper." (13:29)
- "The way you solve your total throughput is by scaling out rather than scaling up." (14:43)

## Tools & references mentioned
- Modal
- Llama
- Qwen
- DeepSeek
- vLLM
- SGLang
- TensorRT-LLM
- KV caching
- paged attention
- multi-token prediction
- speculative decoding
- LLM Almanac
- Hopper
- Blackwell
- BF16
- FP8
- FP4
- OpenAI
- BERT
- Cursor
- Devin
- TGI
- Transformers

## Who should watch
- You are choosing between self-hosting an open-weight model and paying an inference provider, and need throughput data for a concrete token workload.
- Your product team wants better model quality but has a strict latency target, so the difference between adding context and adding generated reasoning matters.
- You are building or reviewing inference benchmarks and want a method that measures both maximum parallel throughput and low-load latency.

## Related talks

- [AI Engineering 201: Inference](https://aietalks.com/talks/ai-engineering-201-inference) (Charles Frye, Full Stack Deep Learning, 1:43:16)
- [Hacking the Inference Pareto Frontier](https://aietalks.com/talks/hacking-the-inference-pareto-frontier) (Kyle Kranen, NVIDIA, 20:25)
- [Mastering LLM Inference Optimization From Theory to Cost Effective Deployment](https://aietalks.com/talks/mastering-llm-inference-optimization-from-theory-to-cost-effective-deployment) (Mark Moyou, NVIDIA, 33:39)
- [Trends Across the AI Frontier](https://aietalks.com/talks/trends-across-the-ai-frontier) (George Cameron, ArtificialAnalysis.ai, 17:52)
- [Running LLMs Locally: Practical LLM Performance on DGX Spark](https://aietalks.com/talks/running-llms-locally-practical-llm-performance-on-dgx-spark) (Mozhgan Kabiri chimeh, NVIDIA, 10:16)