Open-source TTS models such as Orpheus can use LLM optimizations because they have an autoregressive LLM backbone.
2
For voice models, once generation is fast enough for real-time audio, time to first byte and concurrent streams matter more than maximizing tokens per second.
3
Production latency also depends on client code and network infrastructure, so session reuse, concurrent requests, data-center placement, and low-overhead service connections matter.
Summary
Philip Kiely explains how to optimize Orpheus TTS for production voice systems. Orpheus uses a Llama 3.2 3B backbone, which allows Baseten to apply TensorRT-LLM and FP8 quantization. Its audio decoder can also run with torch.compile and PyTorch inference mode on the GPU, while dynamic batching sends accumulated work every 15 milliseconds. The target is different from ordinary LLM serving: Orpheus needs about 83 tokens per second for real-time streaming, after which time to first byte and concurrency become the main goals. Kiely describes results on an H100 MIG, including up to 150 milliseconds to first byte and higher concurrent-stream capacity than an off-the-shelf setup. He is careful about the limits of those measurements. Network location, session setup, request concurrency, streaming protocols, and connections between listening, thinking, and talking components can add more latency than runtime optimization removes. The talk ends by placing TTS optimization inside the larger voice-agent pipeline, including future work such as fine-tuning and voice cloning.
TTS models can use much of the LLM optimization stack
Kiely uses a useful simplification: many TTS systems are autoregressive transformers that are close to LLMs, unlike diffusion-style models. Orpheus TTS is built on a Llama 3.2 3B backbone and uses a causal language-model architecture. That makes tools developed for LLM serving applicable to TTS. Orpheus has a larger vocabulary for speech-specific tokens, including tokens for sounds such as laughter, and it has an extended context length for rope scaling. Any optimization must support those changes rather than assuming an ordinary text model.
Voice serving changes the performance target after real-time speed
For ordinary LLM serving, teams often push tokens per second as high as possible. Kiely gives Orpheus a different target: about 83 tokens per second is enough for a real-time stream. Once the system reaches that rate, more generation speed has less value than reducing time to first byte and increasing concurrency. The practical goal is to fit many voice agents on one GPU, or less than one GPU, while keeping responses quick. This shifts the optimization question from peak generation throughput to how many simultaneous connections the service can handle at an acceptable latency.
TensorRT-LLM, FP8, and compiled audio decoding improve the runtime
Baseten uses TensorRT-LLM because Orpheus has an LLM-style backbone. Kiely also describes FP8 quantization on Hopper hardware, including quantization of the KV cache, and says it works well for this model despite the usual concern that small models may lose performance when quantized. The audio side needs separate work. The system uses SNAC as an audio decoder, then runs the decoder with torch.compile and PyTorch inference mode on the GPU. Token batching must work across the whole pipeline, and the service supports multiple streaming protocols.
Dynamic batching sends audio work on a short timer
The audio path uses dynamic batching rather than token-level continuous batching. Requests are packed into a batch, then dispatched every 15 milliseconds. Increasing the batch size can trade some latency for higher throughput. Kiely says the implementation is often CPU-bound, even though next-token generation and audio decoding run on the GPU. Both loops reach the CPU at different points, so CPU capacity can limit the number of simultaneous streams. This gives a concrete tuning direction: adding CPU resources can increase capacity efficiently when the GPU is not the limiting component.
The reported runtime result is measured on part of an H100
On an H100 MIG, which is half of an H100 GPU, the off-the-shelf implementation handles 16 simultaneous streams with variable traffic and 24 with constant traffic. The MIG setup provides Hopper features such as TensorRT-LLM performance and FP8 support without allocating an entire 80 GB GPU to a 3-billion-parameter model. Kiely reports that the optimized implementation reaches 150 milliseconds time to first byte in real-world testing. He also limits the claim: this is one part of the pipeline, not the latency a user experiences from end to end.
Kiely warns that infrastructure and client behavior can add back the time removed by runtime optimization. A request sent from New York to a service in California can add substantial delay, as can creating a new session for every request. Basic examples often send requests sequentially and stream audio to a local computer. Production code should instead share a session, generate concurrent traffic, and use a benchmark-style multiprocessing pool. The test must reflect the concurrency the service is expected to handle, rather than measuring one isolated request.
Voice-agent latency depends on connections between several models
TTS is only one part of a voice agent. Kiely describes the pipeline as listening, thinking, and talking, with infrastructure connecting the models. Keeping components in the same data center reduces network overhead. He also warns about DNS hairpinning, where traffic leaves and returns to a location unnecessarily. Saving 10 milliseconds at each step can save 40 or 50 milliseconds in a pipeline with four or five steps, including chunking and interruption handling. WebSockets or gRPC may be more appropriate than HTTP streaming in systems built with tools such as LiveKit or Pipecat.
Custom voices add work beyond runtime optimization
Kiely says the talk does not cover all the work needed for a production voice system. Fine-tuning, custom voices, zero-shot voice cloning, and removing static or popping at the ends of messages all require additional engineering. These features sit on top of the inference runtime and interact with the rest of the voice pipeline. His conclusion is that infrastructure and client implementation deserve as much attention as model execution, and may matter more when the runtime has already reached a real-time generation rate.
"We can actually get all the way down to 150 millisecond time to first byte in real world testing that we've done."09:55
Who should watch
You are serving open-source TTS models and need lower first-byte latency without allocating a full GPU to a small model.
Your benchmark looks fast, but production voice requests are slower because of session creation, sequential clients, network distance, or protocol overhead.
You are building a voice agent and need to reason about latency across speech recognition, model reasoning, speech generation, and the services connecting them.
