Gabber chose Orpheus because it was an open-source streaming voice model that could produce emotive speech in real time.
2
Fine-tuning away Orpheus's initial silence reduced clone latency from roughly 600 milliseconds to about 100 milliseconds at P50.
3
vLLM's batching, LoRA support, and FP8 dynamic quantization let Gabber run multiple voice generations and clones on L40S GPUs.
Summary
Neil Dwyer explains how Gabber deployed Orpheus, an open-source text-to-speech model, for real-time consumer applications. Gabber needed voice infrastructure that could approach free for products such as AI companions, therapists, toys, and trainers. Orpheus produced audio tokens at a rate that could keep up with speech, and LoRA fine-tunes gave Gabber higher-fidelity, more emotive voice clones than one-shot cloning. The largest latency problem was silence at the start of some Orpheus voices. Fine-tuning removed most of that wasted generation, reducing latency from about 600 milliseconds to about 100 milliseconds at P50. For infrastructure, Gabber uses vLLM for batched inference with multiple LoRAs and FP8 dynamic quantization. Its load balancer uses a consistent hash ring so sessions stay on GPUs that have the required clone in memory. A WebRTC backend proxies sessions over WebSockets to GPUs, with Redis tracking placement.
Gabber targets consumer AI applications that need voice to cost almost nothing
Gabber builds infrastructure for real-time AI personas with voice, memory, video inputs, and tool calling. Dwyer's focus is consumer applications rather than call centers, customer support, or AI sales development. He names AI companions, multiplayer characters, therapists, personal trainers, and children's toys as examples. Some voice platforms cost up to $5 per hour, which Dwyer says does not work for most consumer products. Gabber initially used hosted voice models, but its cost target forced the company to bring inference in-house and run models on its own GPUs.
Orpheus generates audio tokens fast enough for streaming speech
Orpheus began with a Llama 3 billion-parameter model trained on about 100,000 hours of voice and text data. It outputs audio tokens from the Snack codec, which are decoded into 24 kHz audio. The model produces about 85 Snack tokens per second of audio, so a deployment needs roughly 90 to 100 tokens per second to stay ahead of real time. Falling below that rate creates gaps in playback. Gabber adopted Orpheus because it was the first open-source streaming voice model Dwyer considered good enough for production.
LoRA fine-tunes give Gabber more useful voice clones than one-shot cloning
Dwyer says one-shot cloning does not work well enough for Gabber's needs, especially because Orpheus had around 100,000 hours of pretraining rather than the much larger data scale where some zero-shot behavior emerges. Gabber therefore uses low-rank fine-tunes for individual voices. In his example, a 16-rank, alpha-32 fine-tune used about 10 minutes of data, although he says 30 minutes would be better. The resulting clone is imperfect because the training set was overfit, but it retains the speaker's voice and responds to language cues with emotion.
Initial silence in Orpheus wastes much of the latency budget
Dwyer identifies time to first token, tokens per second, network latency, and what Gabber calls head-of-line silence as latency factors. Some Orpheus voices begin with about 600 milliseconds of silence, probably because of how voice-actor recordings were prepared for training. Gabber filters that silence out of playback, but the GPU still spends time generating it. On L40S machines producing about 100 tokens per second, the wasted silence adds nearly half a second. A LoRA fine-tune can train the silence away, bringing the example clone to about 100 milliseconds of latency at P50.
Streaming systems need to generate audio during the endpointing pause
In a real-time conversation, Gabber waits to decide whether the person has finished speaking, then adds a short pause because endpointing models are imperfect. The system starts the language model during that pause. Gabber begins producing Orpheus audio after two sentences, or sooner if the response is complete, so the first audio packet can arrive before the pause ends. Dwyer says pauses around half a second to one second may be a good range as endpointing improves, while delays above roughly one and a half seconds sound bad. This matters more because Gabber lets customers bring their own language models.
vLLM makes batched inference with several LoRAs practical on L40S GPUs
Gabber needs to run several generations concurrently on one GPU and serve multiple LoRAs in the same batch. It also wants one load balancer to hide the separate models used for different languages. Dwyer credits vLLM with supporting batched inference with LoRAs. The FP16 model was slower than real time on L40S hardware, so Gabber uses vLLM's FP8 dynamic quantization. He says it requires essentially no manual work because the scaling is handled automatically. With a batch of 10, Gabber reached 105 tokens per second for base voices and 95 tokens per second for LoRA voices.
A consistent hash ring keeps clone sessions near the right GPU
LoRAs range from about 100 to 200 megabytes, so Gabber wants requests to reach a GPU that already has the needed clone in memory. Its load balancer uses sticky sessions and a consistent hash ring. Servers are placed around the ring through virtual nodes, and a request is assigned by hashing its identifier and choosing the nearest server. Removing a server causes only a limited number of migrations instead of rebalancing everything. If a clone becomes popular, Gabber can add that LoRA to more servers and scale its capacity without redesigning the system.
Gabber separates WebRTC connections from GPU inference sessions
At the system level, a WebRTC backend terminates client connections and opens a WebSocket connection to the GPU layer. When a session starts, the backend connects to an available GPU, asks Redis which GPU should handle the request, and then proxies the session to that destination over another TCP connection. The GPUs are in the same data center and private network, so Dwyer considers the extra hop acceptable. He admits Redis is not an ideal long-term choice if the system grows beyond its current needs.
"Most consumer use cases they need something pretty close to free."03:44
Who should watch
You are deploying streaming text-to-speech and need to understand where latency is spent before choosing larger hardware.
Your product needs many custom voices, and you need a serving design that keeps LoRAs in GPU memory.
You are evaluating open-source voice models for consumer applications where hosted speech APIs are too expensive.
