# Serving Voice AI at Scale Arjun Desai, Cartesia & Rohit Talluri, AWS | AI Engineer World's Fair 2025 | 17:05 Source: https://www.youtube.com/watch?v=knH3fmGAteQ Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/serving-voice-ai-at-scale Published: 2025-06-27 Tags: edge, inference, latency, voice ## TL;DR - Voice AI needs millisecond-level responses because pauses that are acceptable in text feel awkward in conversation. - Cartesia builds voice systems around quality, latency, and controllability, using state space models to reduce inference latency. - Production voice agents still benefit from separate speech-to-text, language-model, and text-to-speech components because this gives teams more control. ## Summary Arjun Desai and Rohit Talluri discuss the engineering constraints behind enterprise voice AI. Desai argues that conversational systems need responses in milliseconds, since users notice pauses that would be acceptable in text applications. Cartesia focuses on natural voice quality, fast time to first audio, and control over accents, cloned voices, background sounds, and other details. Its state space model architecture maintains a state during generation, avoiding the quadratic inference cost associated with longer transformer inputs. The speakers also discuss how voice agents depend on several connected systems, including speech recognition, a language model, and text-to-speech. Desai says orchestrated systems currently provide more control than speech-to-speech models for production use. Cartesia also runs models on edge devices, where it reports lower latency than a cloud round trip. The conversation covers healthcare, customer support, gaming, creator licensing, data requirements, evaluation, and AWS distribution through services such as SageMaker JumpStart and Amazon Bedrock. ## Key ideas ### Voice interfaces need responses in milliseconds because conversational pauses feel unnatural [02:09](https://www.youtube.com/watch?v=knH3fmGAteQ&t=129s) Desai compares voice with text, where waiting 500 or 600 milliseconds can be acceptable because people do not read at 200 tokens per second. In a conversation, a one-second delay feels awkward. Voice agents also have to handle interruptions, accents, background noise, and subjective preferences about how a voice should sound. Customer-support callers become annoyed when an agent does not respond quickly and accurately. These constraints make latency part of the user experience rather than a backend performance detail. ### Cartesia treats quality, latency, and controllability as separate requirements [02:51](https://www.youtube.com/watch?v=knH3fmGAteQ&t=171s) Cartesia's voice work starts from three requirements. The voice must sound natural. The system should produce its first audio quickly, giving the rest of the agent more time to reason. The voice must also be controllable because an agent's manner of speaking reflects a company's brand. Desai connects this control to voice cloning, accents, and the ability to reproduce background sounds such as phone beeps. A voice that sounds too perfect can feel strange when a caller expects the small sounds of a real phone conversation. ### State space models keep generation latency from growing quadratically with input length [03:47](https://www.youtube.com/watch?v=knH3fmGAteQ&t=227s) Desai describes state space models as an alternative to transformers. He says transformers have quadratic scaling as inputs grow, affecting memory and runtime. State space model generation at inference time is O(1) because the system maintains a state from which it can continue generating. Cartesia developed this approach because state space models had generally performed worse than transformers in recurrent settings. Desai says Cartesia has closed that quality gap, with models that are better on latency and quality. ### A voice agent depends on slack across the language model and speech components [05:27](https://www.youtube.com/watch?v=knH3fmGAteQ&t=327s) Text-to-speech is only one part of a voice agent. The system also needs a language model and speech-to-text, and Desai says those models often were not built for low-latency workflows. Faster speech generation gives the language model more slack in the end-to-end pipeline. This is why he treats latency on the voice side as a way to make the entire agent more usable, rather than as an isolated benchmark for the text-to-speech model. ### Voice AI is already being used in support, healthcare, and games [06:35](https://www.youtube.com/watch?v=knH3fmGAteQ&t=395s) Desai names healthcare and customer support as current voice AI markets. He also points to real-time gaming, where non-player characters can interact dynamically with players. He expects the range of applications to grow as voice systems become easier to deploy. Talluri describes AWS's interest in bringing more foundation model providers into its ecosystem so customers can choose models for specific use cases. ### Creator marketplaces can give voice actors licensing and distribution options [07:18](https://www.youtube.com/watch?v=knH3fmGAteQ&t=438s) Desai says Cartesia does not aim to replace voice actors. The company has a voice marketplace where creators can make their voices available for licensing. He describes the goal as preserving a creator's personality and identity while allowing other users to work with that voice through the platform. He also says narration remains an important use case, so the platform is intended to amplify voice actors rather than remove them from the process. ### Orchestrated voice pipelines currently offer more control than speech-to-speech models [12:40](https://www.youtube.com/watch?v=knH3fmGAteQ&t=760s) Desai says speech-to-speech models have useful applications, but he does not consider them ready for production or enterprise use cases. Separate components allow teams to control how speech recognition, language generation, and speech synthesis operate. He expects speech-to-speech systems to become faster over time, but says production systems need to function reliably in real-world settings. For now, an orchestrated pipeline gives developers more control over those details. ### Edge deployment can beat a cloud round trip when network latency dominates [13:23](https://www.youtube.com/watch?v=knH3fmGAteQ&t=803s) Cartesia builds models that can run on edge devices as well as in the cloud. Desai says some model capabilities require cloud-scale systems, especially on smaller devices, but other applications benefit from local inference. He frames the decision around whether network latency plus cloud processing is slower than running locally. For Cartesia's models, he says edge execution is about five times faster than making a round trip to the cloud. ### Evaluation and boundary handling matter because failures can occur between pipeline stages [14:40](https://www.youtube.com/watch?v=knH3fmGAteQ&t=880s) When an agent pauses or produces an error, developers need to identify whether the problem is in speech-to-text, the language model, text-to-speech, or the integration between them. Desai calls evaluations an important part of any system. He says outputs often need formatting between stages, and Cartesia handles many of those edge cases so developers do not have to manage them directly. He cites Sonic 2 as improving both model latency and the handling of issues that arise in language-model outputs. ## Notable quotes - "In voice, you don't have seconds to actually give your response back. You have milliseconds." (02:09) - "Speed is of the utmost importance and quality is just table stakes." (01:34) - "Our goal is not to replace voice actors actually." (07:18) - "The goal of the system is not to build something really cute. It's to build something that actually functions for real world use cases." (13:03) - "Running our models on edge are about five times faster than if you were to roundtrip." (14:01) ## Tools & references mentioned - Cartesia - Sonic 2 - AWS - Amazon Bedrock - SageMaker JumpStart - PipeCat - Claude - state space models - transformers ## Who should watch - You are building a voice agent and need to understand where latency enters the speech-to-text, language-model, and text-to-speech pipeline. - Your application needs voice control such as cloning, accents, background sounds, or brand-specific delivery. - You are deciding whether an orchestrated pipeline, a speech-to-speech model, or edge inference fits a production system. ## Related talks - [Giving a Voice to AI Agents](https://aietalks.com/talks/giving-a-voice-to-ai-agents) (Scott Stephenson, Deepgram, 13:08) - [Engineering voice agents: Latency, quality, and scale](https://aietalks.com/talks/engineering-voice-agents-latency-quality-and-scale) (Rishabh Bhargava, Together AI, 24:35) - [Building Effective Voice Agents](https://aietalks.com/talks/building-effective-voice-agents) (Toki Sherbakov & Anoop Kotha, OpenAI, 17:17) - [Building voice agents with OpenAI](https://aietalks.com/talks/building-voice-agents-with-openai) (Dominik Kundel, OpenAI, 1:25:35) - [Voice Agent Engineering](https://aietalks.com/talks/voice-agent-engineering) (Nik Caryotakis, SuperDial, 19:07)