Text-to-speech systems increasingly use autoregressive transformers to generate chunks of audio in sequence, because this makes early audio available before the full response is complete.
2
Neural audio codecs compress dense audio into token sequences that a transformer can process, while preserving voice and acoustic information beyond the words alone.
3
The next latency improvement is streaming text from an LLM directly into speech generation, but the best architecture for doing this is still unsettled.
Summary
Samuel Humeau explains why current text-to-speech systems increasingly resemble language models. Speech carries far more information than text, so models first compress short audio frames into token sequences with neural codecs. A transformer then generates those frames in order, while a decoder turns the tokens back into audio. This setup lets an agent play its first audio packet before the complete response has finished. Humeau demonstrates Mistral's new open-source TTS model with voice cloning, multilingual pronunciation, and an agent that answers conference schedule questions. He also distinguishes Mistral's implementation from the common pattern: its 37 tokens per 80-millisecond frame are generated together with a diffusion-style model. The remaining hard problem is text streaming. Generating separate audio chunks can cause continuity problems, while interleaved and dual-stream designs have different tradeoffs. Mistral has not chosen a final architecture for this yet.
Speech generation matters most inside real-time agents
Humeau says the main current use case for text-to-speech is an agent that combines speech-to-text, a chat model, and speech output. Latency matters on both sides. Real-time speech recognition can finish the transcript when the user stops speaking, while speech synthesis should play audio as soon as its first packets arrive. The ideal setup starts speaking with the first token produced by the language model. This reduces perceived waiting time even when the full audio computation continues in the background. Humeau focuses much of the talk on how systems produce and stream those first audio packets.
Early audio can play before the complete waveform exists
In the live demonstration, Humeau uses a few seconds of recorded speech from a person called Paul, then generates new text in Paul's voice. The first audio packet arrives soon enough to start playback, while the rest of the utterance is still being computed. He says the full computation can finish much later, but the listener already hears the response. The same setup powers a small voice agent that answers questions about conference sessions at 12:20 and 11:15. The agent can respond conversationally because playback begins before every audio packet has been generated.
Voice cloning needs only a short reference recording
The model can clone a voice from only a few seconds of reference audio. Humeau demonstrates Paul's voice, then a French voice speaking English with a recognizable French accent, and finally his own voice. He says the system can infer how someone would speak in another language while retaining characteristics of the reference voice. He also points to a possible use beyond personal assistants: companies may treat a voice as part of their brand identity, much as they define the appearance of a website. The model discussed in the talk does not expose the voice-cloning encoder as open weights, however.
Audio must be compressed before a transformer can model it
Humeau describes audio as measurements of microphone pressure taken several thousand times per second. Generating each sample individually is slow, while producing the whole waveform at once makes early playback harder. Current systems therefore model audio as a sequence of chunks, following the language-model pattern. A frame of roughly 80 milliseconds is converted into tokens by an encoder, and a decoder reconstructs audio from them. The problem is information density. A standard-quality MP3 carries about 200 kilobits per second, far more than spoken text. A neural codec compresses the audio while retaining acoustic details such as the speaker's voice.
Humeau gives Mistral's codec as a concrete example. The system divides audio into 80-millisecond frames, producing 12 frames per second, and represents each frame with 37 tokens. That reduces the stream to about 500 tokens per second. The codec is trained to reconstruct a large collection of audio through a bottleneck. Reconstruction and adversarial losses guide what information remains, and some tokens are encouraged to preserve text information so the content can be recovered. The result carries more than a subtitle track, including acoustic properties needed to reproduce the voice.
A smaller decoder handles the tokens inside each audio frame
Putting every codec token into one long sequence would require too many steps from the main transformer. Humeau says the common design gives the backbone one step per audio frame, then uses a smaller model to reconstruct all tokens belonging to that frame. His example has a 4-billion-parameter backbone and 37 tokens per frame. Mistral's released model differs from this common pattern. It generates the 37 tokens for a frame together with a diffusion model, making the design somewhat different from a standard autoregressive TTS system. Humeau points to Mistral's technical report for the implementation details.
A model that generates speech still needs conditioning to become text-to-speech. Humeau describes two broad approaches. One supplies the complete text context at the beginning, then generates the audio. Another keeps adding text context while the audio is produced. Mistral's released model uses the first approach: the reference audio for the voice and the text to pronounce are provided as context. With the network removed from the measurement, Humeau reports 17 milliseconds from text input to the first playable audio on a single GPU.
Streaming text input remains an open architecture problem
Humeau separates streaming audio output from the harder problem of receiving text continuously from an LLM. Generating independent audio pieces and stitching them together can create continuity problems. Other proposals interleave text and audio in one stream or use separate audio and text streams that interact during inference. Mistral has not decided which architecture is best, and Humeau mentions delayed sequence modeling as another possibility. The benefit would be clear for long answers: speech could begin with the first text produced by the LLM instead of waiting for the entire response. In the demo, the language model produced the short answer in one quick operation, so text and audio only appeared simultaneous.
"We need to somehow compress it and reduce the size of it, maybe drop what's not needed."10:28
Who should watch
You are building a voice agent and need to understand where its latency comes from, especially the difference between first-audio latency and full-response latency.
You want an engineering explanation of neural audio codecs, frame-based generation, and the transformer designs used in modern TTS.
You are evaluating streaming text-to-speech and need to know which design choices remain unsettled before committing to an architecture.
