A conversational agent can combine speech-to-text, an LLM, and text-to-speech while streaming each stage to reduce response delay.
2
ElevenLabs' language detection tool can identify a user's language and switch to a voice configured for that language during a conversation.
3
Agents can use custom tools, knowledge bases, webhooks, and agent-to-agent transfers for tasks such as scheduling and routing support requests.
Summary
Thor Schaeff walks through the parts of a multilingual conversational agent built with ElevenLabs. The pipeline transcribes the user's speech, sends the text to an LLM, and streams the generated text into speech. He explains why ElevenLabs currently uses this text-based pipeline instead of relying only on direct audio-to-audio models: text makes the conversation easier to monitor and inspect. The workshop demonstrates speech recognition across accents and languages, voice selection, agent configuration, language switching, knowledge bases, retrieval, function calling, and custom LLM endpoints. Schaeff also answers implementation questions about latency, tool timeouts, long-running tasks, agent transfers, mixed-language speech, pronunciation dictionaries, and fraud prevention. His answers are practical but sometimes incomplete. Mixed-language input remains difficult, especially when several languages occur in one utterance, and he recommends testing different models for latency-sensitive applications.
The agent pipeline moves from speech to text, through an LLM, and back to speech
Schaeff describes a conversational agent as a sequence of components. The user's speech is transcribed into text, that text is passed to a large language model acting as the agent's brain, and the LLM's text response is streamed into speech. ElevenLabs partners with model providers such as OpenAI and Google rather than supplying the intelligence model in this pipeline. The platform also includes system tools, including language detection and function calling. He contrasts this with audio-to-audio systems such as OpenAI Realtime. Direct audio processing may reduce some steps, but text gives developers a clearer view of what happened in a conversation.
The speech recognizer provides structured information beyond a plain transcript
ElevenLabs' speech-to-text model supports 99 languages, according to Schaeff. Its response can include word-level timestamps, speaker diarization, and audio event tags such as coughing or laughing. He demonstrates a conference call transcript where the system labels different speakers and lets developers highlight individual words on a timeline. Schaeff also shows a Telegram bot that accepts forwarded voice messages or videos, identifies the language, and returns a transcript. He uses Singapore English, a Scottish accent, and noisy audio to show that the model can handle accents and imperfect recordings without fine-tuning.
Voice selection can match languages and regional accents
The ElevenLabs voice library contains more than 5,000 voices. Developers can filter voices by language, accent, gender, and age, then assign different voices to the languages an agent supports. Schaeff uses Singapore as an example because the country has four official government languages. He explains that a Tamil-speaking Singapore application might choose a Chennai-accent Tamil voice from the library. He also demonstrates a cloned version of his own voice. Voice creators can publish voices in the library and receive royalties when others use them.
Language detection switches the agent using recognition confidence and voice settings
The conversational AI dashboard includes a language detection system tool. It can detect when a user asks to change languages, even when the request itself is spoken imperfectly, and it can switch the response to the requested language. Developers configure the available languages and assign a voice to each one. Schaeff later explains that the ASR model produces both a transcript and a likelihood score for the detected language. The system tool uses that confidence information to decide when to switch, then generates the reply with the voice configured for that language. The language behavior can also be customized through the LLM prompt.
Agents can combine an LLM, retrieval, and tools through the dashboard or API
The dashboard lets developers select an integrated LLM provider or connect a custom model through an OpenAI-compatible endpoint. Agents can use uploaded documents, website references, and retrieval-augmented generation. The platform also supports standard function calling for custom tools. Schaeff gives appointment scheduling as an example: a tool can call a CRM or cal.com endpoint, check availability, collect an email address, and schedule the meeting. Developers can configure agents in the dashboard, use JavaScript or Python SDKs in an application, or control the full setup through the API.
Latency depends on model choice, streaming, and how tools are configured
For latency-sensitive phone agents, Schaeff recommends testing models against the application's needs. He mentions Gemini Flash as one possible choice when reducing delay matters, while noting that the right model depends on function-calling requirements and the rest of the use case. ElevenLabs uses flash models by default for speech generation. For slow enterprise tools, developers can use retrieval for large knowledge bases and configure tool timeouts. The agent can tell the caller that it is still looking something up while waiting. Schaeff says the maximum timeout allowed is 120 seconds, though he suggests keeping the wait fairly low for a live call.
Separate agents can handle separate tasks through silent transfers
When one agent has too many onboarding and follow-up tasks, Schaeff suggests using agent-to-agent transfers. Developers can create separate agents for different use cases and even choose a different LLM for each task. An orchestration setup routes the conversation to the appropriate agent. If the voice remains the same, the transfer can happen without an obvious handoff to the user. This structure can also let different teams own different agents. Schaeff presents it as a way to divide responsibilities instead of forcing one prompt to manage every task.
Mixed-language speech remains difficult when several languages occur in one utterance
Schaeff does not claim that the language detector solves code-switching reliably. In a demonstration, an English question includes a German term and an attempted Chinese term. The system handles the German word but fails to recognize the Chinese phrase, partly because of the speaker's pronunciation. He says two intermixed languages generally perform acceptably, while adding a third language tends to make recognition worse, although he does not provide exact benchmarks. He suggests testing recordings with the transcription model and considering audio-to-audio systems such as OpenAI Realtime for language-learning applications, while acknowledging that those systems can introduce accent problems of their own.
Pronunciation dictionaries and safety controls address separate production concerns
For acronyms and company vocabulary, ElevenLabs provides pronunciation dictionaries for text-to-speech. Developers can supply phonetic instructions so terms such as acronyms are spoken in the desired way. Schaeff says there is no specific speech-to-text fine-tuning option for custom acronyms in the demonstrated workflow, but a normalization step in the system prompt may help the LLM correct transcript terms. On safety, he points to voice moderation, blocked terms for published voices, platform monitoring, watermarking, and voice-capture verification for professional voice clones. Generated speech can be traced to the account that created it, which supports action against fraudulent use.
"What we've seen for deploying conversational AI agents at scale and really understanding what is kind of happening, so if you're going sound token to sound token you're kind of flying blind a little bit."10:00
Who should watch
You are building a voice agent that needs to identify a user's language and respond with a matching regional voice.
Your application needs function calls such as appointment booking, CRM lookups, or routing between task-specific agents.
You need practical guidance on latency, long-running tools, mixed-language speech, pronunciation, or voice safety before deploying a phone agent.
