Voice AI is a joint activity between the bot and the user, rather than just a pipeline.
2
Voice agents need to handle sounds, words, interaction, and mental models across both listening and speaking.
3
A user's mental model is what remains after spoken sounds, words, and interactions disappear.
Summary
Midam Kim uses a failed call to show why voice-agent problems should be understood through linguistics. The bot mishears the final letter of her name, applies English pronunciation rules, asks for an account number without enough context, interrupts her while she reads it, and repeats the same failed request without clarification. Kim maps these failures onto two channels, listening and speaking, with four levels in each: sounds, words, interaction, and mental model. The levels depend on one another, so improving speech recognition alone does not solve timing, word choice, or task understanding. Voice also differs from chat because spoken history disappears as it is produced. The user's mental model continues through the call, so that is the experience the system must manage. Kim recommends dynamic handling of recognition, pronunciation, vocabulary, turn-taking, emotion, context, and changing user expectations. She points to ServiceNow's Eva bench as a way to diagnose an agent.
A small spelling error can become a complete conversation failure
Kim begins with a call in which the bot asks her to spell Midam. It hears the final M as N, then reads the name as "Madam" after she corrects it. The bot next asks for an account number without making clear which number it needs. Kim searches for it, reads "A X 4 5 1" slowly, and gets interrupted. The system says it cannot find her record and asks her to repeat the same information. Kim ends the call by asking for a person. She presents this as a typical pattern in voice AI, where several small failures compound into frustration and escalation.
Kim describes human communication as a joint activity. One person produces sounds and words, the other hears them, and the exchange continues back and forth through interaction. Both sides continuously process what is happening and update their mental models. She argues that voice AI should work the same way because this is the communication pattern people know from human interaction. Users therefore expect a bot to respond to their timing, words, intentions, and changing understanding during the call.
The failed call contains errors at several linguistic levels
Kim replays her call using the framework. The system fails to distinguish M from N in speech recognition. Its text-to-speech system applies English reading rules to her name. When she searches for the account number, the bot does not track her confusion. It then fails to recognize the word unit correctly and talks over her. Finally, it does not track her mental model or use interactive clarification. The call is difficult because recognition, pronunciation, timing, and task understanding fail in sequence.
Kim's map has a listening channel and a speaking channel for each party. Across those channels are sounds, words, interaction, and mental model. Listening includes recognizing speech, understanding words, waiting for the right turn, and understanding the user's intention. Speaking includes pronunciation, choosing words the user can understand, speaking at the right time, and providing the information the user needs. Engineering, linguistics, and cognitive science terms can be placed into these cells.
Kim says the framework's components are interdependent and aligned. Work on sounds also requires attention to words. Work on sounds and words must account for interaction, including turn detection and turn-taking. Task completion depends on the mental-model layer as well. If any component is missing, she says, the voice agent can fail. The system also has to manage these relationships over time rather than treating each problem as an isolated feature.
Speech disappears while the user's mental model continues
In chat, the conversation history remains visible as text. In a voice call, sounds, words, and interactions vanish as soon as they are spoken. Kim says the user's mental model is what remains and grows over the timeline. That continuing model is therefore the target for user satisfaction. A system can produce technically correct audio and still leave the user confused if it does not maintain a shared understanding of the call.
Improvement requires dynamic orchestration across the call
Kim recommends choosing and configuring speech-recognition and text-to-speech models, using post-processing and pre-processing, and curating vocabulary shared by the bot and user. She also names turn detection, latency, turn-taking, emotion detection, emotion handling, and context retention. These decisions must change during the call and for different kinds of people, including children and users who are unhappy. She says only this kind of dynamic management can support scalable voice AI.
Kim connects the linguistic framework to business outcomes such as user frustration, task failure, live-agent escalation, abandoned calls, and silent failures. She says ServiceNow has built an end-to-end benchmark called Eva bench for diagnosing a voice agent's status. Her advice is to use the framework to inspect a system rather than look for one fix. She also recommends learning linguistics and hiring linguists, because she sees business implications and linguistic implications as connected.
Users and language change during the life of an agent
Kim says speakers adapt to one another. After hearing her talk, an audience member may understand her speaking style and feel more comfortable in a later conversation. Users similarly adapt to a voice agent throughout a call. Kim asks whether the system is ready for people to use it better the next time. She also asks whether it can handle language change within six months or a year, since pronunciation, vocabulary, and expectations do not stay fixed.
