Voice changes LLM app design because conversations are fast, interruptions are normal, and users change subjects without warning.
2
Tolan routes each turn by emotional stakes, retrieves compressed memories, and rebuilds context instead of reusing a stale prompt.
3
Tolan uses Claude agents across implementation, review, bug triage, and character development, with humans checking the resulting work.
Summary
Paula Dozsa explains why Tolan treats voice as a different engineering problem from text chat. Users expect the system to respond within a couple of seconds, interrupt it, change subjects, and speak while doing something else. A half-second latency increase caused broad product damage, so Tolan measures every pipeline stage and accepts extra latency when it prevents bad interruptions. The team routes turns by emotional stakes, sends lighter exchanges to smaller models, and uses retrieval-based memory instead of passing full transcripts. It rebuilds context on every turn from recent messages, persona, retrieved memories, emotional guidance, and app state. Dozsa also explains why Tolan uses an alien character and how the team keeps personality stable. In the second half, she describes a fleet of Claude agents that writes, reviews, tests, and fixes iOS code. The team also uses agents and production evaluations to create new characters.
Voice makes conversation fast and context unstable
Dozsa says text chat assumes slow turns and stable context. Users wait, read, and usually stay on topic. Voice has to move from the user's finished sentence to Tolan speaking in under a couple of seconds. People talk while cooking, walking, or falling asleep. They change subjects mid-sentence and interrupt. That volatility changes the engineering problem because the system cannot assume a clean, linear conversation. Dozsa gives the example of someone discussing a breakup, suddenly asking whether they left the oven on, and then returning to the original story.
A short response such as 'yes' or 'yeah' should not always count as a turn, and Tolan must avoid speaking over a user who is still forming a thought. Dozsa says the team stopped optimizing for fewer interruptions in general and focused on reducing cases where the agent jumps in too early. Its turn-taking system reads speech patterns to decide whether an interruption is real. The team accepted about 60 milliseconds of extra latency and cut the worst early aborts by more than half.
Dozsa breaks the voice loop into separate stages: detecting the end of an utterance, transcription, the model's first token, the end of generation, text-to-speech producing its first byte, and playback. Saying that the product feels slow does not identify the cause. She says moving to GPT-5.1 on the Responses API cut time to speech by more than seven-tenths of a second. That improvement mattered because the full round trip has to stay within the short timing window that makes spoken conversation feel natural.
Tolan uses a tiered model fleet. A frontier model handles relationship-defining moments such as onboarding, a user's first conversation, the first few days, and emotionally serious exchanges. A cheap classifier called the tone router reads the emotional state on every turn. Casual back-and-forth can use smaller, faster models, while crisis or therapist-style tones always receive the strongest model. Dozsa says the frontier model costs roughly five times as much as the smaller one, and routing about a third of turns to the small model has had almost no measurable effect on retention.
Memory is retrieved and compressed rather than replayed
Keeping a complete transcript does not fit Tolan's two-second loop and causes long sessions to degrade. Instead, the system extracts facts, preferences, and emotional signals, embeds them, and stores them in a vector database with sub-50-millisecond lookups. Each night it merges duplicates, clusters related memories, resolves contradictions, and removes noise. Tolan also generates internal questions about the person and the relationship, then retrieves memories against those questions. Stable memories are separated from volatile information that stays in the live tail of the prompt.
Context is rebuilt because voice conversations drift
Reusing context across turns can keep a cache warm, but Dozsa calls it a trap when the user pivots. The reused context becomes actively wrong. Tolan reassembles the context window on every turn from a recent-message summary, the user's persona card, newly retrieved memories, tone guidance, and real-time app state. This approach matches the way people move between subjects in spoken conversation instead of assuming that the current topic will remain stable.
Dozsa says Tolan chose an alien because it has no fixed real-world reference, so users can project onto it and make it what they need. The baseline character is bubbly, youthful, and irreverent. Alien behavior can also be impulsive or chaotic without violating the user's expectations as sharply as a human character might. The team uses a tone-monitoring system to change how a line is delivered in response to emotional cues while preserving the character's identity across hundreds of turns.
Agents work through code, reviews, and evaluations
Dozsa says Claude had co-authored more code in the iOS app than any individual engineer by late last year. The team makes the codebase carry much of the agents' context and runs implementation agents, review agents, a pull-request shepherd, and a bug-triage bot. These systems use MCP with Linear, Sentry, and Datadog. For a new character aimed at an older demographic, agents mapped personality-related code, wrote a voice bible, and had five judges test different dimensions against production logs through repeated fix-and-verify rounds.
