Voice agents combine the usual problems of language models with transcription errors, streaming constraints, and the latency demands of natural conversation.
2
A production interview agent needed separate agents for drift detection, choosing the next question, and hiding likely transcription errors from users.
3
Synthetic conversations and LLM-judged evaluation metrics made development less dependent on subjective impressions, even without a perfect ground truth.
Summary
Eddie Seagull describes Fractional AI's attempt to automate consulting-style research interviews with a voice agent. The system needed to feel like a conversation while asking a planned set of questions, following up when useful, and supporting many interviews without scheduling consultants. A single prompt quickly became too limiting, so the team added one-question-at-a-time prompting, tool use, goals and priorities, and background agents for drift detection and next-question selection. Audio introduced another problem: Whisper transcription could produce embarrassing text even when the main model understood the conversation. A separate agent therefore decided which transcript segments to hide from the user. As the system grew, informal testing became hard to manage. The team introduced automated metrics judged by an LLM and synthetic interviewees with different personas. Seagull's practical recommendation is to combine constrained tool use, out-of-band text checks, and imperfect but useful evals when building voice applications.
Voice agents make ordinary language-model problems harder to manage
Seagull begins with the familiar difficulties of language models: hallucinations, hard-to-design evaluations, and latency. Voice adds more constraints. Transcription is not as straightforward as text processing, and the system operates as a stream rather than a simple batch exchange. A conversational interface also has to respond quickly enough to feel fluid. Audio models can understand sounds directly, while a transcription model converts them into text, so the two parts of the system may interpret the same interaction differently. His point is practical: voice development inherits the existing problems of LLM applications and adds problems from audio, streaming, and timing.
The interview agent had to preserve human follow-up while removing scheduling overhead
The case study automates qualitative interviews that consultants conduct inside large companies to learn how employees do their work. Forms are cheaper, but they lose the human behaviors that make these interviews useful. An interviewer may improvise, build trust, or ask a follow-up question after someone starts to ramble. Seagull says people often provide more information when they can speak freely than when they must type answers to fixed questions. The proposed agent should feel like a conversation, interview hundreds of people at once, avoid the scheduling burden, and produce transcripts that can later be searched or aggregated.
One-question prompts and tool use gave the agent control over a visible interview roadmap
The first implementation used OpenAI's Realtime API with a large prompt containing the interview instructions and possible questions. That approach could not reliably tell the application which question the model was asking, and it made clicking around a question roadmap difficult. The team changed to one question at a time. The model could call a tool to indicate that it wanted to move on, after which the application supplied the next question deterministically. Separate prompts informed the model when the user revisited a question or skipped to a new one. This made the roadmap and question navigation possible without depending entirely on the model's internal state.
A drift detector was added because the interviewer kept falling into conversational rabbit holes
Once the agent could decide when to move on, it often refused to do so. The model liked chatting, asking follow-ups, and encouraging the interviewee, which led to long detours. Forcing the next-question tool solved that problem by removing too much improvisation. Fractional AI added a background drift detector, a separate text-based LLM call that read the transcript and judged whether the conversation was on topic, whether the current question had been answered, and whether it was time to continue. When the detector strongly indicated that it was time to move on, the main model was forced to call the tool.
Goals and a separate next-question agent gave follow-ups a reason
The one-question-at-a-time design still made the conversation feel linear. The model could dig deeply into the current question or move to the next one, but it did not always understand the purpose behind the question. The team added goals and priorities to the interview plan. For a question about daily responsibilities, a high-priority goal might be to understand regular activities, while a medium-priority goal might be to identify places where AI could help. A separate next-question agent read the live transcript and decided what should be asked next. The goals helped it judge whether an answer was sufficient and shape useful follow-ups.
Transcription errors needed a user-interface fix because the Realtime API exposed few controls
The Realtime API's main model understands audio, but Whisper handles the user's transcription. That separation created visible errors. Seagull describes silence being interpreted as a language switch and background noise becoming text such as "creeping Dippity Dippity Dippity Dippity." The main model could still understand that nothing meaningful had been said, but the user saw a bad transcript. Since the API did not provide enough low-level transcription controls, the team added another agent. It reviewed the conversation and decided whether a transcript block was probably wrong and should be hidden. The original text remained available to the system, while the interface avoided displaying it.
Informal fixes became difficult once every behavior depended on several agents and prompts
The system grew through repeated test interviews and local fixes. Seagull calls this a vibes-driven approach that can be acceptable early on, but it becomes hard to maintain. There were many prompts to update and several side agents whose responsibilities overlapped. When an interview exposed a problem, the team could not always tell which component to change or whether the change helped. A fix for one interview setting could create a regression in another. This made systematic evaluation necessary. The team needed measurements that could guide iteration instead of relying only on impressions from manually testing conversations.
LLM-judged metrics and synthetic interviewees made voice evaluation repeatable
Fractional AI created an evaluation suite with metrics for qualities such as clarity, completeness, and professionalism. An LLM judged each conversation using a tuned prompt. Seagull is clear that this is not perfect because there is no objective ground truth for a natural interview, but it is more useful than relying only on intuition. The team also generated synthetic conversations. Each simulated interviewee received a persona, such as a snarky teenager in charge of a Fortune 500 company, and answered as that character. Running the same evaluation suite across a roster of personalities and job functions produced average measurements over the kinds of people the system might interview.
"The basic kind of call the API, do some prompt engineering and get it into a good place is very helpful, it gets you very far in the development process, but it's not enough to build your robust app."17:25
Who should watch
You are building a voice application and need to understand where a simple Realtime API integration will break down.
Your conversational agent gets stuck in follow-up loops, loses track of its interview plan, or exposes poor transcriptions to users.
You need a repeatable way to evaluate an open-ended voice interaction before sending it to real users.
