Voice agents need to start speaking in about 950 milliseconds, so a model that spends a full second reasoning has already missed the interaction.
2
Ace moves lesson planning, student-state tracking, and turn coordination into a state machine and other code outside the model.
3
A small model can work well for voice when scaffolding gives it a strict brief, but it can drift on long structures without those rules.
Summary
Joel Allou and Ornella Bahidika describe Ace, a live AI voice tutor built around a small model. Their model choice comes from the timing of voice conversations. Ace needs the model to start talking in about 950 milliseconds, while a frontier model that reasons for a full second already creates a noticeable break. The system therefore moves lesson planning, tracking what the student knows, deciding what happens next, and other logic into a state machine and related code. The model receives a structured summary each turn and focuses on speaking. In their comparison, a larger Opus 4.7 model takes a couple of seconds to answer a simple question, while Haiku 4.5 answers in about 900 milliseconds with the surrounding system in place. The tradeoff is engineering work. Small models can drift on long structures, so they need strict rules and explicit scenarios. That scaffolding is paid for in code rather than on every model turn.
Voice agents have about 950 milliseconds to start speaking
Ornella says a pause of even one second can make a voice agent feel broken. Ace therefore sets its model a target of starting to talk in about 950 milliseconds. A frontier model that spends a full second thinking has already lost the interaction, regardless of the quality of its eventual answer. For this application, the practical constraint is milliseconds rather than model intelligence measured in the abstract. The team chooses a small model because it can fit the timing required for a live conversation.
Ace moves lesson decisions out of the language model
The model does not decide what happens next in the lesson, track what the student knows, or explain which step comes next. A separate system handles those jobs and sends the model a summary on every turn. Joel describes this as extracting the thinking from the model so it can focus on speaking. Ace uses a state machine to coordinate lesson steps and an additional layer to derive some of the mastery a student needs before the lesson is complete. Logic about display, lesson progression, and answers happens outside the model.
A state machine supplies the structure that reasoning would otherwise provide
Joel says frontier models such as Claude 4.7 are good at reasoning through a lesson and a student's question, but that reasoning can take several seconds. Ace instead encodes the scenarios needed for a lesson in a state machine. The system coordinates each step, determines what should happen next, and produces the information the model needs to say the answer. This leaves the model with a narrower task. It does not need to reconstruct the lesson state from the conversation before generating each response.
The small-model setup cuts response time from seconds to about 900 milliseconds
The demonstration compares a simple question answered with Opus 4.7 against the same question answered with Haiku 4.5 and Ace's surrounding implementation. In the first example, the larger model reasons for a couple of seconds before returning an answer. In the second, the smaller model responds in about 900 milliseconds. Joel attributes the difference to removing thinking, logic, and reasoning from the model and putting them into code. The result feels almost instant because the system has already done the smart parts before the model speaks.
Small models need strict scaffolding to stay organized
Joel is direct about the cost of this design. Haiku 4.5 can drift when it has to maintain a long structure without scaffolding, so the system needs strict rules. The team pays for that work in code rather than on every model turn. Their rule is to choose the fastest model allowed by the latency budget, then spend engineering time on the surrounding state machine, reasoning process, scenarios, and handling rules. They apply the same approach to other real-time and high-volume applications where latency matters.
"Pick the fastest model that your latency budget allows and then spend the rest of your time actually building the scaffolding."04:20
Who should watch
You are building a voice agent and a model's first-token latency is making conversations feel interrupted.
Your application needs lesson state, workflow control, or other structured decisions that a model currently handles slowly.
You want to use a small real-time model and need a concrete account of the state-machine and rule-based work required around it.
