Patient-facing healthcare AI cannot rely on A/B tests, rollback, or vendor benchmark scores because a bad call can harm someone immediately.
2
Ufonia's Matrix framework uses simulated patients, clinician-defined hazards, and a validated language-model judge to test Dora before real patients hear it.
3
The team improves the system by optimizing prompts against clinical cost functions, then moves from simulation to supervised real-world deployment as evidence grows.
Summary
Jared Joselowitz explains how Ufonia evaluates Dora, a voice agent that calls patients for post-operative follow-ups and pre-operative checks. Dora has made around 200,000 clinical calls across 20 UK hospitals, with a contract to reach a million patients within two years. Because it asks about symptoms and gives advice, it is a regulated medical device. Standard software release tactics fail here. A 5% rollout could expose thousands of patients, and a rollback cannot undo a spoken call. Ufonia starts with clinical hazards, then uses Matrix to simulate conversations with an LLM patient and grade them with a second LLM judge. The simulated patient was judged more realistic than a real patient in three of four comparison sets. The judge matched or slightly exceeded clinicians on 240 cases, with near-perfect sensitivity. Prompt optimizers then improve the system against a cost matrix. Simulation allows fast iteration, but supervised testing with real patients remains the final source of evidence.
Patient-facing AI removes the usual software safety nets
Jared Joselowitz says healthcare AI loses three familiar release protections when it talks to patients. Teams cannot A/B test by deliberately assigning someone to a worse variant, since that can be unethical and illegal. They cannot undo a call after the words have been spoken. A vendor's model card and a score such as 92% on a benchmark will not explain or defend a harmful clinical interaction in a post-incident review. These constraints shape Ufonia's safety and evaluation work for Dora, rather than being handled after the product is already deployed.
Dora is a voice AI agent that phones patients for conversations such as post-operative follow-ups and pre-operative checks. These calls would usually take time from clinicians. Joselowitz says Dora does not replace doctors, but takes this repetitive work off their schedules. The system has completed around 200,000 real clinical calls across 20 UK hospitals. Ufonia is contracted to scale to a million patients within two years. The company also launched in the United States, where Dora was live in two clinics and signed up for six more across four states at the time of the talk.
Giving symptoms advice makes Dora a regulated medical device
The demonstration shows Dora asking about blurry vision after cataract surgery, clarifying when the symptom began, explaining that vision should improve in the first few days, and advising the patient to avoid swimming for a month. Joselowitz says this combination of asking about symptoms, giving advice, and answering questions makes Dora a medical device. Its safety case therefore has to answer three questions: what the software does, what could go wrong, and how the team ensures that the failures do not happen.
Ufonia begins by listing what could actually hurt a patient. Examples include missing sudden vision loss or severe pain, inventing an answer to a medical question, and ignoring a distressed patient while continuing the conversation. Joselowitz says the team has documented many hazards, with dozens to account for. This changes the evaluation target from general model quality to specific clinical failure modes. The team writes scenarios around those hazards and checks whether Dora triggers them before exposing real patients to a change.
A small rollout and rollback cannot contain patient harm
The normal software loop is to release to a small percentage of users, watch a dashboard, roll back a bad change, and iterate. Joselowitz says this works because a team can afford to be wrong briefly when users are not literally harmed. Five percent of a patient population could still mean hundreds or thousands of people receiving unproven care. A rollback cannot reverse a conversation that has already happened. By the time a dashboard turns red, a patient may already have been hurt, so the reactive loop is no longer enough.
Matrix simulates clinical conversations before deployment
Ufonia borrowed an idea from self-driving cars, which use large amounts of simulation before carrying passengers. Its clinical simulation framework is called Matrix. An LLM called Patbot plays a patient in a defined clinical scenario, then has a conversation with Dora. Scenarios are grounded in real clinical workflows and can include behaviors such as asking whether the agent is human or AI. Simulated patients allow the team to test many situations quickly. The framework does not rely on hired actors, which Joselowitz says would be too slow for rapid iteration and varied cases.
Both the simulated patient and the judge need validation
The team first checked whether Patbot followed its assigned script, then tested whether its conversations seemed like real patient interactions. In a patient and public involvement study, real patients compared a conversation between a real doctor and patient with one between Dora and Patbot. Across four conversation sets, most participants thought the simulated patient was more realistic in three sets. Matrix then uses another LLM, called Bev Judge, to inspect each dialogue against expected behaviors and clinician-defined hazards. It returns pass or fail, identifies triggered hazards, and explains what went wrong.
Joselowitz describes prompt behavior as brittle. Formatting changes and reordering few-shot examples can produce large changes in benchmark performance, so manual prompt tuning is subjective, hard to reproduce, and slow. Ufonia uses a prompt optimizer called GEPA, from the same group associated with DSPy. The optimizer reviews failures, asks a stronger model to reflect on them, and updates the prompt through repeated iterations. The team defines what good means with clinicians, then gives the optimizer a cost matrix. Missing a red flag receives a much larger penalty than raising a mildly annoying false alarm, so the process can optimize for sensitivity or another clinical metric.
Simulation earns the right to test with real patients
Passing simulated tests does not prove that Dora helps real people. Simulation is the fast inner loop, where thousands of runs can happen without exposing anyone. Real patients remain the outer loop and provide the only real-world proof. Ufonia moves through stages, starting with simulation, then user testing, supervised clinical evaluation, and monitored deployment. Clinicians stay involved at each stage. The amount of autonomy depends on the evidence collected. Each call, dataset, pinned prompt, and judge verdict is traced to the hazard it addresses, because the deliverable for regulation is the evidence behind the model.
"You can't actually AB test on patients. Of course, randomizing patients into a worse variant is unethical and often illegal."00:50
Who should watch
You are releasing an AI system that asks users about symptoms, gives advice, or can cause harm through a single interaction.
Your current safety process depends on a small rollout, dashboards, and rollback, but you cannot reverse what the system says to a user.
You need a repeatable way to turn clinician-defined hazards into simulations, evaluation metrics, prompt changes, and deployment evidence.
