A hybrid agent system lets patients manage multi-day medication treatments by text while keeping human operators involved when interactions are uncertain or complex.
2
Treatment blueprints in Google Docs give the agent approved language and workflow guidance, so new treatments can be added without changing application code.
3
A separate evaluator scores each interaction for confidence and complexity, sending cases below a threshold to a human for approval or feedback.
Summary
Dan Mason walks through a telemedicine support system that Stride built with Aila Science for at-home treatment after early pregnancy loss. Patients communicate with Ava by text message. The agent reads a treatment blueprint, tracks anchors such as medication events and time zone, schedules reminders, answers questions, and updates the patient state. LangGraph provides the visible workflow, while MCP connects the agent to treatment documents and the surrounding application. A second agent evaluates the proposed response and can escalate it to an operations associate. Mason is direct about the system's limits. Confidence scoring is hand-tuned, language models can misjudge their own work, and token costs are significant. The design relies on approved medical language, restricted patient data, retries for malformed tool calls, and human review for complex cases. Blueprints remain editable by the client's medical team, which lets them create new treatment workflows without writing more code.
The agent replaces button-pushing in a human-operated treatment workflow
The original product required an operator to read each patient's message and click a button representing the patient's position in a treatment flow. Patients might report taking medicine, ask about bleeding, or raise another treatment question. Aila needed to support more patients without adding operators and wanted to reuse the system for other regimens. Stride rebuilt the decisioning around an LLM while keeping the existing operations team involved. The virtual operations associate proposes messages, questions, actions, and state changes. Human operators become supervisors who review cases the system marks as complicated or uncertain.
Treatment blueprints keep medical guidance editable outside the codebase
The system uses treatment blueprints and a knowledge base containing medically approved language. A blueprint describes the treatment, its phases, anchors, schedules, triggers, and the messages that apply in different situations. The documents are maintained in Google Docs, exported to Markdown, and checked into the application. They are structured and self-referential, so an overview points the agent to the document for the current phase and to triage material for questions outside the normal flow. Mason says the format is intentionally readable by the client's physician's assistant. Adding a new treatment can therefore involve new documents rather than new state-management code.
The virtual operations associate maintains patient state through tools
For each incoming patient message, the agent receives the current state, reviews the conversation, and decides what should happen next. It can answer a question, update patient data, modify anchors, schedule messages, or change the treatment phase. An anchor records an event such as when medication was taken. If a patient corrects the timing, the system can change the anchor and recalculate later messages. The LLM does not directly edit every field. Python controls which state operations are available, and the graph serializes the resulting state when the interaction ends. The durable system stores the exchanged messages, scheduled unsent messages, and treatment state.
The system favors complete context documents over retrieval for treatment logic
Stride chose not to use retrieval-augmented generation for the main treatment materials. The agent reads a small set of framework documents, the current treatment overview, and the blueprint for the patient's phase. The documents point to one another, and the model can request triage material when the blueprint does not answer a question. Mason says the team did not trust retrieval to return every piece of context needed to understand the shape of a treatment. The documents are small enough for Claude's context window, and the explicit links make the agent's information path easier to inspect. A less frequently used question-and-answer CSV could be a candidate for RAG later.
The agent answers curveballs while trying to return the patient to the blueprint
Patients do not always answer the question the system asked. Someone might be asked whether they took the medication and reply that their stomach hurts. The agent is instructed to answer the patient's question first, then gently ask the treatment question again. It can also skip ahead when a patient says they already completed an earlier step. The blueprint says not to ask questions the patient has already answered. If the issue falls outside the approved material, the agent can use the triage knowledge base or escalate. Some situations instruct the system to stop texting and direct the patient to a doctor, call 911, or get human help.
A separate evaluator combines confidence with interaction complexity
After the virtual operations associate proposes changes, an evaluator agent reviews the interaction. It considers whether the patient's intent is understood, whether the response follows the available guidance, and whether the proposed state changes look right. It also deducts for circumstances that are complex even when the message itself appears correct, such as changing patient data, setting an anchor, scheduling multiple messages, creating duplicate messages, or leaving reminders for events that already happened. In one example, several simultaneous changes produced a 70% score, below the configured 75% threshold, so a human had to approve or revise the response. Approval sends the messages. Feedback restarts the active thread so the agent can apply the instruction.
The architecture uses retries and invalidation to contain model failures
The graph includes retry paths for malformed tool calls and premature turns. If the model produces invalid JSON for a tool call, the system detects it, removes the bad message, and asks the model to try again. This matters because leaving the malformed call in context can lead the model to invent the tool result or the blueprint it was supposed to read. The system also handles rapid patient messages by waiting for a configurable delay and invalidating earlier running threads when a newer message arrives. It then processes the combined context and responds once. These are software controls around the model rather than treatment-specific decision nodes.
Evaluation measures whole output states against replayed conversations
Stride stores datasets and traces in LangSmith, then runs a separate evaluation harness that preprocesses dates, times, and other details before replaying conversations. The harness calls the LangGraph agent and uses PromptFu's LLM rubric to compare the new output state with the expected state. The rubric allows harmless wording differences while checking for broken behavior, such as missing scheduled messages or incorrect state changes. The team tests happy paths and known edge cases from prior conversations. Human-reviewed failures can lead to changes in the blueprint or prompts, but the feedback is not automatically added to model training data.
"We needed something that was able to do the job, clear about what it was doing, and that was steerable by humans in a really obvious way."12:53
Who should watch
You are building an LLM-driven workflow where a human team needs to inspect and override model decisions.
Your business process changes often, and domain specialists need to maintain guidance without editing application code.
You need to evaluate full state transitions and scheduled actions rather than judging a chatbot's final text alone.
