Healthcare AI safety failures often come from architecture choices made before the model generates a token.
2
Emergency escalation, intent routing, and identity verification should run in deterministic code before the model sees a turn.
3
Production safety needs continuous judges, member feedback, human review, and a check that the judge itself is correct.
Summary
Rashi Agrawal describes how Hinge Health approaches safety for member-facing healthcare AI. She starts with failures from consumer health assistants, including dangerous diet advice and under-triage of emergencies. Her design uses architecture to protect PHI before it reaches storage, separates production from non-production, and limits access by role and region. Deterministic code handles decisions that cannot be allowed to fail, such as routing someone to 911 or 988, choosing a high-stakes capability, and verifying identity. The model handles the remaining conversation. After launch, automated judges, member feedback, sampled traces, and human review monitor live traffic. When stakeholders disagree about a launch, Agrawal recommends judging severity by the worst plausible harm, defaulting toward the safer mistake for safety issues, and making risk acceptance explicit. Before changing an agent because a score falls, teams should verify that the judge is right. A false judge signal can lead to an unnecessary or harmful fix.
Healthcare AI is already a production safety problem
Agrawal says about 40 million people use frontier models to triage healthcare issues. She gives a case of a healthy 60-year-old man who asked how to reduce salt, followed advice to use sodium bromide for three months, and reached the emergency room with paranoia and hallucinations. His bromide level was 200 times the safe limit, and he spent three weeks in the hospital. She also cites a Mount Sinai safety test in which a consumer health AI under-triaged life-threatening emergencies half the time. ECRI named chatbot misuse the top health technology hazard of 2026.
Architecture determines what failures are possible
Agrawal sets out three foundations: the constraint is the architecture, deterministic rules belong above the model, and safety is a continuous evaluation layer. She says many healthcare AI failures come from decisions made before any token is generated. Launch is where production risk begins, rather than where safety work ends. The architecture should make certain failures impossible instead of relying on policies that people must remember to follow. When architecture is not enough, teams need a framework for human decisions about what to change, delay, or accept.
Agrawal contrasts a reactive approach, where teams redact PHI from logs or dashboards, with an architectural approach. Her recommendation is to strip PHI at the pipeline boundary during ingestion, before it reaches the data lake. A developer opening a dashboard then has no PHI to redact because it was never stored there. Production and non-production environments should remain completely separate, with no pipes between them. Access to raw PHI should depend on both role and geographic region, and HIPAA, FDA good machine learning practice, and state laws should shape the system from the start.
High-stakes decisions belong in code before the model
Agrawal says probabilistic models are useful for generation but unreliable for behavior that must never fail. A code layer should run first on every turn and make irreversible decisions before the model runs. That includes deciding whether a situation requires emergency escalation, whether to route someone to 911 or 988, or whether a clinician should enter the loop. The model handles the long tail of conversation, but it does not get a vote on high-stakes calls. Her mental model is a stack with code above the model.
Agrawal points out that a model is not a guardrail, and a model with a system prompt is not a guardrail either. She describes the authority hierarchy published by frontier labs, with root, system, developer, user, and guideline layers. Every layer above the user can be overridden through prompt injection, in her reading. If the labs do not trust a prompt as a security boundary, she says healthcare teams should not trust it either. Authentication, which checks who is on the other end, belongs in code because it is a security boundary.
Emergency routing and intent ownership need deterministic paths
For self-harm, suicidal ideation, or an acute medical emergency, Agrawal says code should route the member to 911 or 988 before the model sees the turn. Code should also help choose which capability owns a conversation, such as clinical support, technical support, education, or exercise recommendations. A clinical question should not quietly reach a generic technical support agent. Identity verification must happen before any capability touches member data. Across these cases, code makes the irreversible decision and the model handles what remains.
Production safety requires live signals and human review
Agrawal rejects treating evaluation as a pre-launch checklist. She recommends judges that continuously score live conversations across clinical accuracy, safety, escalation, relevance, drift, and refusal. Member thumbs-up and thumbs-down feedback catches tone problems and issues the judges miss. Random trace samples should cover capabilities, with 100% sampling for high-stakes cases. People still need to read the signals and act on them. She says the bottleneck is having enough people for that work, not compute or model capability. A new production failure should become a new judge or monitoring rule.
Launch decisions should follow harm, not schedule pressure
Agrawal describes five stakeholders reviewing one issue five days before launch. Clinical sees member safety risk, legal sees regulatory exposure, compliance sees audit risk, product sees adoption risk, and engineering sees velocity risk. Her decision framework starts with the worst plausible outcome, not the average outcome. Severity comes from the harm caused, not from team capacity or fix difficulty. Teams can fix, delay, or accept a risk with explicit sign-off. For safety bugs, uncertainty should lead to holding and fixing. For small polish bugs, uncertainty can favor shipping.
Agrawal gives a production example where a clinical accuracy score falls from 4.9 to 4.5 and stays there. The first reaction might be to change the agent's prompts, but the score could be wrong. In one caffeine example, the agent gives FDA guidance of 400 milligrams for most adults and adds context about pregnancy and medications. The judge flags that context as a hallucination, so the judge needs fixing. In another example, the agent says 1,000 milligrams per day is fine, and the judge correctly flags the unsafe answer. Teams should ask whether the judge is right before changing the agent.
"Most AI safety failures in healthcare are not model failures. They are architectural decisions that were made before even a single token was generated."02:35
Who should watch
You are building a healthcare assistant that may receive member messages, clinical questions, or sensitive personal data.
Your team currently relies on prompts to handle emergency escalation, routing, or identity checks.
You have live evaluations and launch disagreements, but no clear way to verify judges or make human review manageable.
