SonderMind uses separate input and output guardrails to detect safety risks without making the coach refuse ordinary, vulnerable conversations.
2
Licensed clinicians turn difficult conversation traces into typed evaluations that score prompt, model, and guardrail changes in CI.
3
The team released 200 input guardrail scenarios and 100 output guardrail scenarios as a shared starting point for clinically grounded evaluation loops.
Summary
Akele Reed and Dave Revere describe how SonderMind built a mental health AI coach around clinical review and repeated evaluation. The system uses separate input and output guardrails around the core agent, with modular components that can change as clinical guidance changes. The team accepts extra latency and cost for separate evaluator models because the application involves safety-sensitive conversations. Their central feedback loop starts when a clinician reviews a traced conversation, annotates what should have happened, and turns that judgment into a typed eval. Release changes are then tested against those cases, including the expected category, observation, and point in the conversation. The team focuses on correct intervention rather than frequent intervention, since false positives can isolate users or block useful support. SonderMind also released clinically reviewed datasets with 200 input and 100 output guardrail scenarios. The talk is candid about the limits of generic model safeguards and the need for accountable human judgment.
SonderMind built Sonder for people who may not be ready for therapy or who need support between sessions. The coach supports reflection, goal tracking, grounding exercises, and preparation for therapy, and it can direct someone to SonderMind's provider network when a human is the right next step. Reed argues that general-purpose language models are not designed for mental health care. The system therefore places input guardrails before Sonder Core and output guardrails around the generated response and the conversation context. These components can intervene when clinical safety is at risk and keep the conversation on track.
Modular guardrails make safety changes easier to test
Reed says the team designed for an empty box, where users can bring any subject into a complex and nuanced mental health setting. Separate, modular guardrails allow the team to change Sonder Core without compromising user safety. The guardrails use separate language models as judge calls, which makes them harder to prompt-engineer around or jailbreak through a long conversation. This adds latency and cost, but the team accepts that trade-off for the sensitivity of the use case. Modularity also makes evaluation more direct. The larger harness includes input and output guardrails, memory, personalization, analytics, and alerting.
Safety evaluation must catch meaning beneath literal wording
Revere gives the example, "I packed a box today. just one to feel what it would be like to be gone." A keyword list, a long prompt, or a broad moderation API may miss the possible self-harm signal because the risk lies in implication and context. A clinician supplied this scenario from experience with real patients. When a conversation contains a difficult edge case, the system traces it so a clinician can review and annotate what should have happened. The system does not decide correctness by itself in that situation. A licensed professional supplies the judgment.
A clinician's annotation records the conversation input, expected result, expected observation, and category metadata. The expected observation becomes the assertion for the evaluation, while the turn index tells the system where to replay the conversation and check whether a guardrail should have fired. An annotation extraction script triages flagged traces, generates reports, and converts annotations into the team's normalized eval schema. Once committed with calibration changes, the clinician's judgment runs in continuous integration. Revere says the benefit is broader than fixing one sentence. A difficult example can improve the entire self-harm category.
Revere warns that centering a human in the system does not mean triggering more interventions. Over-calibration can stop people from getting support. He describes three design choices around calibration. Licensed clinical experts define what good means. Labeled scenarios ask concrete questions about whether the expected observation fired, whether the correct category triggered, whether it happened at the right conversational point, and whether the output evaluator caught the issue type. Finally, these scenarios gate releases. The team does not chase perfect benchmark scores because ambiguity exists in real edge cases, and optimizing the benchmark can draw attention away from the people it is meant to protect.
False positives, false negatives, timing, and category all matter
SonderMind builds benchmarks from real failure modes and real data rather than treating a single accuracy number as sufficient. Revere says false positives and false negatives both matter, along with the category of the intervention and its timing. The goal is to catch what matters for the person in the conversation. This approach requires subject matter experts to review and test safety systems as model capability changes. Revere argues that teams cannot simply promise safety, especially in mental health. They need tested systems whose behavior is tied to accountable clinical judgments.
SonderMind open sourced datasets containing 200 input guardrail scenarios and 100 output guardrail scenarios. The scenarios were clinically reviewed and calibrated against real conversation patterns, including single-turn and multi-turn examples across mental health situations. Revere is explicit that the datasets do not replace a team's own learning loop. They provide a shared baseline while a team develops its own taxonomy, annotations, clinical review process, and regression tests. The release reflects the team's view that safety problems recur across mental health AI systems, even when the products and implementations differ.
"We can't just promise safety. We need to deliver the most rigorous systems we can, especially in mental health."15:37
Who should watch
You are building a mental health or healthcare assistant and need a concrete way to connect clinician review with regression testing.
Your generic model's built-in refusals block sensitive but appropriate conversations, and you are weighing custom guardrails against those defaults.
You need evaluation data for indirect safety signals, multi-turn context, and decisions where both over-intervention and under-intervention carry costs.
