Domain-specific systems matter more than model sophistication when an LLM must handle the last mile of an industry workflow.
2
A failure mode ontology connects domain expert reviews to measurable priorities and production-derived evaluation datasets.
3
A domain expert product manager can direct engineers through a tight loop of experiments, evaluations, and production decisions.
Summary
Christopher Lovejoy argues that vertical AI applications reach a baseline with strong models, then need a separate system to capture domain context and improve the final part of performance. He illustrates the problem with Florence, Anterior's clinical reasoning system, which must interpret ambiguous requirements around conservative therapy, symptom improvement, and medical-record documentation. His proposed adaptive domain intelligence engine combines user-defined metrics, a failure mode ontology, production data, domain expert review, and targeted evaluations. Clinicians label incorrect outputs, identify how they failed, and suggest additions to the system's domain knowledge. A domain expert product manager uses this information to prioritize work and give engineers measurable targets. Engineers can then test prompts, models, fine-tuning, or other changes against failure-specific datasets before a production decision. Lovejoy is clear that the approach requires bespoke tooling and appropriately qualified reviewers, with the level of expertise depending on the workflow.
The last mile is about workflow context, not raw model power
Lovejoy says models have become capable of reaching a strong baseline, but specialized applications still struggle with the last mile. The model must understand the particular workflow, industry, and customer context well enough to make the required decision. At Anterior, models reached about 95% on the primary task of approving care requests. The team then used its surrounding system to get close to 99%. His claim is that the winning company in a vertical industry will build the best process for turning domain insights into pipeline changes and iterating with customers.
Clinical questions contain hidden decisions that domain experts must define
Lovejoy uses a knee arthroscopy review to show why a seemingly simple question can be difficult. Florence must determine whether the record documents at least six weeks of unsuccessful conservative therapy. Conservative therapy can mean physical therapy or weight loss, but medication may be conservative in one case and more aggressive in another. The team must also decide whether partial symptom improvement counts as success and how much can be inferred from a record that mentions treatment once. These choices are domain rules that a general model cannot safely infer on its own.
Metrics should describe the business harm customers want to avoid
The first measurement step is to define what users care about. In Anterior's health insurance work, customers want to minimize false approvals, since an approval can lead to payment for care that was not needed. Other examples have different priorities. Contract analysis might focus on missed critical terms, fraud detection on preventing dollar loss, and education on test score improvement. Lovejoy recommends forcing the team to identify the one or two metrics that matter most, with domain experts and customers working together to define them.
A failure mode ontology turns errors into an actionable map
A failure mode ontology lists the different ways an AI system can fail. For medical necessity review, Anterior groups failures into medical record extraction, clinical reasoning, and rules interpretation, with more specific subtypes beneath them. Lovejoy says this structure should be led by people with domain knowledge rather than by someone inspecting AI traces without workflow context. The ontology becomes useful when paired with the main performance metric because the team can see which failure modes produce the most false approvals and address them in priority order.
Production reviews create evaluation sets that match real inputs
When domain experts label failure modes in production cases, the company gets ready-made datasets for iteration. These cases reflect the real input distribution more closely than synthetic data. If one failure mode causes many false approvals, the team can collect the relevant production cases and give that set to an engineer. Each pipeline version can then be tested against it. The resulting chart shows whether a change improves the chosen failure mode and whether it causes regressions elsewhere.
Domain experts can suggest knowledge changes through the same review tool
The review interface can let a clinician mark an output as correct or incorrect, assign a failure mode, and propose domain knowledge that could fix the problem. Lovejoy gives examples involving the meaning of suspicion in a medical context and a scoring system that the model cannot access. Those suggestions can be evaluated before they enter production. The team can choose automatic release through evals or keep a person in the loop. In his example, a production case can be reviewed, addressed with added knowledge, tested, and released on the same day.
A domain expert PM translates clinical findings into engineering targets
In the proposed team structure, domain experts review live outputs and provide metrics, failure modes, and suggested improvements. A domain expert product manager sits between that work and engineering. The PM can tell an engineer to fix a specific failure mode and set a target, such as raising its score from 0% or 10% to 50%. The engineer can try prompt changes, different models, fine-tuning, or other experiments against the prepared eval set. The PM then reviews the measured impact and the wider product context before deciding whether the change should go live.
Reviewer expertise must match the workflow being evaluated
Lovejoy says there is no universal definition of a domain expert. For clinical reasoning, Anterior wants someone with substantial clinical experience, ideally a doctor with relevant specialty knowledge. Simpler workflows may only require a more junior clinical reviewer or someone who has performed the workflow in practice. He also says Anterior initially hired people in-house to generate review data, although customers could eventually validate the AI themselves through a customer-facing version of the tooling.
"These data sets are super valuable because they're coming directly from production data, which means you know that they're representative of the kind of input data distribution that you're going to see more so than synthetic data would be."10:59
Who should watch
You are building an AI product for a regulated or specialized workflow where a general-purpose model reaches a good baseline but still makes costly domain-specific mistakes.
Your team has clinicians, lawyers, operators, or other specialists reviewing outputs, but those reviews do not yet feed into metrics, eval datasets, or engineering priorities.
You need a practical division of work between domain experts, product managers, and engineers for improving an LLM application with production cases.
