Vertical AI success depends on how an organization uses expert judgment, not only on the sophistication of its models.
2
Domain experts can work as oracles, evaluators, or architects, depending on whether quality depends on taste, can be measured, and needs automated improvement.
3
A principal domain expert should own AI quality early, then grow with the product into evaluation or system design.
Summary
Chris Lovejoy argues that vertical AI is mainly an organizational problem. Models are often good enough, but products still need a way to capture the details of a customer workflow and decide what good output means. He presents three roles for domain experts. An oracle assesses outputs and improves the product directly. An evaluator defines quality metrics and works with engineers to improve performance. An architect designs systems that learn from usage and improve with less manual intervention. The right role depends on whether quality can be measured, how much variation the product handles, and whether manual iteration is fast enough. Lovejoy uses Granola, Tandem, and Anterior to show how teams can move from direct expert editing toward distributed review and automated improvement. He recommends appointing one principal domain expert, giving that person ownership, and hiring for relevant workflow experience plus adjacent skills such as prompting, data science, product work, or engineering.
Vertical AI needs an organization that can apply expert judgment
Lovejoy says the system for incorporating domain insight matters more than the sophistication of the models or pipelines. Vertical products have to handle the specific workflows and nuances of the customers they serve. He points to the gap between a capable front-end model and the organization around it, which must decide whether an output is good and how to improve it. He identifies three recurring mistakes: failing to hire domain experts or hiring them too late, hiring the wrong type of expert, and placing them in the organization without giving them the right responsibilities.
A domain expert is needed to decide what good AI quality means
Lovejoy's case for domain expertise starts with appraisal. A company needs to compare approaches by examining their outputs and needs a shared sense of what quality looks like. That judgment can come from formal expertise, such as a doctor for a healthcare product or a lawyer for a legal product. It can also come from informal expertise already present in the organization. He stresses that the relevant person needs experience with the actual use case. Being a doctor alone may not be enough for a medical coding product if the person has never worked with medical coding.
The oracle, evaluator, and architect divide the work in different ways
In the oracle model, the domain expert assesses the current output and improves it directly. They may tweak prompts or add documents and tools. In the evaluator model, the expert defines what should be measured, builds a way to collect the data, and works with engineers on changes. Reviews by clinicians, customer signals, and an LLM judge are possible inputs. In the architect model, the expert designs a system that performs assessment and improvement with less human intervention, using mechanisms that learn from user interactions.
The choice of role depends on taste, measurement, and iteration speed
The first decision is whether performance can be captured in meaningful metrics. If it cannot, and taste matters more, Lovejoy recommends an oracle approach. One person may be enough at a small scale, while larger products may need several experts for different output segments. If quality can be measured, the next question is whether engineers can iterate quickly enough through manual changes. If they can, an evaluator may be sufficient. If manual iteration is too slow and improvement can be automated, the organization needs an architect. Teams often start with an oracle and evolve as the product and its scale change.
Granola uses a writer as an oracle because meeting-note quality involves taste
Granola's first employee, Joe, had a writing and journalism background. She wrote the prompts, studied research, and spoke with many users to understand what makes a good meeting note. She remains the main gatekeeper of AI quality, reviewing the notes produced by the product and improving them directly. Lovejoy says this works because there is no objectively perfect meeting note. Quality involves human taste. Meeting notes are also the product's main output, so direct human review and improvement can remain workable even as the company scales, alongside evaluations and internal tools that let others contribute to prompts.
Tandem scales an oracle through doctors who own local variations
Tandem's medical AI scribe began with Roy, a medical doctor who had also worked at McKinsey. He reviewed medical notes and updated prompts. As the product expanded across specialties, countries, note types, and use cases, one person could no longer manage the work. Tandem moved to a decentralized oracle model, hiring doctors to handle subsets of the output and updating its platform to support many prompt customizations. Each expert could work with customers in a particular specialty or location, understand their needs, and keep the relevant prompt version up to date.
Anterior moved from direct review to measurement and automated adaptation
At Anterior, Lovejoy initially built the product, reviewed outputs as a doctor, and updated prompts and code. As demand and variation grew, he defined metrics and failure modes, built a review dashboard, and hired clinicians to assess selected outputs. The resulting information let the clinical and engineering teams work together on improvements. Manual fixing still became too slow because organizations interpreted policies and rules differently. Lovejoy therefore moved toward an architect role, designing methods for the system to learn from usage at the edge. The output was measurable because a prior authorization decision could be correct or incorrect based on medical evidence.
A principal domain expert needs ownership and room to grow
Lovejoy recommends one person who is accountable for AI quality and can make decisions. This avoids a committee where everyone shares responsibility and no one has final authority. The person should be in product decisions, rather than acting only as a consultant who reviews outputs. He describes a company with two senior clinicians who had unclear authority and limited ownership. Progress on the broader quality system was slow, and both later left, taking context with them. He recommends hiring for breadth around a strong base of direct domain experience, then pairing the expert with specialists when needed.
"The system for incorporating domain insights is more important than the sophistication of your models or your pipelines."01:32
Who should watch
You are building a vertical AI product and need to decide whether a domain expert should edit the product, define its evaluations, or design an improvement system.
Your team has domain specialists who review outputs but lack clear ownership, authority, or a path into product and engineering decisions.
You are moving from a small product with one expert to many customers, specialties, countries, or policy variations.
