AI-generated code is only useful when the product requirements, system design, and evaluation criteria describe the right problem.
2
A claims review system should start with business constraints, data sources, retrieval choices, and a simple controlled workflow with human review.
3
Production systems need offline evaluation and live monitoring for guardrails, response quality, cost, latency, reliability, and user overrides.
Summary
Apoorva Joshi presents a repeatable framework for designing AI systems from an initial business problem through production. She applies it to a health insurance claims review system. The process starts by defining the users, current pain, constraints, performance needs, AI's role, and measurable success metrics. It then covers data sources, update schedules, preprocessing, retrieval methods, and a simple architecture. For the example, the design combines retrieval-augmented generation, a controlled workflow, and human review for complex cases or denials. Joshi then explains guardrails for invalid inputs and unsupported outputs, along with metrics for faithfulness, missing citations, processing time, cost, and system health. After launch, teams should monitor regressions and signals such as human overrides and review time. She is direct about the risk of letting coding agents choose an over-engineered architecture before the team knows what is failing.
Product requirements should describe the problem before they describe the solution
Joshi says wipe coding is fine when the stakes are low and the builder can easily judge the output. Real systems need a clear specification because other people depend on them and mistakes can have consequences. The product requirements phase asks what is being built, who will use it, and what constraints apply. A business problem should state the users, current situation, and measurable pain without prescribing an agent, a multi-agent system, or another architecture. Her example identifies medical reviewers, a two-day average processing time, and delays to patient care.
Business and performance constraints shape the design
Before designing the claims system, the team must identify rules that could restrict its implementation. Patient data must remain in the approved cloud environment, models must be available there, complex cases require senior physician review, and denial decisions need a human reviewer. Joshi also asks about response-time requirements, monthly LLM inference spend, and uptime SLAs. These details determine which models and vendors are possible. They also establish where full automation is unacceptable. In this example, AI is complementary because claims review already exists, reactive because a submitted claim triggers it, and semi-autonomous at most.
A success metric connects the AI system to the business problem
Joshi recommends one or two specific, measurable, achievable, relevant, and time-bound success metrics. For the claims application, she proposes reducing urgent claim processing time from two days to one hour within 90 days of launch. This metric gives the team a baseline, a target, and a deadline. It also keeps the project tied to the original user problem instead of allowing the team to optimize an isolated model score. The metric describes the result the business needs, while leaving the implementation open.
Data preparation must account for both content and freshness
The claims system needs clinical guidelines, internal coverage policies, and patient claims history. The guidelines and policies are assumed to be PDFs in Confluence, while claims are stored in MongoDB as they are processed. Each source has a different update schedule. Clinical guidelines may change annually, coverage policies quarterly, and patient history whenever a new claim arrives. The urgent-claim target also suggests hourly updates for relevant information. Long documents should be chunked, embedded, and enriched with metadata such as procedure names and publication dates. Patient history is already structured, but personally identifiable information should be removed before sending it to an LLM.
Clinical guidelines and coverage policies are suitable for vector search, but medical diagnosis and procedure codes may not be retrieved well by semantic similarity alone. Joshi recommends metadata pre-filtering or hybrid search, which combines vector and keyword retrieval. Patient claims history has a different access pattern. The system can retrieve it with an exact match on a patient name or identifier. The retrieval choice follows from the data-processing decision, rather than being selected as a generic feature of the AI stack.
The first architecture should be simple, controlled, and evaluated
Joshi warns against jumping straight to an agent because of hype or allowing a coding agent to decide the architecture. The claims workflow receives a claim and clinical notes, retrieves relevant guidelines, policies, and patient history, and passes that context to an LLM for an approval or rejection recommendation. Complex cases go to a senior physician, and denials go to a medical reviewer. The final decision and its reasoning are logged to MongoDB. She classifies the design as retrieval-augmented generation, a controlled workflow, and human-in-the-loop. The team should evaluate this simple design, find its failures, and then iterate.
Guardrails and evaluation measure different kinds of failure
Guardrails define acceptable behavior for a probabilistic LLM system. Input checks should reject invalid, irrelevant, or harmful requests, such as asking a claims application to write a poem. Output checks can reject a decision that lacks citations for the guidelines or policies that informed it. Evaluation also needs response-quality measures, such as faithfulness to retrieved information, plus an application metric such as claim processing time and system-health measures such as recommendation cost. Joshi suggests tracking claim rejection rates and missing-citation rates so teams can investigate changes instead of guessing.
Production optimization includes cost, latency, and reliability
A prototype with acceptable accuracy is not automatically ready for production. Joshi says teams may need more testing and iteration around cost, latency, and reliability. Accuracy work focuses on the information in the LLM context, with prompt engineering and reranking as possible techniques for the claims system. Semantic caching and batch processing can reduce cost or speed up repeated work. Structured outputs can improve reliability by requiring every response to contain both a decision and citations. After launch, monitoring should include the offline metrics plus signals such as how often reviewers override the AI verdict and how long they take to review it.
"The product spec is the hard part now, it's not the code anymore."27:11
Who should watch
You are building an AI feature for a real business process and need a way to define the problem before choosing an agent or model.
Your system handles regulated, sensitive, or high-stakes data and needs explicit human review and data constraints.
You have a working prototype but need to plan evaluation, monitoring, cost controls, and production reliability.
