LLM applications need evaluations because their outputs are nondeterministic, depend on business data, and can take multiple paths through an agent system.
2
Product managers can replace one-off vibe checks with a loop of production examples, datasets, experiments, model- or code-based evaluators, and human review.
3
An evaluation should become part of the product requirement, with human labels used to check whether an LLM judge is actually measuring the intended quality.
Summary
Aman Khan presents evaluations as the testing discipline for LLM products. Traditional software tests rely on deterministic behavior, while agents can hallucinate, choose different paths, and depend on enterprise data. He demonstrates an AI trip planner with multiple agents, then traces its calls, edits its itinerary prompt, and compares prompt versions across a dataset. LLM judges can label properties such as friendliness, discounts, correctness, and hallucination, but Khan does not treat their output as trustworthy by default. Human annotations and code checks are needed to measure whether the judge agrees with the intended standard. His suggested workflow starts with a small development dataset, adds difficult examples as they appear, and repeats the process on production data. For PMs, evaluations can become acceptance criteria that give engineers concrete requirements and give product teams a way to inspect the whole system.
AI product managers face a higher bar for specifying AI systems
Khan says product management expectations have changed as AI systems have become part of the product. A PM working with an AI engineer may need to specify how an agent system should behave, rather than hand over only a conventional requirements document. He describes moving from trying AI tools in his own work to using prototypes as higher-resolution inputs for engineering. The difficult point arrives when the team must move an LLM or agent from a prototype into production. Khan calls this a confidence slump caused by limited tooling and education around reliable AI systems. The role still owns the end product experience, including whether the system behaves acceptably.
Agent evaluations differ from ordinary software tests
Khan compares an eval with software testing, then gives the differences. Conventional software is deterministic, while an LLM agent can be persuaded that "1 plus 1 equals 3." An agent can also take several possible paths, so a unit test cannot describe every behavior. The evaluation has to account for the data the agent uses, especially when the data is part of the product's reason for existing. Khan also says teams may not want to eliminate every unusual model behavior. They need to decide which behavior is acceptable for the use case and test that standard. This makes agent evaluation dependent on the system architecture, context, and business data.
An LLM judge needs a role, task, context, goal, and labels
Khan describes an LLM-as-a-judge evaluation as a prompt with several parts. It sets the judge's role, states the task, supplies the text or other context to inspect, defines the goal, and provides terminology and output labels. His toxicity example asks the judge to classify text as toxic or not toxic, but the same structure can evaluate a business-specific property. He warns against asking a model to return a numerical score directly. In his view, LLMs remain unreliable with numbers because of how tokens are represented. A better pattern is to ask for a text label, then map that label to a score when a scoring system needs one.
Data turns vibe coding into a repeatable development process
Khan reframes vibe coding as a useful prototyping activity that becomes unsafe when a team uses one attractive output to decide that a product is ready. He calls the alternative "thrive coding": building in the same fast way while using data to gain confidence in the result. In the trip-planner demo, a single itinerary looked acceptable at first, but a revised prompt still failed to stay short and sounded overly generated. It did include a discount request, which showed why a single example can create a misleading impression. Khan's point is that prompt and model decisions should be tested across many examples rather than judged from one row.
Tracing exposes how a multi-agent product actually works
The trip planner uses separate agents for budget, local experiences, and research, which then feed an itinerary agent. Khan shows traces as the input, output, and metadata around a request. The trace contains spans, which are units of work with timing and process type. In the example, spans include agents, tools, and LLM calls. A visual representation makes the parallel calls and their handoff into the itinerary agent easier to inspect than a set of function calls in code. Khan says this gives PMs a concrete way to ask what the agent does, where outputs go, and how the parts of the system connect.
Prompt experiments need datasets instead of single outputs
Khan builds a dataset from itinerary spans and uses it to compare two prompt versions. He describes a dataset as a collection of examples that can feel like a spreadsheet, where people can label outputs as good or bad. The example uses synthetic itineraries generated through the same server, so the data is only a demonstration. The experiment then runs both prompts across the collection rather than on one itinerary. One prompt limits the output length and gives a specific discount instruction. The comparison reveals both output speed and behavior changes, including a large latency difference when one prompt leaves the response length unspecified.
Khan creates one judge for friendly versus robotic tone and another for whether the output offers a discount. The judge can apply a label over a large dataset, which is more practical than asking people to inspect every row. He still uses a "trust but verify" approach. The platform shows the judge's explanation, and Khan questions whether its friendliness assessment matches his own view. Human subject-matter experts then label examples in the same dataset. A code evaluator compares the human label with the judge label. When the two disagree, the team has evidence that the evaluation prompt itself needs work. Khan says evaluators need evaluation too.
Evaluation datasets grow through difficult production examples
Khan recommends starting with a small development dataset, running experiments, and adding examples until the team is comfortable shipping. Once the system is in production, new data provides more cases to review and return to development. His self-driving example moves from straight roads to left turns, then to left turns with a pedestrian on the sidewalk. Each new failure creates a more specific dataset. He says teams will not know every difficult scenario in advance. The required bar also depends on the product. Healthcare and legal systems may need a higher standard than a travel agent. Human review of disagreements can add difficult or borderline examples over time.
Evaluations can become product acceptance criteria
Khan suggests that PMs treat an evaluation as a new form of requirements document. Instead of handing engineering only a PRD, a PM could provide the evaluation dataset and the evaluator that define the acceptance criteria for the AI feature. This gives the team a shared way to test the end product. He also argues that PMs should become more technically literate because tools such as Cursor make it easier to inspect code and build prototypes. PMs still own the user-facing result, while engineers may own production quality and infrastructure. The boundary can change as building and testing become faster.
"When the people that are selling you the product are telling you that it's not reliable you should probably listen to them."06:46
Who should watch
You are a PM responsible for an LLM or agent feature and need a way to replace subjective demo reviews with evidence from representative examples.
Your team already has traces or production data, but you need a process for turning them into datasets, experiments, and acceptance criteria.
You are considering an LLM-as-a-judge evaluator and want to check it against human labels before relying on its scores.
