Evals provide test cases for AI applications whose outputs can change between runs.
2
Useful evals depend on the application, so RAG, code generation, and agents need different measurements.
3
Scaling evals requires suitable data, cached results, orchestration, parallelism, and a repeated measure-monitor-analyze-iterate process.
Summary
Muktesh Mishra argues that evaluations should be treated as a basic part of developing AI applications. They help teams test nondeterministic outputs, understand what changes when prompts change, measure progress, and support trust in customer-facing systems. He starts with data, recommending small synthetic datasets that are labeled, observed, and refined over time. One dataset is rarely enough because different application flows need different cases. The evaluation method must also match the system. RAG and question-answering systems may use accuracy, similarity, or usefulness, while code generation needs functional correctness against the codebase. Agent evaluation includes trajectories, multi-turn conversations, and tool calls. At scale, teams should cache intermediate results, run evaluations in parallel, aggregate results, and run them often. Mishra also describes a trade-off between human review and automation, with the right balance depending on the need for speed or fidelity.
Evals give teams a way to test nondeterministic AI applications
Mishra describes evals as test cases for measuring an AI application's output. They address a basic problem with large language models: the same input can produce different outputs. Teams also need to know what breaks when they change a prompt, which tools and metrics to use, and which models are suitable. Measurement connects the system to business goals, lets teams track whether it is improving, and supports trust and accountability when customers rely on its output.
Evaluation starts with small, labeled datasets that keep changing
Mishra recommends starting with synthetic data when building an evaluation set. Teams can generate artificial examples and compare application outputs against them. The dataset should grow through observation and continuous refinement. Labels should cover different aspects of the application and its flows. He warns that one dataset is rarely sufficient. Separate datasets may be needed for separate flows, applications, or goals, so the evaluation cases match what the system is meant to do.
Teams should define goals and evaluate every flow and path
The first step in evaluating everything is to define the system's goals and objectives. Mishra recommends structuring the application into modules, improving data handling, and testing outputs and paths across all flows. An application with several routes cannot be evaluated through a single path. The data should also be organized around those flows, since different parts of the system may need different cases and different measurements.
An evaluation must match the kind of application being tested
Mishra rejects the idea of one universal eval. A typical RAG application can be judged with measures such as accuracy, similarity, or usefulness. Code generation needs different checks, including functional correctness against the actual codebase and the robustness of the generated code. Agents require trajectory evaluation because they can take different routes through a task. Their multi-turn conversations and tool calls also need separate checks for correctness and generated data.
Scaling evals depends on execution and repetition as well as metrics
To run evals at scale, Mishra recommends caching intermediate results, using regression checks, and paying attention to orchestration and parallelism. Teams need to decide how evaluations run, aggregate their results, and execute them frequently. He summarizes the operating loop as measure, monitor, analyze, and repeat. The point is to use the results to improve the system rather than treat evaluation as a one-time test.
Human review and automation should be balanced around the use case
There is no fixed evaluation strategy. Some situations need humans in the loop, while others can use automated tests. Mishra frames the choice as a trade-off between speed and fidelity. Automation can run quickly, but human judgment may be needed for subjective outputs. He also advises relying on a defined process rather than expecting tools to automate everything. Teams must decide how evaluations will be run and where human involvement belongs.
Eval development applies test-driven habits to AI systems
Mishra introduces the term eval development, drawing a comparison with test-driven development in conventional software. Teams should define evals from their use cases and include both positive and negative cases. His closing process is continuous: measure, monitor, analyze, and iterate. The data remains central, and teams should choose a balance between fidelity and speed instead of applying the same evaluation setup everywhere.
"Evals is the fundamental approach where you are writing sort of test cases to measure your AI applications."02:11
Who should watch
You are building a RAG chatbot, code generation system, or agent and need to decide what to test.
Your prompt changes keep producing unexpected regressions, and you need a repeatable evaluation process.
You are choosing between automated tests and human review, with different needs for speed and fidelity.
