An eval set assembled before launch can tell you whether a change helped on the cases you expected. It cannot tell you what frustrated users will try next, whether a judge agrees with the people doing the work, or what happened after an agent called a tool. A production eval loop starts by defining the actual range of user requests and locating failures inside the application. Traces and human explanations become regression cases; calibrated judges and simulations make more of them repeatable. Evaluation then stays connected to the running system, where changes to users, tools and agent behavior keep supplying new evidence. The suite survives only if the team continues to collect failures, test its graders and revise what good means after launch.
Why first: Pesok establishes what an application eval has to cover: not one expected answer, but the range of requests real users can make. Dhinakaran follows those requests into the application and shows why one overall score cannot locate a bad router decision.
Why second: Pesok maps the requests an application must survive. Dhinakaran maps the system they travel through, from the router down to spans and components, so an engineer can tell what failed. Goyal then asks how that evaluation work stays useful as the model and application change.
Why here: Dhinakaran shows where to attach evals. Goyal supplies the maintenance discipline around them: keep the data current, treat scoring as the product specification, and make model comparisons cheap. Zapier shows where the next cases come from once users get hold of the agent.
Why here: Goyal describes the evaluation machinery. Willinski and Balocco feed it with failed runs, tool traces and signs of user frustration. They also find that narrow unit evals can punish a better model for taking a different route. Hetzel follows with a practical path from those observations to tested graders.
Why here: Zapier has a flywheel collecting failures. Hetzel explains how a team grows into it: preserve why reviewers passed or failed a run, derive narrow checks, and recreate the external state behind tool calls. Lyft then tests the synthetic users and judges that decide whether an agent is ready to launch.
Why here: Hetzel turns expert explanations into graders. Ung and Sharma show how both sides can lie: a helpful simulator makes the agent look better than it is, while an uncalibrated judge produces a score nobody should ship on. Mabrouk makes that judge the subject of the next experiment.
Why here: Lyft validates a binary judge against held-out human labels. Mabrouk pushes further, showing that prompt optimization can improve measured agreement without making the evaluator trustworthy. Feyzkhanov widens the target from one judge to the complete environment in which an agent run can be repeated.
Why here: Mabrouk makes one evaluator auditable. Feyzkhanov makes the full run repeatable with a task, an environment, an oracle solution and layered verification. That lets a company test the private system it actually ships. Gupta follows by moving the same workflow-level evidence into production operations.
Why here: Feyzkhanov reconstructs a production case offline. Gupta closes the loop online, measuring task outcomes, tool behavior, escalation and recovery as the agent runs. Koc ends with the problem that remains: users and agents keep changing faster than a fixed suite can follow.
Why last: Gupta puts continuous evaluation around a production agent. Koc asks what happens when that agent, its tools and its users change. His answer is to draw new cases from traces while judging the outcome instead of prescribing one acceptable route. The pack ends with an eval suite that has to change because the product does.