AI applications keep changing, but many evaluation systems still treat them like fixed software tested against fixed datasets.
2
Agents can use traces, rubrics, and telemetry to update test suites and correct problems as user behavior and application behavior change.
3
An evaluation should define the outcome an agent must reach, then let the system adapt how it gets there.
Summary
Vincent Koc argues that static benchmarks are a poor fit for AI agents that change through user intent, tools, context, and self-modifying harnesses. Traditional practice starts with handcrafted examples, offline tests, and regression suites, but this leaves a gap where teams do not learn how systems behave under unusual or changing conditions. Koc calls the resulting problem eval calcification. He proposes evaluations that use traces to find new cases, run continuously in production, and feed telemetry about failures and cost back into the harness. The evaluation can then change as the customer base and agent change. Koc also argues that tests should describe an intended end state rather than prescribe one correct answer, since different users may need different interactions. His central warning is that the less predictable 20 percent of agent behavior is where business failures will appear, so evals need to behave more like living software than frozen datasets.
AI evaluation copied static software testing and left out deliberate breakage
Koc contrasts ordinary software testing with current AI evaluation. Software teams use examples, unit tests, manual regression suites, CI/CD, chaos engineering, and observability. AI teams often rely on static benchmarks, handcrafted questions, and offline evaluation before deployment. That process checks whether an application gives the expected answer to known cases, but it does not explore what happens when the system is pushed into unfamiliar situations. Koc describes chaos engineering as breaking technology in random ways to find its limits. He says AI evaluation has largely missed that practice, leaving teams with large datasets that explain little once an agent behaves unexpectedly.
AI applications are adaptive while their benchmarks remain fixed
The underlying mismatch is that AI applications are not static, even though teams evaluate them as if they were. Koc points to OpenClaw, where the harness can change as users create skills and perform other tasks. Software can also be shipped and modified at high speed, which makes a benchmark that was prepared once less useful over time. He cites research on adaptive testing for LLM evaluations and asks why benchmarks should remain static when the applications they measure keep changing. Selective testing is an early move in this direction, but Koc presents the larger shift as making the evaluation itself responsive.
Agent development moved from prompt tweaking toward intent-driven systems
Koc describes a progression from prompt engineering to context engineering and then to intent engineering. Prompt engineering involved repeatedly changing words in an instruction and hoping the model improved. Context engineering added structure through retrieved data, tool calling, and systems such as RAG. Breaking a larger agent into components made it possible to check whether a particular tool or MCP component did what it was meant to do. The newer systems try to understand a user's intent and adapt the experience. That makes evaluation harder because two users may receive different interactions while both systems are behaving appropriately.
Intent-based agents require evaluations that allow ambiguity and variation
For a fixed question such as 1 + 1, a test can compare the answer with a known target. Agent behavior is less constrained. Koc asks how teams should define ambiguity, personality, and organizational expectations when an agent can respond differently to different people. He points to rubrics as one way to judge behavior without requiring one exact answer. The evaluation can describe what good performance looks like while allowing the agent to choose its path. This gives teams a way to measure intentful behavior without pretending that every interaction has a single canonical response.
Traces can turn changing user behavior into new evaluation cases
Koc proposes that agents should self-curate test suites from application traces. He gives a rough pattern in which 80 percent of an agent's activity remains stable while the customer base changes the other 20 percent. New customers may ask different questions or use the agent in unfamiliar ways. Those traces contain evidence that the application has shifted, yet conventional evaluation may ignore them. A system could feed the traces into an agent, detect what has changed, inform the owners, and update the tests. The suite would grow from real usage instead of remaining tied to the examples chosen before deployment.
Telemetry can put evaluation and correction inside the running harness
The proposed evaluation loop stays active rather than running only before release. Agents can evaluate behavior continuously, while telemetry tells the harness what is failing and how much the system costs. Koc describes a paper about telemetry in the loop, where an agentic application or MCP system uses operational data to set conditions and correct itself. He gives the example of a harness encountering an error, fixing the issue, and continuing. The point is to use observed failures as input to the system instead of trying to predict every failure in advance. Koc presents this as a form of self-correction rather than a replacement for visibility.
An evaluation should specify the destination and let the agent find the route
Koc compares this approach with auto research, where a program receives a goal, a target, and a reward signal, then tunes itself. The same pattern could apply to an agent: define the user's intended outcome and let the machine optimize toward it. In that model, the evaluation is not the dataset or the initial set of test cases. It is the end state the system should reach. The agent performs the work while the evaluation checks whether the intended result has been achieved. This also makes room for behavior that cannot be reduced to one fixed answer.
The changing minority of agent behavior is where business failures will appear
Koc says roughly 80 percent of an agent's behavior may be stable and defined, while the remaining 20 percent keeps changing. That changing portion includes unusual questions and strange uses of the agent. He expects those cases to create serious operational problems if teams do not watch and adapt to them. His recommendation is to treat evaluations as code, software, or a living agent rather than as a point-in-time dataset. The evaluation should monitor the changing part of the system and update itself as the application, users, and data change.
"Our evals don't become the data set or the starting point, our evals become like what is the end state that we want to get to and then we just let the machines do the work."13:02
Who should watch
You maintain an AI agent whose users are changing faster than its test set, and production surprises keep creating new failure cases.
Your evaluation process ends after offline tests and does not use traces, operational telemetry, or real user behavior.
You are designing a harness that can adapt to intent and need a way to judge outcomes without requiring one exact response.
