Changing an agent's harness can alter benchmark results by more than 30%, so evaluations need to match the system and environment where the agent will run.
2
Public benchmarks help shortlist configurations, but teams need realistic internal tasks, repeated runs, full traces, and measurements tied to their actual goals.
3
An agent can flag a problem without stopping work, and its recorded trace may be unreliable, so quality gates, audit systems, and human escalation need independent enforcement.
Summary
Stephanie Jarmak argues that agent evaluations are easy to misread because small changes in prompts, harnesses, tools, and measurements can change the result. She describes a coding-agent test where a prompt change reduced cost by 28% but did not make the agent use the intended tool. In another example, changing only the harness produced a difference of more than 30% between model configurations. Public benchmarks are useful for shortlisting, but decisions should rely on tasks that resemble real work in the target environment. Jarmak recommends treating evaluation like an experiment: vary known factors, run non-deterministic systems repeatedly, retain complete traces, and inspect the whole information pipeline. The measurement must match the business goal. She also explains why agent warnings need enforcement in code, why human review can become a bottleneck, and why agent records may need independent verification. Production remains the final test, with shadow deployments and observability used to manage risk.
Jarmak changed the initial prompt to encourage a coding agent to use a code search MCP tool. The run became 28% cheaper, but the agent still did not use the tool. That cost change was worth investigating as experimental variation, yet it did not answer the question she cared about. The system appeared to receive information that was not actually present because the tool had not been called. Evaluation results only make sense when the measured change matches the decision the team is trying to make.
Using a third-party comparison with about 63 Terminal-Bench tasks, Jarmak describes how model rankings changed when the harness changed. One configuration made Opus 5 perform better with OpenHands, while another harness made Astra perform better. The difference in effectiveness was more than 30%. Her conclusion is that teams should evaluate the closest possible version of the model, harness, tools, and environment they will actually deploy. A public benchmark result may not predict behavior in that setting.
Jarmak defines an evaluation as a measurement used to make a decision about a product, process, or change. Public benchmarks provide a shared reference and help teams create a shortlist, but they should not automatically decide what an organization adopts. A law firm, for example, should not treat coding-agent benchmark results as directly relevant unless it plans to use coding agents. Teams can build internal benchmark sets that reflect their workloads and improve their configurations through repeated iteration.
Once a team has shortlisted configurations, Jarmak recommends identifying the variables that may affect the result. Tasks should cover real workload families, since an agent may be better at code understanding than bug fixing. Systems are non-deterministic, so teams need to decide how many runs to perform and account for interactions between model, harness, tools, and other changes. Retaining traces and observability data lets the team separate random variation from a real improvement.
When an agent fails to use a tool, the investigation should begin by checking whether the tool was available in the context. The next questions are whether the agent called it and what values it returned. The same inspection should continue through the rest of the pipeline. Even after a tool is improved and used correctly, the final task outcome may not change. Failed attempts and non-use are evidence about the system and should remain visible during debugging.
Jarmak compares search behavior using precision and recall. In one example, an agent returned 295 files when six were needed. A team that only values finding every possibly relevant file might accept recall of one, while her use case also valued avoiding irrelevant results. A single benchmark score hides this choice. Teams need to understand what a validator measures and why, then choose a metric that reflects the business cost of extra, missing, or incorrect information.
In Jarmak's software factory, agents can identify that something is going wrong, but the first setup had no mechanism that forced work to stop. She separates detecting or flagging a problem from enforcing a gate in code. The system must also define how the case reaches a human. This distinction matters when teams automate more work while still treating agent output as untrusted.
Jarmak describes a system with a 1% false alarm rate. If every alarm went to a human, that could create 10,000 reviews. Teams therefore need to estimate whether domain experts can handle the resulting queue, adjust alarm behavior, and consider the downstream consequences of missed or unnecessary reviews. The right escalation policy depends on the consequences of the errors and the review capacity available.
Jarmak discusses research in which security issues were seeded into an agent's input. A hash check on the agent's own record suggested that everything passed, and the injected issues were not detected. An independent comparison of the full traffic history, model changes, and related records detected them. For systems with meaningful security requirements, teams need to ask whether the retained trace can be trusted and whether an independent record is necessary.
"You want to be actually evaluating the closest configuration that you can that's going to reflect the work and the environment that you're going to be measuring."03:31
Who should watch
You are comparing agent models or harnesses and need results that predict behavior in your own tools and workflows.
Your team relies on public benchmarks but has not built an evaluation set from real production tasks.
You are adding automated approval or escalation and need to control false alarms, review volume, and the trustworthiness of agent traces.
