Benchmark scores are approximations, so engineers should compare them with real use rather than treating them as facts.
2
A useful eval runs an agent in an isolated environment, records its traces, and turns failures into specific engineering changes.
3
Improvement has three zones: fixing obvious bugs, adapting the harness to each model family, and avoiding benchmark overfitting.
Summary
Ara Khan argues that evals fail when engineers treat benchmark scores as objective truth or replace measurement with personal preference. Scores can still guide engineering if teams use recent, precise tests and wait for new models to settle before switching. Khan describes how Cline built and used coding-agent evals with Terminal Bench, isolated environments, parallel runs, and detailed traces. After each run, another agent can inspect failures and group them by cause. The resulting changes may involve container CPU and memory, timeouts, prompts, retry behavior, or other harness details. Khan separates improvements into obvious bug fixes, model-specific adaptations, and benchmark overfitting. Cline began at 43% on Terminal Bench and improved through infrastructure settings, timeout changes, and prompt techniques, without simply changing to a better model. His conclusion is practical: build or choose a relevant benchmark, hill climb on it, and check the score against the experience of using the agent.
Benchmark scores should be treated as approximations
Khan criticizes the habit of reading a model leaderboard as a direct statement about real capability. Similar scores do not mean models behave the same in practice. He describes one group that trusts objective metrics too much and another that relies only on taste, vibes, and anthropomorphizing models. His position is between those extremes. Evals are neither the final authority nor useless. Engineers should use them as evidence, then compare that evidence with actual work. He also advises waiting a couple of weeks after a new model appears before changing systems, because model preferences and rankings change quickly.
Old general benchmarks can miss the work engineers care about
Khan recommends looking for evals that are both new and precise for the task at hand. He points to OpenAI's statement that SWE-bench Verified no longer measures frontier coding capabilities. Simple tasks such as solving Fibonacci or doing matrix multiplication may be valid tests, but they do not capture the problems encountered in real software engineering. A useful benchmark needs to approximate the actual work. That requires judgment because a coding agent can encounter many different failure paths, and even coding tasks have a very large search space.
Agent evals must grade the whole process, not only the final answer
A single-turn language-model test can often use a limited answer space and a binary grade. Coding agents are different. A user may ask an agent to repair an MCP server, and the agent may read files, search documentation, install an environment, run scripts, and execute tests. The evaluator must determine whether the task worked, whether it broke something else, and how the agent reached the result. Khan says this makes agent evals harder to build. The grade has to cover a long interaction with tools and an environment rather than one short response.
Terminal Bench makes real coding tasks runnable in isolated environments
Khan describes Terminal Bench, created by researchers at Stanford, as a collection of 89 coding problems that approximate real programming work. The tasks include race conditions, database issues, and infrastructure problems, and can take an agent 30 to 40 minutes. An eval run gives each task an isolated environment, such as a virtual machine, containing the repository and setup. The chosen coding agent is installed inside that environment. Harbor helps standardize the machines and run tasks in parallel, so the total run is limited by the slowest task rather than the sum of all task setup and execution times.
Failure traces turn an eval score into an engineering plan
After a run, Khan recommends reviewing every failed task rather than stopping at the aggregate score. A trace contains the agent's language-model calls and other actions. Another agent can inspect those traces and classify why each task failed, such as a missing pass condition or a broken retry tool. This process lets the team portfolio allocate failures and identify small changes that may move the score. The eval becomes a way to locate defects in the system. The useful output is not just a number. It is a list of concrete levers to test in the harness, prompts, environment, and tools.
A score measures the model, the harness, and the task set
Khan says a coding-agent eval tests three parts of the system. It tests the model, since a very capable model can sometimes compensate for a poor agent. It tests the harness, including whether the coding environment makes good use of the model. Anthropic models may work better with Claude Code than with another coding agent because the harness fits that model family more closely. It also tests the problems themselves. A perfect score on tasks that do not resemble users' work does not say much about the product. The task set must therefore be credible as well as repeatable.
Cline's improvement came from system changes rather than a model swap
Cline started at 43% on Terminal Bench. Khan says the team improved its result by changing container CPU and memory settings, raising timeouts, and adjusting the agent's thinking behavior. Asking a model to think more can sometimes hurt because it may loop for a long time instead of improving the answer. The team also tried prompt techniques that fit Anthropic model families. Khan stresses that those techniques do not necessarily transfer to Codex or Gemini model families. The lesson is that a model's public reputation does not guarantee that it will work well inside a particular harness.
Teams should improve through three zones without overfitting
Khan divides eval-driven improvement into three zones. Zone one contains obvious flaws, such as a harness bug that crashes a run or rate limiting, and those should be fixed. Zone two contains nuanced changes, including model-specific prompts, prompt size, and harness behavior. This is where teams explain why a model that is widely praised performs poorly in their setup. Zone three is overfitting, where a team changes the system to cheat a benchmark and then advertises the score. Khan tells the audience to avoid that zone. The practical loop is to find a relevant benchmark, hill climb on it, and also ask whether the product feels good to use.
"The truth is somewhere in the middle that the evals are not the end all be all. They're also not completely useless."03:02
Who should watch
You are choosing models from leaderboards and need a way to test whether they work in your own coding harness.
Your agent has a disappointing benchmark score, but you do not know whether the model, environment, tools, or prompts are responsible.
You are building an eval and want to improve results without quietly fitting the system to one benchmark.
