Production traces can be reconstructed into repeatable simulation tasks with the same data, tools, files, and environment state.
2
A private benchmark measures the full agent system on cost, latency, retries, policies, and task success, rather than only model pass rate.
3
Benchmarks should run in CI, expand from observability traces, and gate agent releases as the system changes.
Summary
Rustem Feyzkhanov argues that teams need private benchmarks built from their own production traces. A trace becomes a replayable task when the team reconstructs the database state, APIs, tools, files, policies, and workflow the agent encountered. Different agent configurations can then run against the same environment and be compared on success, cost, latency, and retries. A task includes an environment, instructions, an Oracle solution that proves the task is solvable, and verifiers that inspect the final state, trace, and artifacts. Long tasks can use intermediate steps and early stopping. Feyzkhanov is direct about failure modes: agents can reward hack simulations, verifiers can be too broad or incorrect, fixtures can be missing, and success rates can vary. He recommends treating benchmarks as software with their own CI pipeline, then connecting observability, experiments, and release gates so failures add new tasks and every agent change is tested.
Offline simulations make production traces repeatable
Traces show the input prompt, the actions an agent took, and its output. They help teams find production failures, but rerunning variants against production does not produce a clean comparison because the database state and tool versions can change. Feyzkhanov's approach reconstructs a trace as a task and runs it offline with different agent configurations. The environment and evaluators stay the same, while the team compares success rate, cost, latency, and retries. This tests the complete agent system, including prompts, thinking settings, skills, and tools, instead of comparing models in isolation.
A private benchmark measures the system a company actually ships
Public benchmarks such as SweepBench, TerminalBench, and Co-Bench focus on particular domains or interaction types. A company's benchmark needs its own use cases, tools, policies, and production environment. Public results help establish a prior about model capability, but they do not answer whether an agent follows company rules or solves work at an acceptable cost and latency. The benchmark can support release checks, model selection, trace debugging, regression detection, and optimization. Feyzkhanov also describes using simulation data for error training, including an example where a small planning model was fine-tuned to match a larger planning model on specific tasks.
Each task needs an environment and an Oracle solution
A simulation task starts with an input prompt. The agent interacts with APIs, MCP tools, databases, and files, then produces an output, trace, final environment state, and artifacts. Verifiers turn those results into metrics. The Oracle solution runs through the same sequence using a known solution instead of the real agent. Its purpose is to prove that the task can be solved at all. Feyzkhanov describes a Harbor-style file layout with an instruction.md file, an environment defined by a Dockerfile or Docker Compose setup, hidden Oracle and verifier files, and metadata. This structure makes experiments repeatable.
The simulated environment should resemble production without running production
The environment is a small version of production. Its database, API services, tools, and files should match what the agent normally uses, while avoiding the cost and risk of running the full production system for every experiment. A database snapshot, side containers, mocked APIs, and MCP tools can provide the required dependencies. Real users cannot be placed in a simulation, so a simulated user can be implemented as an LLM with a prompt and extra context that guide its behavior. The agent should not be able to tell that it is operating inside a simulation.
Long tasks need intermediate checks and early stopping
Some agents work through tasks that span hours or contain many dependent actions. Feyzkhanov recommends splitting these simulations into intermediate steps. Each step can have its own prompt and verifiers, allowing the run to stop early when the agent has clearly failed. This avoids spending the rest of a long run after the task has already gone off track. The environment records more than the final text: it can include database changes, API responses, user replies, files, and other artifacts that later verifiers inspect.
Verification combines deterministic checks, judges, and targeted human review
A verifier can inspect the final environment state, the trace, and the artifacts. Deterministic checks work well for exact outputs and tool calls. An LLM, harness, or agent can judge less mechanical properties such as trace quality and planning. Subject matter experts should review selected cases rather than every run, especially when different verifiers disagree about whether the agent succeeded or whether its trace was good enough. This layered approach lets teams check both what the agent produced and how it reached the result.
Benchmark tasks can fail in ways that make evaluation misleading
Feyzkhanov lists several benchmark defects. An agent may recognize that it is in a simulation and reward hack the environment. A task may be too easy, or its verifiers may be so broad that incorrect behavior still passes. The opposite can also happen when verifiers are wrong and every agent fails. Agent behavior may vary substantially across runs. These problems need to be caught during benchmark development. The benchmark itself is software made of code and files, so it needs engineering checks, pinned dependencies, correct base images, complete fixtures, and runs that establish both solvability and difficulty.
Observability and experiments form two connected loops around the benchmark
After deployment, one loop expands the benchmark by turning failures from observability traces into new tasks. A second loop runs experiments on the expanded benchmark with a new agent configuration, records the results, and uses them as a release gate. Teams can establish a baseline, change one part of the agent, rerun the evaluation, and then repeat the full experiment before release. Feyzkhanov warns against putting every fix into the prompt. A missing procedure may belong in a skill, context overload may require changing task hardness, and a required schema may belong in structured output.
"Basically, public benchmark is useful to orient and build your prior, but your private benchmark is useful to ship."03:28
Who should watch
You are preparing to release an agent and need to compare prompts, models, tools, or skills against a stable environment.
Your public benchmark scores look good, but you still need to measure policy compliance, cost, latency, retries, and behavior on company-specific work.
Production traces reveal failures, yet your team has no repeatable way to turn those failures into regression tests.
