A replay agent can copy one successful action sequence per task and match or exceed the frontier model that produced those sequences on deterministic benchmarks.
2
Pass@k on a deterministic computer-use environment is formally equivalent to measuring the success rate of such a replay agent.
3
Trustworthy evaluation requires varied, verified environments and confidence intervals that account for both action randomness and environmental variation.
Summary
Pierluca D'Oro argues that computer-use benchmarks can reward agents that never understand or inspect the screen. His replay agent records a successful trajectory for every task, compiles those actions into a script smaller than one megabyte, and replays them blindly. On deterministic benchmarks, this script can match or beat the model from which it was extracted. He connects this exploit to pass@k, which becomes the replay agent's success rate in a deterministic environment. D'Oro proposes the PRISM principles for benchmark environments: variation across data, appearance, and starting state; verification of every generated configuration; sandboxing; privileged verifiers; and faithful reproduction of real systems. DIGIWORLD applies these ideas to mobile apps with millions of verified configurations. He also argues that evaluation metrics need uncertainty estimates that include both action and environment variation. Base-case rollouts can produce intervals with only about 20% coverage, causing costly model-selection errors.
A replay agent can perform computer-use tasks without seeing the screen
D'Oro defines a replay agent by running a frontier model on every benchmark task, saving each successful trajectory, and compiling the recorded taps, typing, and scrolling into a script. When a task arrives, the script blindly replays its stored action sequence. For a benchmark with hundreds of tasks, the resulting agent can be less than one megabyte. It is still a valid agent under standard benchmark evaluation, even though it has no perception or decision-making during execution.
Deterministic benchmarks let blind scripts match the models that generated them
On benchmarks such as OSWorld and MobileWorld, D'Oro says the replay agent's success rate is the same as, or better than, the frontier model from which its traces came. The reason is determinism. If the benchmark presents the same task in the same state each time, the stored sequence can be reused. D'Oro argues that a blind script beating the model that produced it is evidence of a benchmark problem, rather than evidence of strong computer-use ability.
Pass@k becomes a formal measure of replayability in deterministic environments
Pass@k is usually described as the probability that at least one of k attempts succeeds. D'Oro and his collaborators prove in the paper that, on a deterministic environment, this metric is exactly the success rate of the replay agent. A successful trajectory can simply be reused, so additional attempts do not test whether the agent can adapt to new conditions. In D'Oro's framing, pass@k turns the replay exploit into a metric.
PRISM requires variation, verification, sandboxing, privileged checks, and realism
D'Oro proposes the PRISM principles for building computer-use environments. Benchmarks should vary factors such as task data, visual appearance, and initial state, while checking that every generated combination is valid. They should run in sandboxes, provide privileged information for verification, and reproduce real systems faithfully. Variation prevents fixed action tapes from working everywhere. Verification prevents invalid tasks from entering the benchmark. Realism matters because the resulting score should say something about performance on the system being reproduced.
DIGIWORLD creates millions of verified mobile-app configurations
DIGIWORLD applies these design principles to 15 Android mobile apps across different domains. It contains 387 verified scenarios and 3.2 million configurations. A task can vary the amount of money sent, the recipient, the contacts or emails in a data profile, the app theme, and the starting screen. D'Oro says even a small number of choices across these factors produces millions of combinations, with the possibility of reaching billions as the benchmark scales.
A compiler-like system rejects broken task configurations
Generating a large amount of software does not automatically create a useful environment. DIGIWORLD starts with a parameterized task template, a matching verifier, mock data, and a base UI state. Its compiler-like system combines these inputs into configurations and rejects combinations that are invalid. D'Oro describes the main work as software engineering that keeps the generated tasks both diverse and valid. This verification step allows the benchmark to scale without filling it with broken cases.
Variation exposes weak robustness in frontier models
Once the benchmark varies its configurations, replay agents lose most of their performance, apart from tasks that are naturally repeatable. The same infrastructure can test whether frontier models retain performance when the task starts on a different screen or when the app theme changes. D'Oro says most frontier models perform poorly on these variations. A model that looks strong on one task presentation may therefore have much weaker performance across valid versions of the same task.
Honest uncertainty must include both action and environment variation
D'Oro identifies two sources of variation in computer-use evaluation. A model can produce different action trajectories across runs, which is the source usually measured by repeated rollouts. A varied benchmark also changes the environment through different data, themes, and starting states. That second source matters because real systems do not present one fixed base case. His paper gives a method for computing confidence intervals that accounts for the benchmark's structure and captures both forms of variation.
Base-case rollouts can make 95% intervals cover the truth only about 20% of the time
A nominal 95% confidence interval should contain the model's true performance about 95% of the time. D'Oro says that when evaluation uses only rollouts on a single base case, observed coverage can fall to roughly 17% to 20%. The intervals can look small while missing the real performance. He gives a deployment example with one million tasks, a 4% actual difference between models, and mistakes costing about $12 to $20 each, which can add up to hundreds of thousands of dollars in a month. A wider, properly computed interval may instead say that more evaluation is needed.
"Coding agents can generate a lot of software, but a lot of software is not the same as an effective environment."07:31
Who should watch
You are building computer-use evaluations and need to know whether fixed task states are making scores easy to game.
You compare models for deployment and need confidence intervals that reflect changes in both model actions and task environments.
You maintain benchmark infrastructure and want a concrete pattern for generating and rejecting varied configurations.
