Long horizon is a relative measure, so human task time, model tokens, steps, and tool calls should be considered together.
2
A task measures long-horizon ability only when early decisions change the later state of the environment.
3
Open-ended work needs judges that inspect the final environment state and process long trajectories without leaking privileged information.
Summary
Rayan Garg argues that long-horizon progress depends on how tasks, environments, and verifiers are designed. A task can look long because unrelated work has been chained together, while a shorter task can contain real difficulty when an early mistake changes everything that follows. Human completion time is useful, but it varies with expertise and can misrepresent work that models automate easily. Model-specific measures such as tokens and tool calls are noisy, yet they reveal limits around context, coherence, and compaction. As tasks become more open-ended, deterministic tests are often insufficient. Judges need to inspect the final environment state, access tools with read-only safeguards, and process long trajectories in a queryable form. Garg also argues that dense rubrics require careful QA, since inconsistent grading wastes training compute. He closes by questioning whether several finance benchmarks are genuinely long horizon, broad enough, or granular enough to train capable models.
Long horizon is a relative measurement rather than a fixed task category
Garg begins by separating the idea of a long task from a fixed definition of long horizon. The boundary changes as agents improve, so a task considered long a year ago may no longer qualify today. He describes horizon as a scalar measure that helps compare tasks, rather than a binary label. One common approach uses human time and a success threshold. For example, a 50% threshold at 16 hours means that a model completes tasks that take a human 16 hours with 50% success. This gives an endpoint for comparison, but it does not fully describe the task or its difficulty.
Human time and model tokens answer different questions
Human completion time provides a reference point, while tokens, steps, and tool calls describe the model's own workload. Model-specific measures are noisy because the model, harness, and context-management method change the count. A task taking GPT 500,000 tokens does not directly tell you what it would require from another model. Still, token use can expose technical limits around long trajectories, coherence, context windows, and compaction. Garg says the same model completing longer trajectories after an improved context window or compaction method gives useful evidence about the technical frontier, even though humans do not measure their work in tokens.
Human estimates become less reliable near the frontier
Human hours can make tedious work look harder than it is for a model. Garg uses a large Excel formatting task as an example. A financial analyst might spend days changing colors and themes manually, while a model could write a Python script to finish the work quickly. Human estimates also vary with the experts who provide them. As tasks move toward work that only the top 10%, 1%, or 0.1% of people can do, those estimates become especially noisy. Garg's conclusion is that human and model measures should be kept together because they describe different limitations.
Environment complexity comes from coordination and state changes
Garg defines environment complexity through the tools and external systems an agent must coordinate. A realistic software task may involve Grafana for observability, GitHub for CI/CD, AWS CloudWatch, and a database. Complexity also depends on how much the environment changes during the task. Simply chaining unrelated tasks can create an artificial long horizon. A meaningful task has dependencies between decisions, so that an earlier action changes what the agent must do later. The environment therefore measures more than the number of steps. It tests whether the model can manage information and state across connected tools.
Sequential dependencies create real long-horizon difficulty
The talk distinguishes parallelizable complexity from sequential complexity. An agent analyzing a large codebase can split files among subagents, inspect them in parallel, and combine the results. That increases workload without necessarily creating deep dependency. A different situation occurs when an agent uses dashboards or logs. A bad early query or a misread can send all downstream work in the wrong direction, with consequences that appear later. The difficulty comes from the environment state and the effect of earlier decisions. Garg treats this cascading structure as a more meaningful test of long-horizon capability than serially attaching independent tasks.
Ambiguity makes tasks more realistic and evaluation harder
Ambiguity includes the instructions, artifacts, and starting information given to the agent. Garg says real human work often begins with incomplete information, so environments should test exploration rather than provide a fully specified path. More ambiguity creates more possible routes and more ways to reach a correct result. That creates a direct evaluation problem: standardized grading becomes harder when correct solutions differ widely. A reference trajectory cannot cover every valid path. The environment and verifier must allow exploration while still identifying whether the resulting work is correct.
Final-state verification matters when deterministic tests do not apply
As tasks move from mathematics and tightly defined coding problems into software and file-based work, a Python script, test suite, or formal proof may not be enough to verify the result. Garg describes judges or critic models that inspect both the final state of the environment and the trajectory that produced it. The final state can show whether the requested changes actually took effect. The trajectory can reveal reward hacking, such as escaping a sandbox or viewing a hidden test suite. Both views matter because a correct-looking result reached through invalid access is not a valid solution.
Judges need environment access with strict safeguards
Garg presents judges as agents that may need the same tools as the worker agent. In a deployment-failure task, a judge might inspect GitHub CI/CD logs and AWS logs, check the code changes, and confirm that the redeployment works. Tool calls alone are unreliable evidence. The judge needs to inspect the resulting environment directly, especially for open-ended work. Its permissions must be restricted so it cannot mutate the environment after the task. Read-only access can prevent an accidental deployment or other change. The judge's harness therefore needs to scale with the environment while preserving these safeguards.
Long trajectories should become queryable instead of being pasted into one context
A long trajectory cannot always be stuffed into a judge's context window as one basic language-model call. Garg suggests storing the trajectory in a database, enriching it with metadata, using subagents to process parts of it, and separating phases such as log analysis, coding, and post-change checking. This makes important steps searchable. The judge can then investigate likely failure points rather than treating every action as an undifferentiated text record. Reusing the worker's harness can help the judge inspect the environment, but the processing pipeline still needs to make long histories usable.
"A huge reason we did that is because we feel like a lot of the literature and data being produced right now and being used to train and evaluate models is actually flawed."18:17
Who should watch
You are building agent benchmarks and need a definition of long horizon that does not depend on one noisy number.
Your environment spans software tools, logs, databases, or other systems where an early mistake can change later work.
You are training agents with judge models and need practical guidance on permissions, trajectory processing, rubrics, and grader QA.
