Accepted agentic tasks produced cleaner, more meaningful failures than rejected tasks.
2
With the same model, compute budget, and task count, low-quality tasks improved the base model by about 1%, while high-quality tasks improved it by about 6%.
3
Task quality problems often come from mismatches between requested outcomes and tests, or from unstated dependencies that leave the model without needed context.
Summary
Kobie Crawford describes Snorkel's study of task quality in TerminalBench-style agentic environments. The team divided tasks into accepted and rejected buckets after checking whether each task was achievable, non-trivial, functionally correct, and supported by a reliable environment. Accepted tasks averaged twice as many tool calls, lower pass rates, and more output tokens, which suggested genuine difficulty. Their failures also carried better training signals because they usually came from the model failing to complete a difficult task rather than from broken task logic or environmental problems. In an RL training comparison using the same model, compute budget, and number of tasks, low-quality tasks improved the base model by about 1%, while high-quality tasks produced about a 6% improvement. Crawford also discusses underspecified tasks, implicit dependencies, human experts, LLM judges, rubrics, and the difficulty of evaluating tasks with multiple valid outcomes.
Task quality remains important when training moves into agentic environments
Crawford connects Snorkel's focus on data quality to agentic tasks. The study uses TerminalBench-style tasks with a containerized environment and a task definition inside it. He argues that changing the model architecture or evaluation harness can affect results, but the quality of the underlying tasks still matters. The team wanted empirical evidence rather than an assumption that task quality affects training. Their setup examines whether tasks provide useful training examples for models that must work through tools and multiple steps.
A useful task must pass checks for solvability, difficulty, correctness, and environment behavior
Snorkel evaluates four parts of a task. The task must be achievable, non-trivial, functionally correct, and run in a reliable environment. Containerization supports reproducibility, isolation, and parallel rollouts. The research harness checks these criteria before a task enters the training or research set. Tasks that pass become accepted tasks. The rest go into a rejected bucket. Crawford uses these buckets as a practical comparison between higher-quality and lower-quality tasks.
Accepted tasks look harder in several measurable ways
When Snorkel compared the two buckets using models including Claude 4.5 and Codex, accepted tasks averaged twice as many tool calls. They also had lower pass rates and required more output tokens. Crawford treats these patterns as signs that the accepted tasks require more steps, more interaction with external tools, and more reasoning. A low pass rate alone could be misleading, though, because a task might be failing for an environmental or specification problem rather than because the model lacks the needed capability.
The value of a hard task depends on whether its failures are meaningful
The team grouped failures into categories such as logic errors, incomplete tasks, and environmental problems. Crawford distinguishes a model failing to reach the logical conclusion of a difficult task from a task that no model could complete because its environment or setup is broken. Accepted tasks produced cleaner failures. Those failures indicated that the model needed to improve at a real task skill, so the examples could support hill climbing during training. Rejected tasks more often supplied tactical or irrelevant failure signals.
Task quality created a fivefold difference in training uplift
Snorkel ran two RL training experiments with the same model, compute budget, and number of tasks. Training on low-quality tasks improved the base model by about 1%. Training on higher-quality tasks improved it by about 6%. Crawford describes this as a 5x difference in uplift caused by task quality alone. The result supports his view that a lower pass rate is useful only when it comes from genuine task difficulty. Bad specifications can instead add noise to the training set.
Rejected tasks can make evaluation noise hide real model improvement
In response to a question about mixing accepted and rejected tasks, Crawford says the study did not provide a direct answer about whether rejected tasks pull performance down. He describes another pattern from comparisons involving TerminalBench versions and other public benchmarks. Some tasks never get completed because they cannot be completed as written. Those tasks can make it harder to tell whether a model improved, because their failures add noise to the evaluation rather than measuring a model capability.
Ambiguous tests can make a task appear harder without teaching a useful skill
Crawford gives two common causes of rejected tasks. The task description may fail to specify the outcome clearly, while the tests expect behavior that was never requested. A task can also rely on an implicit dependency without telling the model that the dependency exists or providing it in the model's context. These mismatches make the task appear difficult, but the model cannot solve it from the information it received. Crawford treats this as a specification problem rather than evidence of a useful challenge.
Human rubrics and LLM judges help extend evaluation beyond simple pass or fail
For longer tasks and domains with several possible valid outcomes, Snorkel combines human annotators with LLM judges. Crawford says rubrics contain a longer set of criteria that can support both qualitative judgments and more quantitative comparisons. Human experts provide ground-truth information, which helps inform the LLM judges. The team measures agreement between individual humans and between humans and judges as part of quality assessment. Explicitly verifiable coding tests remain easier to evaluate than tasks with fuzzy outcomes.
"Those kinds of mismatches are some of the places where you can see where the task becomes or at least appears to be harder because the tests don't match with the requested task setup."14:32
Who should watch
You are building coding or terminal-agent training data and need to separate genuine difficulty from broken task setup.
Your benchmark has many failures, but you are unsure whether they measure model capability or missing context, bad tests, or unreliable environments.
You are evaluating longer-horizon tasks with human annotators or LLM judges and need ideas for rubrics and agreement checks.
