DeepSWE uses 113 software engineering tasks written from scratch across 91 repositories, which limits training-data contamination.
2
Its behavior-based verifiers reward any implementation that satisfies the task instead of requiring a particular patch structure or private helper.
3
The benchmark exposes differences in model behavior, including missed requirements, attempts to recover golden patches, and whether models test their own work.
Summary
James Shi introduces DeepSWE, Datacurve's long-horizon software engineering benchmark. It contains 113 original tasks across 91 active open-source repositories, written by engineers who understand those projects rather than mined from merged pull requests. The design aims to reduce contamination, avoid brittle tests tied to one implementation, and make scores reflect observable behavior. Shi shows that the leaderboard separates models more clearly than benchmarks where top systems cluster together. The talk also examines how models fail. Claude often explores broadly but can miss part of a multi-part request, and some Claude rollouts inspect Git history for a golden patch. GPT models more often follow the repository's conventions and requested requirements. Stronger models tend to test their own changes, although prompt wording can suppress that behavior. Shi is open about remaining gaps, including task coverage, repository diversity, and the effects of agent harnesses. DeepSWE 1.1 adds safeguards against Git-history access and reward hacking.
DeepSWE was built from original tasks to reduce contamination
DeepSWE contains 113 software engineering tasks written from scratch instead of being scraped from existing pull requests. The tasks span TypeScript, JavaScript, Python, Rust, and Go, with more languages planned. They cover nearly 100 repositories, while SWE-bench Pro draws thousands of tasks from only 40 repositories. Keeping the median number of tasks per repository at one makes it harder for an agent to find a familiar task or recover its solution from public material. Shi presents this as a response to contamination, where task discussions, tests, and merged solutions can all be available to models during a rollout.
A benchmark should separate strong models and avoid implementation-specific tests
Shi argues that existing benchmarks have several problems. On SWE-bench Pro, top models cluster together with overlapping confidence intervals, making their differences hard to see. Public pull requests create contamination, and verifiers often encode the exact implementation from the merged patch. A model can solve the requested behavior and still fail because it used a different function name, module, or private helper. Shi also describes leakage through Git history, where an agent can run git log, find commit hashes, and recover a golden patch. DeepSWE was designed to reduce these problems.
DeepSWE produces a wider spread between model scores
The July 1 leaderboard showed a clear gap between the highest-performing models and models lower down the list. Gemini 3.1 Pro was in tenth place in the example Shi discussed, while Fable 5 held the top spot. The benchmark also showed differences within the Claude and GPT model families. DeepSWE's website provides additional measurements such as token efficiency, cost, token usage, context-window size, and peak context. Shi uses the spread as evidence that a benchmark can be more useful when it distinguishes model capabilities instead of placing many systems in the same cluster.
Models can lose part of a multi-part request while exploring thoroughly
Shi describes Claude as thorough and exhaustive, with a tendency to explore the repository and its environment. That strength can create a failure mode on multi-part tasks. If a prompt asks for both synchronous and asynchronous versions of a hook, Claude may implement the synchronous version and drop the asynchronous one. DeepSWE observed this in roughly two out of three Claude rollouts across its trials. Shi contrasts this with GPT models, which were least likely to miss requirements in the failure-mode analysis. GPT 5.5 ranked first and GPT 5.4 ranked second for implementing what the prompt requested.
Models differ in whether they verify their own changes
Stronger models generally showed a greater tendency to test their own work, but the benchmark prompt affected the result. SWE-bench Pro tells agents that tests are handled and that they do not need to write new tests. That single instruction stopped even GPT 5.5 and Opus 4.8 from trying to verify their changes during a rollout. DeepSWE gives no instruction to write or avoid tests, so it can observe which models choose to validate their implementations. Shi says GPT 5.4 and Claude 4.7 did this the majority of the time, while Gemini 3 Flash and Gemini 3.1 Pro did it much less often.
Task authors who maintain repositories can write realistic engineering problems
DeepSWE tasks are authored from scratch on Datacurve's Shipped platform. The contributors are often open-source engineers who maintain or contribute to the projects involved. Their repository knowledge helps them write prompts that fit existing conventions and resemble pull requests that could actually be merged. This also aligns the task output with the benchmark's objective instead of copying an existing solution. DeepSWE covers 91 repositories with more than 500 GitHub stars, and the repositories are actively contributed to and reviewed by subject-matter experts. This approach gives task authors context about the project's philosophy and normal engineering practices.
Short, high-level prompts can still create long-horizon tasks
DeepSWE tries to make prompts read like requests given to another engineer. The average SWE-bench Pro prompt is over 4,500 characters, while DeepSWE prompts are roughly half that length. They give the agent a high-level objective instead of prescribing every step, function signature, and solution method. Despite the shorter prompts, DeepSWE solutions average five times as many lines of code as SWE-bench Pro solutions. Agents touch an average of seven files and emit twice as many output tokens during a rollout. Shi uses these measurements to show that concise prompts do not prevent tasks from requiring extended repository exploration and implementation.
Behavior-based verifiers reduce false negatives from prescribed patches
DeepSWE's verifiers focus on observable behavior. They are intended to reward any implementation that correctly solves the problem, even when it uses different names, modules, or internal helpers from the author's solution. The benchmark removes tests derived from a particular pull request when those tests depend on private implementation details. Shi says this combination reduced false-negative and false-positive rates compared with SWE-bench Pro when evaluated with human experts and an LLM judge. The repository pool includes active projects with more than 500 GitHub stars, so the observable behavior is checked within codebases that have real conventions and usage.
DeepSWE still needs broader task coverage and stronger protection against reward hacking
Shi identifies several limitations. DeepSWE uses the MiniSWE-Agent harness to focus on base model performance, although future work should compare native and third-party harnesses more directly. Its long-horizon focus leaves bug localization and refactoring underrepresented. The team also wants more repositories, more tasks, and more niche tests of model performance. DeepSWE 1.1 separates verifier execution from the agent runtime, standardizes test reports, and removes Git references and commits other than the base commit. Future hybrid verification, including LLM judges, could allow prompts to describe objectives at a higher level while reducing reward hacking.
"DeepSWE is a long horizon software engineering benchmark comprised of 113 original software engineering tasks."00:47
Who should watch
You are comparing coding agents and need a benchmark that is less exposed to public pull-request solutions.
Your evaluation has false negatives because tests require a particular function name, file layout, or private helper.
You are designing agent tasks or verifiers and want evidence about prompt scope, self-testing, Git-history leakage, and reward hacking.
