The Agents Were Fine. Everything Around Them Broke.

Joey Wang, Harvey12:58 · Nov 2026 · 4,291 views
Thumbnail for The Agents Were Fine. Everything Around Them Broke. Watch on YouTube
TL;DR
  1. 1

    After 10,000 agent-written pull requests, Harvey found that the surrounding platform was the source of most failures.

  2. 2

    Spectre lets engineers choose their agent while providing shared cloud access, checks, follow-up loops, review queues, and merge controls.

  3. 3

    Once agents could call every action, alert storms and task bursts filled the cluster, so Harvey added deduplication, fair scheduling, and automation guardrails.

Summary

Joey Wang describes Spectre, Harvey's internal cloud agent platform, after it handled 10,000 agent-written pull requests. Harvey did not try to replace Cursor, Claude Code, Codex, or other agents. It built the shared layer around them, including secrets, real application environments, alerts, tickets, checks, review, and merging. A follow-up loop became especially useful when it could read failed CI and push another fix. Spectre then made every user action callable from the command line, by agents, and by CI. That flexibility produced failures at a larger scale. One alert triggered about 900 tasks, while a command-line burst of about 950 tasks filled the cluster. Harvey responded with alert deduplication, a fair launch queue, and limits for unattended automations. Wang is candid that merge throughput is still unresolved. His advice is to build the harness around agents, make actions callable, and cap what any person, alert, or agent can start.

Key ideas
00:00

The harness becomes the product once agents reach real company work

Wang opens with Harvey running more than one agent per minute on average, day and night, and says the agents were the easy part. After 10,000 agent-written pull requests, about 460 people were using the system. The work also moved beyond code, with marketing asking an agent in Slack to turn a prototype and talk track into a storybook for a product video. Spectre grew around these unexpected uses. It provides the surrounding machinery for secrets, real systems, alerts, tickets, checks, review, and merging. Wang's point is that a model can write code while the platform determines whether the work reaches completion.

01:24

Engineers can keep their preferred agent while Spectre owns the shared environment

Harvey did not rebuild its infrastructure separately inside Cursor, Claude Code, Codex, or every future agent. The instruction became, "Just ask Cursor to use Spectre." An engineer can start on a laptop, move the session to the cloud, and bring it back. The cloud supplies secrets, the real stack, alerts, tickets, common rules, and enough capacity for concurrent work. Once code exists, Spectre can respond to an alert, create a real copy of the stack, attach proof to the pull request, and let a follow-up loop read failed CI and push another fix. The pull request then enters the same review process as any other.

03:37

A follow-up loop turns code generation into a completed pull request

Wang recommends building the follow-up loop if a team can build only one part of the workflow. Spectre enabled it for everyone in early October, and four out of five agent pull requests merged into the backend had passed through it that month. The loop handles the work after the first code change: it reacts to CI failures, makes another fix, and keeps the pull request moving while people are away. Harvey also created shared project pages where engineers record testing knowledge, flags, and trusted skills. Agents working in that project can use the same information, while the audit trail and checks stay consistent across agent choices.

05:19

Making every user action callable lets agents coordinate other agents

In May, almost all Spectre tasks started from a browser or Slack, while exactly one came from the terminal during the month. Harvey changed the rule so that if a person could click an action, an agent could call it. Starting a task, enabling the follow-up loop, turning on automerge, and publishing a page became available from the command line. Laptop agents, agents inside sandboxes, and CI could all use the same actions. By October, the command line was starting about 6,500 tasks per month. Wang describes a coordinator agent splitting a large coding task into 10 milestones, sending work to Spectre, and using separate sandboxes for end-to-end validation.

07:06

Task coordination exposed ordinary infrastructure bottlenecks

As agents began coordinating many tasks, the slowest part was not model syncing. Checking out each repository took more than a minute on average. Spectre addressed this by keeping a copy of every repository on every machine and letting tasks borrow it. This is a small example of Wang's larger argument: once agents can start work quickly, storage, scheduling, and execution paths become visible limits. The platform has to remove those delays before more capable coordination produces useful throughput.

07:27

Uncontrolled fan-out can turn one alert into hundreds of duplicate investigations

In mid-August, a release bug generated one alert per workspace. A workflow tagged an agent on every alert, and each tag started a task. A normal day produced about 50 such tasks, but the incident created around 900 in four hours. The agents were all chasing the same bug, and the alert channel became unusable. Harvey changed repeated alerts so they joined an investigation that was already open. Wang presents this as a platform failure caused by allowing an event to fan out without checking whether equivalent work was already underway.

08:28

Fair scheduling protects the platform when users follow the intended workflow

A later burst came from a human following Harvey's advice to hand work to Spectre. About 950 command-line tasks arrived in roughly three hours, filling the cluster and placing everyone else's work behind one laptop. Harvey added a fair launch queue four days later. Tasks receive a position in line, the dispatcher starts them when capacity is available, people at a keyboard go first, and users can see their place. In a low test with a 300-task burst, task latency fell from almost two seconds to 70 milliseconds. Wang says the platform still has room to improve, because several changes landed together and the team cannot identify which fix mattered most.

10:45

Unattended automations need budgets, expiry dates, and automatic shutdown

The fastest-growing source of Spectre work was no longer a person asking for a task. Automations could wake on a schedule or event, read a triage alert, inspect a pull request, and write a post. About 110 people had automations running in the previous month. Harvey gave them guardrails immediately: an automation turns itself off after five consecutive failures, and it can carry a budget or an expiry date. Wang warns that an unattended agent can start thousands of tasks. He also says merge throughput remains unresolved as agents continue opening pull requests, and only one in 20 currently carries proof that it works before human review.

"The work that's lost is the harness that runs it, checks it, and gets it merged."12:28
Who should watch
  • You are building an internal agent platform and need to decide what belongs in the shared layer around different coding agents.
  • Your agents can already start work, but alerts, CI, repositories, queues, or merge systems are becoming the bottleneck.
  • You are adding unattended or agent-to-agent automation and need concrete failure modes and guardrails before increasing task volume.