60% of Our Data PRs Now Merge With No Human Review

Chetas Joshi, Robinhood28:00 · Nov 2026 · 4,440 views
Thumbnail for 60% of Our Data PRs Now Merge With No Human Review Watch on YouTube
TL;DR
  1. 1

    Robinhood combines deterministic scripts, an AI reviewer, and human approval for different kinds of data pipeline risk.

  2. 2

    The agent found an unregistered downstream consumer that would have failed after an integer column was widened to a big integer.

  3. 3

    As of August 31, 2026, about 60% of data pipeline PRs merged without human review, with about 132 engineering hours saved each month.

Summary

Chetas Joshi describes Robinhood's hybrid reviewer for data pipeline pull requests. Deterministic checks query systems such as the schema registry, data catalog, lineage tool, and staging environment. They handle facts that can be checked directly. An agent then reads the repository, follows pipeline definitions and access grants, and looks for risks outside the recorded system state. It can approve low-risk changes or escalate with comments, reasoning, and a risk score. Human reviewers retain decisions about sensitive-data grants, changes to data meaning, and cases where the agent lacks confidence. In the example, the agent found an unregistered job that would fail when an integer column received a value too large for its type. Joshi also explains how human feedback becomes manually updated review rules. As of August 31, 2026, about 60% of these PRs merged without human review, saving about 132 engineering hours per month. He is candid about limits, including separate system execution and the agent's inability to query live production facts.

Key ideas
00:00

Machine-generated PRs create a review queue that human context cannot absorb

AI coding assistants make schema changes, pipeline code changes, configuration changes, and data-access grants cheap to open. Those PRs arrive faster than the small data platform team can review them. Existing CI catches broken code, inefficient queries, and some downstream or privacy risks, but it cannot answer every question about how a production table is used. The people who know what is safe to merge remain limited, so the queue grows even when the automated checks are green. Joshi frames the problem as machine-speed changes meeting human-speed review.

03:36

Deterministic checks handle facts before the agent makes judgments

Robinhood separates the reviewer into layers. Scripts check authoritative systems such as the schema registry and catalog. They verify that a new source column exists, that tables have owners, and that personal information is labeled. Other scripts walk registered lineage for downstream breakage and run performance tests in staging at production-scale data volume to check cluster sizing and freshness SLAs. A failed check turns CI red and blocks the PR. The agent does not repeat these fact checks, which keeps its work and cost focused on questions that require context.

10:31

Repository search catches downstream consumers that registered lineage misses

The agent reads the diff, PR description, commit messages, pipeline definitions, and access grants in the repository. It looks for query anti-patterns, sensitive-data requests, and consumers outside the lineage record. In the example, the widened integer column looked safe to every registered consumer. The agent found an unregistered job that copied the value into a table where the column was still an integer. A sufficiently large value would make that downstream write fail. The agent explained the full chain and escalated it. Joshi is clear that repository search cannot see ad hoc queries or dashboards, so finding no consumer is never proof that none exists.

14:56

The agent approves ordinary changes but cannot decide protected cases

A merge needs both passing deterministic checks and an approving review. The agent can approve when it finds no serious risk and has enough confidence. It must escalate grants involving sensitive data, changes that alter the meaning of data rather than its movement or storage, and any verdict where it lacks confidence. In the example, a human confirmed that the unregistered consumer was live. The downstream table was changed first, then the original PR merged. This division keeps accountability with a person for decisions Robinhood does not want an agent to make.

18:06

Human corrections become manually maintained review rules

The system records the agent's verdict and reasoning in its review comments. A human replies with agreement or an explanation of why the finding is wrong. Each week, the on-call engineer reads this feedback and updates the rules by hand. In one example, the agent objected to labeling a column nonsensitive because its source was PII. The author and a human reviewer explained that the pipeline encrypted it before writing it to the lake, so the label was appropriate in that specific pipeline. Joshi stresses that encryption is not a blanket rule for changing sensitivity. The relevant pipeline context has to be present.

20:24

Evaluation must count both missed risks and unnecessary escalations

Robinhood evaluates the reviewer with two error types. A bad PR that was approved is a miss, while a fine PR sent to a person is a false escalation. A system that escalates everything may avoid misses, but it does not reduce review work. As of August 31, 2026, the miss rate was about 2%, mostly because rules lacked context rather than because the agent reasoned incorrectly. One miss involved columns retagged as personal information and removed from an open copy while a downstream reader still depended on them. Robinhood turned that incident into a rule to search for those readers.

23:29

The results reduce review work without removing human accountability

As of August 31, 2026, about 60% of data pipeline PRs merged without human review. Joshi reports about 75% less manual review effort, roughly 132 engineering hours saved each month, and about 30% fewer incidents tied to these PRs. The protected decisions still go to people by design. He also describes current architectural limits: the script and agent layers run independently, the agent cannot query live layer-one checks or production state, and there is no automatic fallback when a component cannot finish. A proposed future design would give the agent live checks as tools, but it was not built yet.

"Scripts prove what can be proven. An agent judges what needs context and people own the calls only they should make."03:36
Who should watch
  • You own data platform or production pipeline reviews and automated changes are creating a backlog for a small group of reviewers.
  • Your team is considering an AI reviewer and needs a concrete boundary between deterministic checks, model judgment, and human decisions.
  • You need an evaluation approach that measures both missed risks and the human time consumed by unnecessary escalations.