How Ramp Tests Its AI Bookkeeping Agent Before Customers Do

Ryan Stevens, Ramp18:39 · Nov 2026 · 5,963 views
Thumbnail for How Ramp Tests Its AI Bookkeeping Agent Before Customers Do Watch on YouTube
TL;DR
  1. 1

    Ramp breaks bookkeeping into tasks with explicit success criteria before building an agent benchmark.

  2. 2

    Synthetic businesses give Ramp stable data and repeatable back-tests before it has enough production users.

  3. 3

    Skill ablation, cross-period memory tests, and scheduled regression runs produced useful improvements, while prompt auto-tuning did not.

Summary

Ryan Stevens explains how Ramp evaluates Stack, its AI agent for bookkeepers, before relying on customer data. The team first breaks bookkeeping into tasks such as reconciliation, data entry, variance analysis, and schedules and accruals. It then works with bookkeepers and accounting firms to define what success means for each task. Because early-stage agents lack enough user data, Ramp builds synthetic businesses with complete records, including ERP data, vendors, customers, and accounts payable. These environments support repeatable back-tests. Stevens compares partial-credit accuracy with pass@k, which requires a run to get every criterion right. Ramp found that accounting agents usually fail through reasoning rather than data gathering. Experiments with auto-tuning skills did not work. Removing unnecessary skills helped, and testing memories across two time periods produced a large gain. Ramp now runs evals on a schedule and sends suspected regressions to Slack for investigation.

Key ideas
00:54

A useful agent starts with a precise description of the job

Stevens says the first task is to break down the process, its tasks, and what success looks like. For bookkeeping, the broad goal is closing the books. Ramp worked with bookkeepers and accounting firms to understand the full process for individual tasks. In reconciliation, an accountant compares an accounts payable ledger in an ERP with the bills in another system, identifies whether the numbers tie, finds the cause of differences, and fixes them. This concrete workflow gives the team something it can test instead of asking whether an agent is generally good at accounting.

02:31

Ramp evaluates four bookkeeping task categories

Ramp's benchmark covers reconciliation, data entry, variance analysis, and schedules and accruals. Variance analysis asks why numbers changed, while schedules and accruals involve a more complex accounting process that Stevens does not explain in detail. Each category is turned into a task with criteria for success. The goal is to evaluate the work an accountant needs completed, rather than judging an agent by a broad conversational standard. Stevens treats benchmarks, evals, and verifiability as different names for the same basic activity: finding ways to measure whether the task was completed correctly.

03:17

Synthetic businesses provide repeatable data before production usage

Before an agent has enough users, its data distribution is not available for evaluation, so Ramp builds simulations and synthetic data. In partnership with Mercor, the team created synthetic businesses such as a direct-to-consumer lifestyle company with Amazon and Shopify revenue, a calendar-year accounting setup, and month-end close requirements. Ramp also created the supporting records, including an ERP, customer and vendor lists, and accounts payable data. Stevens calls data the bottleneck because an evaluation needs clean data to run on. A stable synthetic business lets the team run the same task repeatedly while changing the agent.

05:45

An eval is a back-test over a stable world

Ramp models each task as a world, which can be a synthetic business or a snapshot of production data. The agent runs in that world, and the final output is graded against roughly 10 to 20 criteria for a task. Ramp initially focuses on whether the result is right, cheap, and fast. It does not primarily grade the trajectory, such as whether the agent chose the most efficient tools or invoked the ideal skills. Stevens says the team spent four to six months building these evals before launching Stack. He considers that level of work normal when an agent operates in an area for which frontier models are not fully optimized.

09:11

Accuracy and pass@k answer different evaluation needs

Accuracy gives partial credit. In Stevens's example, an agent gets four of six criteria right and scores 67 percent. Pass@k is all or nothing: it asks whether one of the first k runs gets every criterion right. If three runs solve four, five, and then six criteria, pass@1 and pass@2 are zero while pass@3 is one. Stevens says pass@k may matter more in accounting because users need to trust the complete result. A single wrong step can make an otherwise strong run unusable to an accountant.

11:18

Accounting failures usually come from reasoning

When Ramp examined why models failed, it separated reasoning failures from failures to gather data, follow instructions, or reflect. Accounting agents generally gathered the needed information successfully. The harder problem was reasoning through that information incorrectly. Stevens says accounts payable is comparatively easy to explain and that the models still have room to improve their reasoning. His conclusion is direct: a frontier model cannot simply be dropped onto the problem and expected to solve it. The evaluation data helps Ramp identify where the agent needs changes instead of treating every failed run as the same kind of error.

12:18

Auto-tuning prompts did not improve the agent

Ramp tested auto-tuning and auto-researching skills, including an approach Stevens calls Jeepa. The team did not see success. Stevens doubts that hill climbing through text space is a good first approach because it is hard to see what direction a useful gradient would provide. He contrasts this with tuning machine-learning models, where a gradient and a finite search space give the optimization process clearer direction. His warning is limited and practical: auto-research should not be assumed to fix a bad agent. Ramp's willingness to discuss failed experiments gives the evals a role beyond selecting only successful changes.

13:30

Removing skills and testing memory across time produced gains

Ramp used skill ablation to measure what happened when individual skills were present or absent. The aim was to trim unnecessary skills and save context, while creating a candidate set for deleting or compressing them. The team also tested trajectory-distilled memories with roll-forward worlds. It ran the same task in two time periods, wrote a memory after the first period, and then either injected that memory or let the agent search for it during the second period. This tests whether the memory captures implicit customer rules that can be inferred from success criteria. Stevens says memory quality and later success depended heavily on the prompts, and this produced one of the largest gains he saw.

17:01

Scheduled regression runs are safer than strict CI gates

Ramp hosts its evals on an internal platform where teams can add cases and tag them for organization. The cases run on a recurring schedule, such as daily or weekly. A background agent examines failures and regressions, then sends a Slack message with a suspected cause for someone to investigate. Stevens does not favor blocking continuous integration on agentic evals because non-determinism can produce passage rates around 70 or 80 percent. Scheduled runs still make regressions visible without pretending that every evaluation result is stable enough to gate a build automatically.

"The first task in building an agent for a process is breaking down the job, the tasks, and what success looks like."00:54
Who should watch
  • You are building an agent for a domain with little production data and need a repeatable way to test it before launch.
  • Your team has an eval suite but is unsure whether to measure partial accuracy, pass@k, or the agent's trajectory.
  • You are adding skills or memory to an agent and want experiments that can show whether those additions help or hurt.