An automation percentage can describe more AI-assisted work while human review, escalation, training, and rework continue to grow with workload.
2
AI-attributed work and autonomous completion are different claims, so the workflow boundary must state whether completion means drafting, approval, or production deployment.
3
A credible automation claim needs expert-checked evaluation and unit economics that include model costs, review, escalation, and rework for every accepted autonomous task.
Summary
Soukaïna Abelhad gives a practical diligence process for testing automation claims in an AI product or vendor pitch. Her first check compares workload growth with fully loaded human labor. More output can still require more reviewers, trainers, operators, escalations, and rework, which describes assistance rather than labor removal. The second check asks where the workflow boundary ends. AI-attributed work may be drafted by a model and then edited or approved by people, while straight-through processing requires end-to-end completion without human intervention inside a named boundary. The third check examines evaluation. Automated graders can be more optimistic than domain experts, so buyers should ask who labeled the ground truth, how expert agreement was measured, and what bad output escaped. Abelhad then turns surviving claims into a cost per accepted autonomous task, including model, human review, escalation, and rework costs. She is careful about the limits of public examples and does not claim that weak AI positioning caused Builder.ai's insolvency.
An automation percentage can hide a growing human workload
Abelhad says a number such as "55.8% faster" may be a genuine measurement that cannot support the conclusion drawn from it. Her first diligence check needs no model expertise. Compare workload volume with fully loaded human labor over the company's own 18-month history. Count operators, reviewers, trainers, escalation work, and rework that exist because of the AI workflow. She cites Faros observational data in which high-AI-adoption teams merged 98% more pull requests, review time rose 91%, and the average pull request became 154% larger. The data is not a controlled labor study, but it shows why more output does not automatically mean less labor.
AI-attributed work is different from autonomous completion
Abelhad uses Uber's claim that 70% of its pull requests are AI-attributed as an example of a metric that does not reveal the full autonomous rate. A workflow can involve AI drafting, human editing, human approval, or no human intervention. She recommends using the finance term straight-through processing for complete end-to-end work without human intervention inside a named boundary. The diligence question is whether the boundary ends when code is drafted, when a pull request is approved, or when the change reaches production. Without that boundary, the percentage cannot be inspected as a claim about autonomous work.
Evaluation needs comparison with the people who accept the work
A precise definition still does not make a measurement trustworthy. Abelhad cites Meta's finding of a 24 percentage-point gap between an automated grader and the merge decision after adjustment to a human baseline. Four maintainers from three repositories reviewed 296 AI-generated pull requests without knowing whether the patches came from AI or a person. She also describes a project where all outputs passed automated validation but very few survived expert review. Buyers should ask who labeled ground truth, whether experts agree, what the automated judge was validated against, and what bad output escaped.
Public warning signs do not become useful until diligence turns them into a decision
Abelhad discusses Builder.ai carefully. She does not claim that weak AI positioning caused the company's collapse, since its insolvency also involved revenue and financing issues. The Wall Street Journal questioned its AI capability in 2019. The Financial Times later reported that its estimated 2024 revenue had been revised from $220 million to roughly $55 million, and Builder.ai filed Chapter 7 on June 2. Her point is that warning signs can exist without becoming a gate in the diligence process. A buyer needs a repeatable process that turns questions about the claim into a decision.
The useful economic metric is the cost of each accepted autonomous task
After a claim survives the arithmetic, definition, and evaluation checks, Abelhad says it still has to survive the economics. Start with eligible workflow value, then apply actual adoption, straight-through process success, and accepted quality. Subtract AI costs, human review, escalation, and rework. The metric she wants is cost per accepted autonomous task. A failed or escalated task still belongs in the cost account, as does work needed to correct it. This connects the percentage in a deck to the work that was actually completed and accepted.
A diligence conversation should test what must be true for the number to matter
Abelhad summarises the process as three checks: compare workload with fully loaded human labor, define the straight-through rate inside a named workflow, and ask who defines quality and whether experts agree with the evaluation. Only then should the surviving claims be translated into unit economics. Her printable card is intended for vendor meetings and investment discussions. The final test is not whether the number looks impressive. It is what must be true for the number to matter.
