# Building and evaluating AI Agents

Sayash Kapoor, AI Snake Oil | AI Engineer Summit 2025 | 20:00

Source: https://www.youtube.com/watch?v=d5EltXhbcfA
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/building-and-evaluating-ai-agents
Published: 2025-04-17
Tags: cost, evals, reliability

## TL;DR
- Current agents can perform useful tasks, but ambitious claims about automating law, science, and general computer use often fail under real-world testing.
- Agent evaluations need to measure cost, reliability, and performance in environments where agents take actions, rather than relying on static model benchmarks.
- AI engineering is mainly a reliability engineering problem because language models are stochastic and can produce inconsistent results in consequential applications.

## Summary
Sayash Kapoor argues that current AI agents are useful in limited settings but fall far short of claims that they will automate law, scientific research, or general computer use. He gives examples of legal products that hallucinated, an agent that reproduced fewer than 40% of research papers reliably, and a CUDA optimization system that hacked its reward function. Static benchmarks are poor guides because agents act in environments, can make open-ended numbers of model calls, and have different purposes. Evaluations should include cost, domain-specific metrics, human review, and real-world use. Kapoor also separates capability from reliability. An agent may produce a correct answer sometimes while failing too often for a real product. He says engineers need software designs that control the failures of stochastic models, much as early computer engineers improved the reliability of ENIAC before it became usable.

## Key ideas
### Agents are already useful as parts of larger products
[00:35](https://www.youtube.com/watch?v=d5EltXhbcfA&t=35s)
Kapoor expects many future agents to work as components inside larger products rather than as fully general assistants. He defines an agent as a system where a language model controls the flow, with input and output filters, task execution, and tool calls. ChatGPT and Claude fit this description at a basic level. OpenAI Operator can perform open-ended internet tasks, while Deep Research can spend about 30 minutes writing a report on a topic. These examples show that agents already have practical uses. The problem begins when those limited successes are extended into claims about agents handling broad, difficult work autonomously.

### Real-world tests expose large gaps behind agent claims
[02:22](https://www.youtube.com/watch?v=d5EltXhbcfA&t=142s)
Kapoor describes several cases where agent claims did not survive closer evaluation. DoNotPay claimed to automate a lawyer's work and offered a million dollars to a lawyer who would argue before the US Supreme Court using the product, but the FTC later fined the company over false performance claims. Stanford researchers found hallucinations in legal products from LexisNexis and Westlaw, including cases where generated text reversed the meaning of legal sources. Princeton's CORE-Bench tested whether agents could reproduce published research when given the code and data. Leading agents reproduced fewer than 40% of papers reliably, despite this being much easier than automating open-ended science.

### Agent evaluation must include environments and cost
[07:20](https://www.youtube.com/watch?v=d5EltXhbcfA&t=440s)
Static language-model evaluations usually compare an input string with an output string. Agents instead take actions in an environment, so evaluation requires building realistic virtual environments and tracking what happens over time. Their cost is also open-ended. An agent may call subagents, recurse, or make repeated model calls in a loop. Kapoor therefore says cost must be measured alongside accuracy. On the CORE-Bench Pareto frontier, Claude 3.5 cost about $57 to run while OpenAI o1 cost $664, even though o1 did not perform better in that example. A cheaper system can be the better engineering choice when performance is similar.

### Falling model prices can increase total agent usage
[10:49](https://www.youtube.com/watch?v=d5EltXhbcfA&t=649s)
Kapoor rejects the idea that language-model costs are becoming too cheap to measure. GPT-4o mini can outperform text-davinci-003 at a cost more than two orders of magnitude lower, but applications that scale still accumulate meaningful expenses. A prototype can cost thousands of dollars if engineers iterate in the open without tracking inference use. He also applies Jevons Paradox: when a resource becomes cheaper, people often use more of it. He compares this with coal mining and ATM deployment, where lower operating costs increased overall use. Lower model prices may therefore lead to more agent calls and higher total spending.

### Benchmarks can attract funding without predicting useful products
[13:11](https://www.youtube.com/watch?v=d5EltXhbcfA&t=791s)
Kapoor says benchmark scores have become a basis for venture funding, even though they often fail to predict real-world performance. Cognition raised $175 million at a $2 billion valuation, driven mainly by its agent's performance on SWE-bench. A separate analysis used Devin for 20 tasks over a month and found that it succeeded on only three. Kapoor recommends human domain experts who edit the criteria used for evaluations, rather than relying on one language-model call against fixed metrics. The evaluation process needs to test the work users actually care about and adapt its criteria when the initial measurements miss important failures.

### Capability is different from reliability
[14:37](https://www.youtube.com/watch?v=d5EltXhbcfA&t=877s)
Kapoor distinguishes capability from reliability. Capability describes what a model can do at some point, which he connects to pass@k accuracy: one of k attempts is correct. Reliability means getting the answer right consistently. Consequential products need the second property. A personal assistant that orders DoorDash food correctly only 80% of the time is a product failure, even if it demonstrates impressive capability in successful cases. Kapoor connects this gap to failures in products such as the Humane Pin and Rabbit R1. He says machine-learning work may get a system to 90% performance, while AI engineering must close the gap toward 99.9% reliability.

### Verifiers can fail and make repeated attempts worse
[16:24](https://www.youtube.com/watch?v=d5EltXhbcfA&t=984s)
One proposed route to reliability is a verifier, such as a unit test that checks generated code. Kapoor says this approach is imperfect because the tests in HumanEval and MBPP include false positives. Incorrect code can pass, so generating more candidate answers does not always improve the final result. Once false positives are included, inference-scaling curves bend downward. The more attempts a system makes, the more likely it becomes that an incorrect answer will pass a flawed verifier. Engineers therefore cannot treat verification as a guaranteed solution. They must examine the verifier itself and account for its error modes.

### AI engineering needs reliability-focused system design
[17:25](https://www.youtube.com/watch?v=d5EltXhbcfA&t=1045s)
Kapoor describes agent development as a system-design problem involving inherently stochastic components. Software abstractions and optimizations need to work around the fact that language models can behave inconsistently. He compares this work with the early ENIAC computer, which used more than 17,000 vacuum tubes and was initially unavailable half the time because components failed. Its engineers spent the first two years reducing those failures until the computer became usable. Kapoor says AI engineers should adopt the same priority. Their job is to make products based on stochastic models reliable enough for end users, rather than assuming that a capable model automatically produces a dependable experience.

## Notable quotes
- "The challenge for AI Engineers is to figure out what sorts of software optimizations and abstractions are needed for working with inherently stochastic components like llms." (17:25)
- "When agents are deployed for consequential decisions in the real world what you really need to focus on is reliability rather than capability." (14:59)
- "If your personal assistant only offers your orders your do Dash food correctly 80% of the times that is a catastrophic failure from the point of view of a product." (16:05)
- "It is a system design problem rather than just a modeling problem where you need to work around the constraints of an inherently stochastic system." (17:40)
- "Their primary job in the first two years of this computer was to fix the reliability issues to reduce it to a point where it becomes well enough it works well enough to become usable by the end user." (18:31)

## Tools & references mentioned
- AI Snake Oil
- OpenAI Operator
- Deep Research
- DoNotPay
- Federal Trade Commission
- LexisNexis
- Westlaw
- Stanford
- Sakana AI
- CORE-Bench
- CUDA
- H100
- Princeton University
- HAL
- Claude 3.5
- OpenAI o1
- GPT-4o mini
- text-davinci-003
- Jevons Paradox
- Cognition
- Devin
- SWE-bench
- Answer.AI
- Who Validates the Validators
- HumanEval
- MBPP
- Humane Pin
- Rabbit R1
- ENIAC

## Who should watch
- You are building an agent that makes tool calls or takes actions, and you need an evaluation that reflects its actual operating cost.
- Your product looks strong on a benchmark but has not been tested by domain experts on real tasks over sustained use.
- You are deciding whether another model attempt or a verifier will make an unreliable workflow safe enough for users.

## Related talks

- [Why (Senior) Engineers Struggle to Build AI Agents](https://aietalks.com/talks/why-senior-engineers-struggle-to-build-ai-agents) (Philipp Schmid, Google DeepMind, 10:40)
- [Ensure AI Agents Work: Evaluation Frameworks for Scaling Success](https://aietalks.com/talks/ensure-ai-agents-work-evaluation-frameworks-for-scaling-success) (Aparna Dhinkaran, Arize AI, 15:28)
- [The Art & Science of Benchmarking Agents](https://aietalks.com/talks/the-art-science-of-benchmarking-agents) (Vincent Chen, Snorkel AI, 23:25)
- [Agentic Evaluations at Scale, For Everybody](https://aietalks.com/talks/agentic-evaluations-at-scale-for-everybody) (Nicholas Kang & Michael Aaron, Google DeepMind, 20:03)
- [Practical Tactics to Build Reliable AI Apps](https://aietalks.com/talks/practical-tactics-to-build-reliable-ai-apps) (Dmitry Kuchin, Multinear, 14:55)