# AI Agents, Meet Test Driven Development

Anita, Vellum | AI Engineer Summit 2025 | 29:10

Source: https://www.youtube.com/watch?v=U3MVU6JpocU
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/ai-agents-meet-test-driven-development
Published: 2025-02-22
Tags: agents, evals, observability, testing

## TL;DR
- Reliable AI products require continuous experimentation, evaluation, monitoring, and improvement after deployment.
- Agentic behavior ranges from model-level reasoning and tool use to systems that plan, act autonomously, and eventually invent workflows.
- An SEO agent can combine research, writing, memory, and an LLM-based editor to produce a useful first draft.

## Summary
Anita argues that reliable AI systems depend on the workflow around the model. Teams should experiment with prompts, models, retrieval, memory, and orchestration, then evaluate those choices against examples that reflect their actual use case. Subject matter experts should help create the tests, and the testing framework should support custom metrics and nondeterministic outputs. After deployment, teams need logs, latency data, retries, fallbacks, version control, staging, and a feedback loop for production edge cases. Anita also presents a framework for agentic behavior, from basic model reasoning through tool use, structured planning, autonomous monitoring, and creative workflow generation. Her SEO agent demonstrates the approach. It researches search results, identifies missing topics, drafts an article, and sends the draft through an editor that returns feedback to the writer. The talk ends with Vellum's open-source Workflow SDK for defining and debugging these systems in code.

## Key ideas
### Reliable AI systems are built through repeated testing and revision
[00:00](https://www.youtube.com/watch?v=U3MVU6JpocU&t=0s)
Anita says Vellum has worked with hundreds of companies that deployed AI systems in production, and she saw a consistent difference in teams that used test-driven development. They experiment first, evaluate the workflow at scale, deploy it with monitoring, and continue improving it from captured responses. This process applies to simple systems and agentic workflows. Model quality alone does not settle whether a product will work. Teams need to understand their problem and find the right combination of models, techniques, and logic for that specific use case.

### Experimentation should involve domain experts and multiple models
[06:06](https://www.youtube.com/watch?v=U3MVU6JpocU&t=366s)
Before building a production system, teams should test whether models can solve the intended use case. Anita suggests trying few-shot prompting, Chain of Thought, prompt chaining, and more agentic patterns such as ReAct. Domain experts should help tune prompts because they understand what a good answer means, and involving them early can save engineering time. She also recommends staying model agnostic. Different models may fit different tasks, such as Gemini 2.0 Flash for OCR, so the team should compare models against the actual work rather than commit too early.

### Evaluation needs real examples, explicit tradeoffs, and flexible metrics
[07:40](https://www.youtube.com/watch?v=U3MVU6JpocU&t=460s)
At the evaluation stage, teams create a dataset of hundreds of examples and test models and workflows against it. Anita says teams must decide how they will trade off quality, cost, latency, and privacy. Ground-truth data designed with subject matter experts is useful for judging the system's real task, while synthetic benchmarks may not capture the use case. If ground-truth data is unavailable, one LLM can evaluate another model's response. The testing framework should handle nondeterministic outputs, support custom metrics, and allow metrics to be written in Python or TypeScript.

### Production testing includes every workflow step and the surrounding APIs
[09:42](https://www.youtube.com/watch?v=U3MVU6JpocU&t=582s)
Anita recommends evaluating internal nodes during prototyping and after deployment. Teams should log every LLM call, including inputs, outputs, and latency, because agentic systems can take different paths and make their own decisions. API reliability also needs explicit handling through retries and fallback logic. She gives OpenAI's four-hour outage as an example of why a production system may need to switch to another model. Version control, staging, controlled rollouts, and deployment schedules separate from the main application are also needed so prompt changes do not introduce regressions.

### Production data creates the feedback loop for continued improvement
[12:28](https://www.youtube.com/watch?v=U3MVU6JpocU&t=748s)
Once a system is live, teams can capture user responses and edge cases, add those cases to their evaluation data, and test new prompts against them. Anita also recommends caching repeated queries to reduce cost and latency. After a workflow has run reliably for a longer period, its production data may support fine-tuning a custom model for the specific use case. Agentic systems make this work more demanding because they use many tools and APIs, may contain multiple agents, and can execute actions in parallel. Evaluation must check the agents' decisions and behavior as well as their outputs.

### Agentic behavior can be described as increasing control, reasoning, and autonomy
[14:15](https://www.youtube.com/watch?v=U3MVU6JpocU&t=855s)
Anita presents a framework with levels that are still evolving as models improve. At the first level, an LLM retrieves data and produces an answer, with no external planning or decision-making beyond the prompt and model behavior. The next level adds tool use, where the system decides whether to call an API or retrieve more data. Memory becomes important for multi-threaded conversations, and evaluation is needed at each step. A further level adds structured reasoning, planning, task breakdown, tool selection, usefulness checks, and refinement in a finite loop.

### Higher levels keep working and eventually create their own workflows
[22:48](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1368s)
At a more autonomous level, a system can monitor services such as email, Slack, or Google Drive, decide what to do next, act in real time, or ask a person for input. The final level would allow an AI system to create its own workflows, utilities, prompts, function calls, and tools. Anita says this level is currently out of reach because models overfit their training data and carry inductive biases. She places many production systems at the tool-use level, while structured planning and reasoning are where she expects much of the near-term innovation.

### The SEO agent turns research, writing, and review into a testable loop
[22:57](https://www.youtube.com/watch?v=U3MVU6JpocU&t=1377s)
Anita's SEO agent takes a keyword, writing style, and target audience, then analyzes highly ranked Google articles. It identifies useful elements to reinforce and missing topics that require more research. A writer uses that information to create a draft. An LLM-based editor judges the draft against rules in its prompt and sends feedback back through a memory component containing the writer-editor history. The loop continues until a condition is met. In the demo, the workflow produced a draft titled "Mastering Chain of Thought Prompting in AI: A Comprehensive Guide for Developers." Anita says the result gave her useful foundations for further editing.

## Notable quotes
- "Those companies who have adopted a test driven development approach were able to build reliable and stronger systems for production." (00:00)
- "Success isn't just about the models anymore, it's about how you build around it." (03:58)
- "Your AI isn't static, so your workflow should also be dynamic." (09:21)
- "You need to assess the behavior of these agents so that you can make sure that they're making the right decisions and following the intended logic." (13:57)
- "Developers want more code, developers want more control and flexibility, and they want to own their definitions in their codebase." (27:48)

## Tools & references mentioned
- Vellum
- Cursor AI
- DeepSeek-R1
- OpenAI
- GPT-3.5
- GPT-4
- Gemini 2.0 Flash
- Chain of Thought
- ReAct
- RAG
- Graph RAG
- agentic RAG
- Humanities Last Exam
- Redin
- Headspace
- Vellum Workflows
- Vellum Workflow SDK

## Who should watch
- You are building an AI workflow and need a practical testing process before exposing it to production traffic.
- Your agent calls tools or APIs and you need to debug decisions, handle failures, and prevent prompt changes from causing regressions.
- You want to understand how a working SEO workflow combines research, generation, evaluation, memory, and iterative feedback.

## Related talks

- [The Agentic AI Engineer](https://aietalks.com/talks/the-agentic-ai-engineer) (Benedikt Sanftl & Burak, Mutagent, 34:50)
- [Agents Building Agents](https://aietalks.com/talks/agents-building-agents) (Alfonso Graziano, Nearform, 30:14)
- [Ship Real Agents: Hands-On Evals for Agentic Applications](https://aietalks.com/talks/ship-real-agents-hands-on-evals-for-agentic-applications) (Laurie Voss, Arize AI, 2:04:18)
- [Building GTM AI Agents: Lessons from Deploying to 6,000 Users](https://aietalks.com/talks/building-gtm-ai-agents-lessons-from-deploying-to-6-000-users) (Sait Izmit, Snowflake, 20:39)
- [Why (Senior) Engineers Struggle to Build AI Agents](https://aietalks.com/talks/why-senior-engineers-struggle-to-build-ai-agents) (Philipp Schmid, Google DeepMind, 10:40)