# How Zapier Builds AI Products and Features with the Help of Braintrust

Ankur Goyal, Braintrust & Olmo Maldonado, Zapier | AI Engineer World's Fair 2024 | 14:59

Source: https://www.youtube.com/watch?v=fOtTHWeU6B8
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-zapier-builds-ai-products-and-features-with-the-help-of-braintrust
Published: 2024-11-07
Tags: evals, observability, product-strategy, testing

## TL;DR
- Zapier uses evals as part of product decisions, with product managers involved alongside engineers.
- The AI Zap Builder grew from seven manually run unit tests to more than 800 continuously run tests, improving accuracy by nearly 300%.
- Tracing and eval comparisons helped Zapier diagnose regressions during its move between OpenAI models and reduce Copilot response time from about 14 seconds to 3 seconds for streaming responses.

## Summary
Olmo Maldonado explains how Zapier built the AI Zap Builder and Zapier Copilot around continuous evaluation and observability. The team starts with prototypes, releases them to users, and uses eval results to decide what to change. For the AI Zap Builder, product managers helped define priority cases such as valid triggers, actions, step order, support for the top 25 apps, paths, and filters. Synthetic data from a corporate account feeds an evaluation suite that runs in CI and manually, with logic-based and LLM-based graders. Coverage grew from seven manually run unit tests to more than 800 tests, and accuracy improved by nearly 300%. For Copilot, Braintrust tracing exposed the inputs, outputs, tokens, timing, and tool calls inside the agent. This helped the team investigate regressions after changing models, revise prompts and tool settings, adopt GPT-4o, and reduce streaming response time from about 14 seconds to 3 seconds.

## Key ideas
### Zapier releases prototypes early and uses evals to decide what to change
[04:00](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=240s)
Olmo says Zapier tries to get a prototype in front of users as early as possible. The team expects to get some things wrong, then learns and improves through repeated iterations. Evals became the way the team makes product decisions. This replaced an earlier process that did not use evaluation results as the basis for changes. The approach applies to AI products such as the AI Zap Builder, where a user provides a prompt and Zapier generates a Zap, including field mapping and other configuration.

### The AI Zap Builder eval suite includes product requirements as well as engineering checks
[04:43](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=283s)
The AI Zap Builder has to work across more than 7,000 apps and a large range of integrations. Olmo says product managers need to be part of the evaluation discussion because this is also a product problem. The priority cases included correct triggers and actions, correct step order, support for the top 25 apps, and proper handling of internal Zapier features such as Paths and Filters. Those requirements were turned into cases in the eval suite instead of being left as informal expectations.

### Synthetic account data and custom graders make the evals repeatable
[05:39](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=339s)
Zapier built an evaluation framework with Braintrust and seeded it with synthetic data from a corporate account. The data is loaded from Braintrust and run in CI as well as manually. Zapier has its own runner that tests the AI providers each time the suite runs. The team uses both logic-based graders and LLM-based graders. Results are reported across runs, so the team can act on failures and check whether the criteria defined by product and engineering are being met.

### Continuous evaluation expanded test coverage and improved AI Zap Builder accuracy
[07:51](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=471s)
Before building the eval suite, Zapier had seven unit tests that developers ran manually. The new process grew that number to more than 800 tests, which run as part of merge requests and on a continuous basis. The team receives alerts when a regression appears. Olmo reports that this process improved accuracy by nearly 300%, while also saying the product was not at 100% accuracy and still required more work.

### Copilot tracing exposes the internal path of a multi-tool request
[09:35](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=575s)
The first single-shot AI Zap Builder could only go so far, so Zapier moved toward a chat interface that lets users iteratively configure steps and fields. That created a more complex agent system with multiple tool calls. The team needed to see which part of a request was responsible for problems. Braintrust tracing let developers inspect chat-completion inputs and outputs, token counts, response time, and other details at a granular level. This made it easier to understand Copilot performance and iterate on the implementation.

### Eval comparisons exposed model regressions that manual testing had missed
[10:32](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=632s)
Zapier first used GPT-3.5 Turbo for Copilot, then tested other models and selected GPT-4 Turbo for its message router. The change improved the guarantees Zapier could provide around accuracy, but it made the system slower. Later, the team switched to GPT-4o and saw most evaluation scores fall below 80%, whereas they had previously been 80% or higher. Comparing runs showed 22 regressions connected to the model forgetting parts of the system prompt or producing unwanted answers.

### Prompt and tool-choice changes recovered scores after the GPT-4o switch
[12:17](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=737s)
The regressions showed that some prompts had been tuned too closely to GPT-3.5 Turbo. Zapier changed its prompt engineering by relaxing instructions that had been overly elaborate for the earlier model. The team also changed its use of OpenAI tool settings, moving away from the older functions approach and experimenting with making tool choice required. After these changes, most scores recovered. Olmo describes the process as fast trial and error supported by comparisons with previous runs.

### Observability helped Zapier trade model cost and speed against accuracy
[14:01](https://www.youtube.com/watch?v=fOtTHWeU6B8&t=841s)
After the prompt and tool changes, Zapier adopted GPT-4o while continuing to work on its remaining issues. Olmo reports that the streaming Copilot response time fell from about 14 seconds before the adoption to about 3 seconds afterward. The team could compare current and earlier runs, inspect individual examples, and see where scores moved. The talk presents this as an ongoing evaluation-driven development process rather than a finished accuracy result.

## Notable quotes
- "We want to get the Prototype as early as possible to the user." (04:00)
- "This is an engineering problem as well as a product problem." (05:06)
- "Before this we just had seven unit tests that were run manually by devs and now we have over 800 of them." (07:51)
- "We were around 14 seconds and now we're at 3 seconds for a stream based co-pilot." (14:01)

## Tools & references mentioned
- Zapier
- Braintrust
- AI Zap Builder
- Zapier Copilot
- GPT-3.5 Turbo
- GPT-4 Turbo
- GPT-4o
- OpenAI
- Notion
- Airtable
- Instacart
- Vercel
- Zapier Central
- Paths
- Filters

## Who should watch
- You are building an AI feature that must work across many integrations and need product requirements translated into repeatable eval cases.
- Your agent uses several tools and you need traces that show inputs, outputs, timing, tokens, and the path of each request.
- You are changing models or prompts and need to detect regressions before they reach users.

## Related talks

- [Turning Fails into Features: Zapier's Hard-Won Eval Lessons](https://aietalks.com/talks/turning-fails-into-features-zapiers-hard-won-eval-lessons) (Rafal Willinski & Vitor Balocco, Zapier, 16:15)
- [Evals](https://aietalks.com/talks/evals) (Omar Khattab & Vitor Balogh & Rafal Wilinski, Zapier & Ido Pesal, Vercel & Randall, Bolt Foundry & Diego Rodriguez, Krea & Doug Guthrie & Ankur Goyal, Braintrust & Jeff Huber, Chroma & Jason Liu, 567 Studio & John Dickerson, Mozilla AI, 3:59:12)
- [How to build world-class AI products](https://aietalks.com/talks/how-to-build-world-class-ai-products) (Sarah Sachs, Notion & Carlos Esteban, Braintrust, 1:43:46)
- [The Future of Evals](https://aietalks.com/talks/the-future-of-evals) (Ankur Goyal, Braintrust, 05:14)
- [How to Look at Your Data](https://aietalks.com/talks/how-to-look-at-your-data) (Jeff Huber, Chroma & Jason Liu, 567, 19:23)