# Your Support Team Should Ship Code Lisa Orr, Zapier | AI Engineer CODE 2025 | 16:06 Source: https://www.youtube.com/watch?v=RmJ4rTLV_x4 Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/your-support-team-should-ship-code Published: 2025-12-16 Tags: agents, coding-agents, evals, mcp, workflows ## TL;DR - Zapier moved support from triaging integration bugs to fixing them, with engineering review and guard rails around the work. - Scout Agent combines diagnosis, fixability assessment, code generation, tests, and merge request creation inside the team's existing GitLab workflow. - Scout generates 40% of support's integration fixes, while some support team members have increased from one or two fixes per week to three or four. ## Summary Lisa Orr describes how Zapier handles app erosion, the steady stream of integration failures caused by changes to more than 8,000 third-party APIs. Zapier first empowered support staff to fix bugs under engineering review. Orr's team then studied how engineers and support members worked, finding that gathering context from API documentation, logs, internal systems, and the wider internet consumed much of the effort. An API playground failed because it pulled engineers away from their normal tools. MCP tools worked better inside Cursor, but the most useful diagnosis tool was too slow to run synchronously. Scout Agent became the next step. It categorizes tickets, assesses whether they are fixable, generates a tested merge request, and lets support request changes directly in GitLab. Zapier measures categorization, fixability, and solution quality separately. Scout now generates 40% of support's integration fixes and helps support resolve small, fresh bugs before they become engineering backlog items. ## Key ideas ### Zapier's integrations are constantly eroded by changing APIs [01:07](https://www.youtube.com/watch?v=RmJ4rTLV_x4&t=67s) Zapier has more than 8,000 integrations built on third-party APIs, and those APIs keep changing. Lisa Orr compares this to erosion in the Grand Canyon. Some Zapier apps are 14 years old, so API changes and deprecations create continuing reliability problems. The result was a backlog crisis: support tickets arrived faster than the team could handle them. Orr connects that backlog to integration reliability issues, a poor customer experience, and possible churn. The company responded with two experiments, moving support into bug fixing and testing whether AI could help solve the fixes faster. ### Support began fixing a small, controlled set of integration bugs [02:24](https://www.youtube.com/watch?v=RmJ4rTLV_x4&t=144s) Zapier started by moving support beyond triage so team members could fix integration bugs. Orr says support was already close to these problems, and some people wanted to move into engineering later. The team added guard rails: it focused on four target apps, limited the work to app fixes, and required engineering to review merge requests from support. This made support's new coding responsibility practical without opening every part of the codebase to unreviewed changes. ### The first AI tools focused on gathering context and generating tests [03:38](https://www.youtube.com/watch?v=RmJ4rTLV_x4&t=218s) Orr's team shadowed engineers and support members while they worked on app fixes. They found that a large amount of time went into collecting context from third-party API documentation, internet searches, internal logs, and existing information about similar bugs. The team built APIs for separate pain points. A diagnosis tool used an LLM to gather and curate context, while a test case finder used search to locate relevant tests. These tools addressed pieces of the process, but they did not initially fit the way engineers worked. ### Tools only gained usage after moving into existing developer workflows [05:37](https://www.youtube.com/watch?v=RmJ4rTLV_x4&t=337s) The first interface was an internal web playground called Autocode. Engineers had to open another page and choose among many APIs, so engagement was low. Cursor also gained adoption at Zapier, which made some of the team's standalone tools less necessary. Diagnosis was the exception. Support liked it enough to embed it in the Zap that creates a Jira ticket from a support issue. Orr says this showed that embedding tools into an existing process was necessary for usage. MCP then allowed the team to put the tools into engineers' IDE workflows, especially Cursor. ### Slow synchronous diagnosis led Zapier to orchestrate the tools [08:02](https://www.youtube.com/watch?v=RmJ4rTLV_x4&t=482s) MCP improved access, but diagnosis still took too long to run while an engineer was working synchronously on a ticket. Adoption was also scattered across the tool suite, with engineers using only some of the available tools. Orr's team decided that the value would come from connecting the tools rather than asking engineers to choose and sequence them. Scout Agent runs diagnosis on a ticket, uses the result to start code generation, and produces a merge request with the relevant context. ### Scout Agent gives support a tested merge request before the first pickup [10:12](https://www.youtube.com/watch?v=RmJ4rTLV_x4&t=612s) Scout Agent was aimed first at support because support handles small, emerging bugs that arrive fresh from the queue. The workflow starts when support submits an issue. Scout categorizes it, assesses whether it is fixable, and generates a merge request when appropriate. Support then reviews and tests the proposed fix. If it needs changes, the team can request an adjustment in GitLab, where Scout runs another pass and posts an updated merge request. Support submits the result to engineering for review. ### GitLab and asynchronous pipelines removed context switching [11:18](https://www.youtube.com/watch?v=RmJ4rTLV_x4&t=678s) Zapier runs Scout from Zaps embedded in the support team's process. Diagnosis posts its result to Jira, and fixable tickets start a GitLab CI/CD pipeline with plan, execute, and validate phases. The pipeline uses Scout MCP tools and the Cursor SDK. After the merge request is attached to Jira, support can inspect it. The newer rapid-iteration flow lets support chat with Scout inside GitLab when a fix needs tweaks. That feedback starts another pipeline, so support does not have to pull the code into an IDE and push it back manually. ### Separate failure measures show where Scout needs improvement [12:28](https://www.youtube.com/watch?v=RmJ4rTLV_x4&t=748s) Orr evaluates Scout with three questions: did it categorize the issue correctly, did it assess fixability correctly, and did the generated code work? She reports 75% accuracy for categorization and fixability. As the team processes more tickets and receives more feedback, those tickets become test cases for improving the agent. This separates the decision to attempt a ticket from the decision that a code fix is needed and from the quality of the final solution. ### Fresh customer context makes support effective at validating generated fixes [13:18](https://www.youtube.com/watch?v=RmJ4rTLV_x4&t=798s) Scout generates 40% of support's app fixes. For some support team members, output has risen from one or two fixes per week to three or four with Scout. Orr says support has three advantages: it is closest to customer pain, it troubleshoots while logs and context are fresh, and it is best positioned to validate whether a fix addresses the customer's actual problem. Engineering can then focus on more complex work. Orr also says support members involved in the experiment have since become engineers. ## Notable quotes - "Embedding tools is the key to usage as we find out." (07:06) - "So what if we owned orchestration of these tools rather than saying here's a suite of tools you use them as you wish what if we combined them and created an agent to orchestrate this." (09:02) - "40% of supports support teams app fixes are being generated by scout." (13:18) - "They are the closest to customer pain which mean they're closest to the context that really matters for figuring out what's the problem and how to solve it." (14:22) ## Tools & references mentioned - Zapier - Scout Agent - Scout MCP - Cursor - Cursor SDK - GitLab - GitLab CI/CD - Jira - Autocode - Grand Canyon ## Who should watch - You maintain integrations against third-party APIs and need a way to handle a growing stream of small compatibility bugs. - Your support team has strong product context but cannot safely change code without engineering review and workflow support. - You are building coding agents and need to think about context gathering, human validation, asynchronous execution, and separate evaluation measures. ## Related talks - [AI Agents for Performance: Ship Faster, Pay Less](https://aietalks.com/talks/ai-agents-for-performance-ship-faster-pay-less) (Rajat Shah, Netflix, 33:39) - [Agents, codebases, and teams](https://aietalks.com/talks/agents-codebases-and-teams) (Aditya Khandelwal, Amazon AGI Lab, 16:57) - [Making Codebases Agent Ready](https://aietalks.com/talks/making-codebases-agent-ready) (Eno Reyes, Factory AI, 15:33) - [The Agent Awakens: Collaborative Development with Copilot](https://aietalks.com/talks/the-agent-awakens-collaborative-development-with-copilot) (Christopher Harrison & John Peck, GitHub, 1:04:06) - [How Coding Agents Change Software Development Forever](https://aietalks.com/talks/how-coding-agents-change-software-development-forever) (Hailong Zhang, 08:50)