# We Built an AI Support Agent That Resolves 80% of Tickets Matt Lawler, AssemblyAI | AI Engineer World's Fair 2026 | 16:19 Source: https://www.youtube.com/watch?v=pyvRID_CZZU Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/we-built-an-ai-support-agent-that-resolves-80-of-tickets Published: 2026-10-04 Tags: chatbots, deployment, tool-use, voice ## TL;DR - AssemblyAI replaced an off-the-shelf support bot that resolved 10% of conversations with Joey, an agent that resolves 80% end to end. - Joey uses local Markdown documentation, Voyage embeddings, agentic file search, Claude Agent SDK instructions, and Railway deployment. - Matt argues that forward deployed engineers should automate routine support work and build the same products their customers are building. ## Summary Matt Lawler explains how AssemblyAI built Joey to handle support and onboarding at a company receiving around 1,000 API signups each day. The company's off-the-shelf support bot resolved about 10% of conversations, and the team could not change its prompts, tools, or retrieval system. Joey uses documentation checked out as local Markdown, Voyage embeddings, agentic file search, a large CLAUDE.md file, and deployment on Railway. The team can ship fixes in about 30 seconds. Joey reached an 80% end-to-end resolution rate in its first week at about $700 per month, with humans handling the remaining escalations. Lawler also added AssemblyAI's Voice Agent API, connecting speech-to-text, an LLM, and text-to-speech through one WebSocket. The project gives AssemblyAI a support system while forcing its engineers to use the same voice technology that customers build with. ## Key ideas ### A forward deployed engineer cannot personally support 1,000 daily signups [02:08](https://www.youtube.com/watch?v=pyvRID_CZZU&t=128s) AssemblyAI was seeing around 1,000 API signups every day, while Matt Lawler was, until shortly before the talk, the only onboarding engineer. Forward deployed engineers work closely with customers, often committing code to their repositories and meeting several times a month. That model can produce a strong customer relationship, but it cannot cover every inbound customer at this volume. Lawler says the team needs to reserve hands-on work for human-led deals instead of hiring more people to handle every routine question. ### The team chose to automate the FDE role rather than only deflect tickets [03:57](https://www.youtube.com/watch?v=pyvRID_CZZU&t=237s) Lawler tells FDEs to use their customer knowledge to automate themselves out of a job. AssemblyAI first bought an off-the-shelf support bot and pointed it at the documentation. It resolved only about 10% of conversations. With 1,000 conversations, that would remove roughly 100 tickets while leaving the team with 900. The team also lacked access to the bot's system prompt, tools, and retrieval infrastructure, so fixes depended on the vendor's roadmap. They decided to build Joey instead. ### Joey behaves more like an engineer because it can use files, tools, and code [05:17](https://www.youtube.com/watch?v=pyvRID_CZZU&t=317s) Joey is the first line for conversations from AssemblyAI's chat widget, sales contact form, and support alias. Built on the Claude Agent SDK, it has a file system, can write and debug code, call tools, manage infrastructure, and give itself new abilities. Pylon tracks its support conversations and lets Joey remember who customers are. Lawler describes this as closer to an FDE than a standard website chatbot. ### Local Markdown gives Joey current documentation even when the docs site is unavailable [06:12](https://www.youtube.com/watch?v=pyvRID_CZZU&t=372s) AssemblyAI checks its documentation out as local Markdown files and keeps them updated when the documentation system changes. Joey can use those files to answer questions about new releases. Lawler says customers could still get current answers from Joey if the public documentation site went down. Voyage embeddings put relevant documents in front of the agent, while agentic file search lets Joey search across the local files when it needs to combine several resources. ### Fast deployment lets the team fix a live support problem during a conversation [07:22](https://www.youtube.com/watch?v=pyvRID_CZZU&t=442s) Joey runs on Railway because the team did not want to manage EC2 instances or change infrastructure manually. When the team finds a bad conversation in Slack, it can write a pull request and deploy a new version in about 30 seconds. Lawler says they have monitored a live conversation, found a bug, deployed a fix, and had the corrected behavior take effect during the same session. The team controls Joey's behavior and can update it whenever needed. ### Joey reached 80% end-to-end resolution at about $700 per month [08:02](https://www.youtube.com/watch?v=pyvRID_CZZU&t=482s) In the first week after deployment, Joey moved AssemblyAI from a 10% to an 80% end-to-end resolution rate with what Lawler calls a fairly naive implementation. Token and infrastructure costs were about $700 per month. Joey handles all inbound tickets first, and only 20% escalate to a human. Those cases include rate changes, data opt-outs, agreements, and other requests that still need an FDE or legal involvement. ### Every escalation becomes a concrete improvement to Joey [09:01](https://www.youtube.com/watch?v=pyvRID_CZZU&t=541s) Lawler treats unresolved requests as a list of work for the team. If Joey cannot send a BAA, the team can give it a link to the right resource. If it cannot discuss pricing, the team can let customers negotiate with Joey, including quoting a rate based on the customer's requirements. The agent is therefore connected to the team's product and support roadmap. Its limitations show where a new tool, permission, or instruction could remove another human escalation. ### Adding voice makes the support agent a product dogfood test [09:31](https://www.youtube.com/watch?v=pyvRID_CZZU&t=571s) AssemblyAI added its Voice Agent API to Joey so customers can speak to it instead of using text. The API connects speech-to-text, an LLM, and text-to-speech through one WebSocket. It handles real-time latency, pauses, interruptions, and barge-in, and includes voices without requiring another provider. Lawler says building this exposed the same latency, turn-taking, and interruption issues that AssemblyAI customers face, giving him more practical advice for their implementations. ### FDEs should build the product their customers are building [14:24](https://www.youtube.com/watch?v=pyvRID_CZZU&t=864s) Lawler says teaching customers is not enough for an FDE. Engineers should build and ship the same product themselves, including new features they expect customers to use. By building Joey as a voice agent, he had to work through the same problems customers encounter with the Voice Agent API. He argues that this creates better empathy and better technical guidance, while the resulting internal product can also improve customer support. ## Notable quotes - "We can't be the bottleneck to having a good customer experience." (04:03) - "So, we went from 10% to 80% end-to-end resolution rate in just the first week of deploying this build with a pretty naive implementation." (08:08) - "The real thing that we wanted to test out by building Joey was we wanted to dog food our own product, too." (09:19) - "If you're an FDE at your company, you need to be dog fooding your own product all the time." (14:32) ## Tools & references mentioned - AssemblyAI - Claude Agent SDK - Pylon - Voyage - Railway - EC2 - Voice Agent API - CLAUDE.md ## Who should watch - You run a support or onboarding team that is overwhelmed by repetitive API questions and need an example of an agent your team can control and update. - Your company has forward deployed engineers, and you want to decide which customer work should stay human-led and which work can be automated. - You are building a voice agent and want to see an example that combines documentation retrieval, support workflows, and real-time speech. ## Related talks - [Dispatch from the Future: Building an AI-Native Company](https://aietalks.com/talks/dispatch-from-the-future-building-an-ai-native-company) (Dan Shipper, Every, 17:58) - [Building an Autonomous Engineering Org](https://aietalks.com/talks/building-an-autonomous-engineering-org) (Angie Jones, Agentic AI Foundation, 17:36) - [Building Alice's Brain: An AI Sales Rep that Learns Like a Human](https://aietalks.com/talks/building-alices-brain-an-ai-sales-rep-that-learns-like-a-human) (Sherwood Callaway & Satwik, 11x, 22:18) - [Agents Don't Do Standups: Building the Post-Engineer Engineering Org](https://aietalks.com/talks/agents-dont-do-standups-building-the-post-engineer-engineering-org) (Mike Spitz, PFF, 17:50) - [Fighting AI with AI](https://aietalks.com/talks/fighting-ai-with-ai) (Lawrence Jones, Incident.io, 17:29)