# Analyzing 10,000 Sales Calls With AI In 2 Weeks Charlie Guo, Pulley | AI Engineer World's Fair 2025 | 09:49 Source: https://www.youtube.com/watch?v=dvft0Gp9sEE Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/analyzing-10-000-sales-calls-with-ai-in-2-weeks Published: 2025-06-03 Tags: caching, cost, rag, structured-outputs ## TL;DR - A single AI engineer analyzed 10,000 sales calls in about two weeks, a task that would have taken roughly 625 days of continuous manual work. - Claude 3.5 Sonnet was chosen over cheaper models because its lower hallucination rate made the results usable for customer and company classification. - Prompt caching, extended outputs, retrieval-augmented generation, and structured JSON reduced the analysis cost from $5,000 to $500 and produced results in days instead of weeks. ## Summary Charlie Guo describes how Pulley analyzed 10,000 sales call transcripts to refine its ideal customer profile and find customer patterns. Manual review would have required downloading, reading, classifying, citing, and reporting on every transcript, which he estimates at 625 days of continuous work. Smaller models created false positives, so the team used Claude 3.5 Sonnet and added retrieval-augmented generation, chain-of-thought prompting, structured JSON, and citations linked to source transcripts. Prompt caching cut costs by up to 90% and latency by up to 85%, while extended outputs reduced the number of requests. The analysis cost fell from $5,000 to $500. The resulting system helped marketing and sales, then grew into an internal tool with search, filters, and exports. Guo argues that the work required ordinary software engineering around the model, not just a prompt. ## Key ideas ### Manual review could not handle the volume of customer conversations [00:03](https://www.youtube.com/watch?v=dvft0Gp9sEE&t=3s) Guo begins with the limits of human review. A 30-minute call takes half an hour to listen to and annotate, so an eight-hour day allows 16 calls. Even working every day, a person would reach only 224 calls per week. Reviewing 10,000 transcripts would require downloading each file, checking whether it matched the target persona, finding useful evidence, and writing reports with citations. Guo estimates that process at 625 days of continuous work, nearly two years. The company needed a way to examine thousands of conversations without reducing the work to simple keyword matches. ### The project narrowed a broad customer profile into specific patterns [00:50](https://www.youtube.com/watch?v=dvft0Gp9sEE&t=50s) Pulley's existing ideal customer profile was venture-backed startups. Guo says that description helped product decisions but was too broad for marketing. The team wanted details such as an early-stage, venture-backed crypto startup with a CTO as the relevant buyer. Instead of conducting thousands of new interviews, Guo used the company's existing sales calls, which contained hours of sales representatives speaking directly with prospects. The transcripts offered a large source of customer evidence, but their volume made ordinary research methods impractical. ### Model choice depended on false-positive rates, not just price [03:05](https://www.youtube.com/watch?v=dvft0Gp9sEE&t=185s) The team compared GPT-4o and Claude 3.5 Sonnet with smaller, cheaper models. The cheaper options produced false positives that could damage the analysis. A transcript might be classified as related to crypto because a sales representative mentioned blockchain features in Pulley's product. A model could also label someone as a company founder without evidence in the conversation. Guo says the team accepted the higher cost of the stronger models because an analysis built on untrustworthy classifications would have no value. They ultimately selected Claude 3.5 Sonnet. ### Several layers made the model's conclusions easier to check [04:07](https://www.youtube.com/watch?v=dvft0Gp9sEE&t=247s) The system did more than send a transcript to a model and request an answer. The team enriched the raw transcript with retrieval-augmented generation from third-party and internal sources. They used chain-of-thought prompting to make the outputs more reliable, then requested structured JSON where possible. That structure supported citations and created a trail back to the original transcript. Guo says this combination let the system extract company details and customer insights while giving the team a way to verify where each result came from. ### Caching and longer outputs changed the economics of the analysis [05:06](https://www.youtube.com/watch?v=dvft0Gp9sEE&t=306s) The accuracy work increased token use and costs. Claude 3.5 Sonnet often reached its 4,000-token output limit, which forced multiple requests for one transcript. Prompt caching helped because the same transcript was reused for metadata extraction and insight extraction. Guo reports cost reductions of up to 90% and latency reductions of up to 85%. An experimental extended-output feature also allowed complete summaries in one pass. Together, these changes turned a $5,000 analysis into a $500 analysis and produced results in days rather than weeks. ### The analysis became useful to teams beyond the original report [06:33](https://www.youtube.com/watch?v=dvft0Gp9sEE&t=393s) The project began as an executive request for customer insights, but other teams found uses for the system. Marketing used it to select customers for branding and positioning work. Sales automated transcript downloads and saved dozens of hours each week. Teams also started asking questions that manual research would have made too expensive or difficult. The company built a user experience around the analysis, with search, filters, and exports, so the work became an internal resource rather than a single report. ### AI engineering included database design and product architecture [07:14](https://www.youtube.com/watch?v=dvft0Gp9sEE&t=434s) Guo says the project depended on ordinary engineering decisions alongside model behavior. Structured JSON, database schemas, and the system's overall architecture all produced meaningful gains. The model could not simply be added to an existing process as an afterthought. The team also considered future uses and built a flexible interface instead of stopping after the first analysis. His definition of AI engineering includes designing the surrounding software so that model outputs can be used, searched, exported, and checked. ## Notable quotes - "Two years ago, this would have been impossible, or would have at least required a dedicated team working for several weeks. Today, it's something a single AI engineer can accomplish in about a fortnight." (00:26) - "If we couldn't trust the model's data, the entire project would be pointless." (03:46) - "As a result, we turned a $5,000 analysis into a $500 one. And we got results in days instead of weeks." (06:10) - "AI engineering is knowing how to build effective systems around large language models." (07:56) ## Tools & references mentioned - Pulley - Claude 3.5 Sonnet - GPT-4o - Claude - ChatGPT - Gemini - Artificial Ignorance ## Who should watch - You have a large archive of sales calls, support tickets, reviews, or other customer conversations and need to decide whether an automated analysis is trustworthy enough to use. - Your team is comparing cheaper language models with more capable models and needs to weigh false positives against inference cost. - You are building an LLM application and want practical examples of retrieval, structured outputs, caching, citations, and a user interface around model results. ## Related talks - [Contact Center Voice AI: Low-Latency Intelligence Extraction from Messy Audio Streams](https://aietalks.com/talks/contact-center-voice-ai-low-latency-intelligence-extraction-from-messy-audio) (Dippu Kumar Singh, Fujitsu North America Inc., 22:56) - [200 Million Patient Interactions Later](https://aietalks.com/talks/200-million-patient-interactions-later) (Vivek Muppalla, Hippocratic AI, 20:40) - [How to Look at Your Data](https://aietalks.com/talks/how-to-look-at-your-data) (Jeff Huber, Chroma & Jason Liu, 567, 19:23) - [Building Alice's Brain: An AI Sales Rep that Learns Like a Human](https://aietalks.com/talks/building-alices-brain-an-ai-sales-rep-that-learns-like-a-human) (Sherwood Callaway & Satwik, 11x, 22:18) - [The Billable Hour is Dead; Long Live the Billable Hour](https://aietalks.com/talks/the-billable-hour-is-dead-long-live-the-billable-hour) (Kevin Madura & Mo Bhasin, AlixPartners, 17:04)