# Feedback Loops are All You Need Mehedi Hassan, Granola | AI Engineer Europe 2026 | 10:11 Source: https://www.youtube.com/watch?v=ON5LIT0M4do Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/feedback-loops-are-all-you-need Published: 2026-05-10 Tags: cost, developer-experience, observability, testing ## TL;DR - A generic AI chat feature quickly exposes problems with slow search, poor writing style, and incorrect answers. - Granola built internal tracing that shows tool calls, search activity, reasoning, and cost in a UI that product, data, and CX teams can use. - Turning the Electron frontend into a web shell gave every pull request a preview link, allowing Cursor to test changes and upload screenshots automatically. ## Summary Mehedi Hassan describes what happened when Granola added chat to its meeting notes product. A simple implementation worked in a playground, then failed in production: web search consumed context and money, provider updates changed results, and one prompt produced different kinds of summaries poorly for sales, engineering, and HR users. Granola responded by building internal tracing around its AI SDK. The tool records calls, search tools, reasoning, and cost, then presents them in a UI that non-engineers can inspect. Hassan also explains how the team changed its Electron frontend into a web shell. Pull requests now create preview links, so teammates and Cursor can test variants without installing the desktop app. His argument is practical: AI features need short cycles of inspection, testing, and iteration. Better prompting alone cannot provide the feedback needed to understand what failed or decide whether a feature works for users. ## Key ideas ### A one-shot chatbot breaks in ordinary product use [02:03](https://www.youtube.com/watch?v=ON5LIT0M4do&t=123s) Granola's example chat feature answers questions about one meeting, multiple meetings, and shared context. Hassan imagines shipping a one-shot version and then receiving complaints: web search is too slow, follow-up emails do not match the user's style, and a request for meeting coaching produces information about a football coach. These failures come from treating a generic chatbot as a finished product. The feature may work in a demonstration while still missing the details that matter to users in real situations. ### Web search brings cost, context, and provider risks [02:51](https://www.youtube.com/watch?v=ON5LIT0M4do&t=171s) Hassan says adding web search often looks like a single line of code in an LLM provider's interface. In practice, complex searches can increase token use and cost, fill the context, and make each chat cost around 10 pence. At large scale, that is not feasible for Granola. Search quality can also change when a provider ships an update. Granola saw search degrade overnight with no clear explanation and had little choice beyond switching providers. The feature therefore needs its own controls and inspection rather than blind trust in the tool call. ### One prompt cannot match every user's meeting needs [04:01](https://www.youtube.com/watch?v=ON5LIT0M4do&t=241s) The same meeting summary can be useful to one person and wrong for another. Hassan gives sales as an example, where a user may want a deal-focused summary. An engineer may want action items, blockers, or Linear tickets. An HR user may expect a different result. Granola's product philosophy is to stay out of the user's way, so the team needed to understand why the model produced each output and how to shape it for different contexts. Hassan describes LLM behavior as a black box that the team wanted to inspect more closely. ### Internal tracing makes AI failures inspectable by the whole company [04:38](https://www.youtube.com/watch?v=ON5LIT0M4do&t=278s) Granola built its own tracing tools to follow an interaction from the first call to the final output. The interface exposes individual tool calls, the reasons for those calls, search tools, reasoning tools, and cost. The data is structured around the team's needs, and the UI is intended for product, data, CX, and engineering staff. People do not need to write complex CloudWatch queries to find why something failed. Granola saves the records to a database, wraps its AI SDK, and puts most of the effort into the interface. Hassan says the founder uses it to follow the agent loop from front to back. ### A desktop app made testing variants unnecessarily difficult [06:23](https://www.youtube.com/watch?v=ON5LIT0M4do&t=383s) Granola is a desktop app, so only one Electron instance could run at a time. Testing several feature variants in parallel was difficult. A developer had to run the app locally and install dependencies, and a coworker needed to repeat that setup to test the same change. Hassan contrasts this with web applications, where a link can usually provide access to a change. The friction mattered because LLMs made it possible for Granola to try more versions of a feature and present different experiences to different users. ### A web shell gives Electron changes shareable preview links [07:16](https://www.youtube.com/watch?v=ON5LIT0M4do&t=436s) Granola turned the Electron frontend into a web shell and deployed it online. Each pull request now receives a preview link, which lets the team test changes without setting up the desktop app. The team separated Electron's main process from its render process, then abstracted the IPC system APIs so they fall back to web standards in a web environment. They made similar changes for React routers, sessions, and the query layer. The render process became independent of Electron enough to run as a web app. ### Cursor can verify a pull request with a screenshot [07:35](https://www.youtube.com/watch?v=ON5LIT0M4do&t=455s) The preview setup also lets Cursor test a pull request automatically. After a pull request opens, Cursor tests the change and uploads a screenshot into the pull request. Hassan presents this as a simple extension of the preview workflow, rather than a replacement for the team's product judgment. The team can try several variants, see them in practice, and compare the resulting experiences instead of judging only from a Figma design. Automated checks shorten the time before people can inspect the actual feature. ### The product needs a feedback loop around the model [08:48](https://www.youtube.com/watch?v=ON5LIT0M4do&t=528s) Hassan's conclusion is that the answer is not to one-shot better. Granola tries to create a feedback loop that feels like playing tennis with an LLM: inspect what it did, test a change, and send the next iteration back. Tracing helps explain failures inside the AI workflow. Web previews and automated testing help inspect the product experience across variants. This gives the team more confidence that a release connects with users instead of relying on a black box and hoping the feature works. ## Notable quotes - "One prompt can't generally serve everyone." (04:19) - "The most useful part of this is that we structured the data exactly how we want it." (04:58) - "We took our Electron app and we turned the front-end of the Electron app into a web shell." (07:16) - "The answer isn't to one-shot better." (08:48) - "It's about figuring out how you can make that feedback loop where it kind of feels like playing a tennis game with LLM." (08:48) ## Tools & references mentioned - Granola - Electron - Cursor - CloudWatch - OpenTelemetry - AI SDK - Tauri - Linear ## Who should watch - You have an AI feature that works in a demo but produces inconsistent answers, expensive tool calls, or unexplained regressions in production. - Your team ships a desktop app and needs a faster way for developers, product staff, and automated agents to test pull requests. - You want to build observability around model behavior that people outside the engineering team can actually use. ## Related talks - [Turning Fails into Features: Zapier's Hard-Won Eval Lessons](https://aietalks.com/talks/turning-fails-into-features-zapiers-hard-won-eval-lessons) (Rafal Willinski & Vitor Balocco, Zapier, 16:15) - [Building AI Products That Actually Work](https://aietalks.com/talks/building-ai-products-that-actually-work) (Ben Hylak, Raindrop & Sid Bendre, Oleve, 18:42) - [How to build world-class AI products](https://aietalks.com/talks/how-to-build-world-class-ai-products) (Sarah Sachs, Notion & Carlos Esteban, Braintrust, 1:43:46) - [How to Look at Your Data](https://aietalks.com/talks/how-to-look-at-your-data) (Jeff Huber, Chroma & Jason Liu, 567, 19:23) - [BotDojo Launch: Enhancing AI Assistants with Evaluations and Synthetic Data](https://aietalks.com/talks/botdojo-launch-enhancing-ai-assistants-with-evaluations-and-synthetic-data) (Paul Henry, BotDojo, 05:47)