# Small Bets, Big Impact: Building GenBI at a Fortune 100 Asaf Bord, Northwestern Mutual | AI Engineer | 22:50 Source: https://www.youtube.com/watch?v=LU9KgcZDRfY Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/small-bets-big-impact-building-genbi-at-a-fortune-100 Published: 2025-12-23 Tags: data-pipelines, enterprise, human-in-the-loop, rag ## TL;DR - Northwestern Mutual built GenBI through small, fundable research phases instead of asking leadership to approve one large uncertain investment. - The system starts with certified reports and metadata, then adds SQL generation only when an existing report cannot answer the question. - Each six-week phase produces a business deliverable, measurable learning, and a decision point where leadership can stop funding. ## Summary Asaf Bord explains how Northwestern Mutual is developing GenBI, an LLM-powered analytics copilot, inside a company that sells long-term financial commitments and therefore has little tolerance for instability. The team used real, messy enterprise data and worked with BI practitioners from the start. That gave them domain examples for evaluation and helped future users support the work. To gain leadership support, they split the effort into six-week phases. Early phases focused on natural-language questions, metadata, report discovery, and light data pivoting. Each phase could become a product and could be stopped independently. The architecture uses separate metadata, retrieval, SQL, and BI agents, with governance and orchestration around them. The team began with certified reports rather than generated SQL, which reduced the risk of invented answers. Bord also reports measurable benefits, including automating about 80% of report-finding work that occupied roughly two BI staff members. ## Key ideas ### GenBI gives business users direct access to answers grounded in company data [01:23](https://www.youtube.com/watch?v=LU9KgcZDRfY&t=83s) Bord defines GenBI as a fusion of generative AI and business intelligence. It is an agent that helps people answer business questions with data, in the way a BI specialist might. The goal is data democratization. A user should not have to wait for a BI team to find a report, explain what it means, and understand the user's situation before offering help. Northwestern Mutual has extensive data, funding, use cases, and technical talent, but its long-term life insurance and wealth management commitments make stability important. Bord frames the work as a way to introduce innovation without ignoring that responsibility. ### Real messy data exposes production problems and brings users into the research [04:56](https://www.youtube.com/watch?v=LU9KgcZDRfY&t=296s) The team chose actual enterprise data instead of synthetic or cleaned datasets because the difficult parts of production are found in messy data. Bord says the gap from a proof of concept to production is broad, especially when the team does not yet know how the system should behave. Working with real data also put the team alongside people who use it every day. Those subject matter experts supplied real questions, answers, and evaluation cases. Their involvement changed adoption as well. When the system became mature enough for production, users were already asking how to package it for practical use. ### Trust grows when access expands in stages that match user expertise [06:54](https://www.youtube.com/watch?v=LU9KgcZDRfY&t=414s) Northwestern Mutual did not release the tool to everyone at once. The first users were BI experts who could check answers and recognize errors, much like using GitHub Copilot to speed up work they already know how to do. Business managers were the next group because they were familiar enough with the data to spot problems and provide useful feedback. Executives were left for later because they need concise answers they can trust, and Bord says the system is not accurate enough for that audience yet. The team also worked in a sandbox and kept client data out of the research environment. ### Certified reports provide a safer first step than generating SQL from scratch [08:49](https://www.youtube.com/watch?v=LU9KgcZDRfY&t=529s) The initial system did not try to write SQL for every question. Bord says schemas are messy and context is hard to infer, so the team first used information already verified in the ecosystem. BI teams told them that much of their work involved sending people to the right certified report and helping them use it. The system therefore retrieves approved reports instead of making up information, then presents the same asset in a faster, interactive way. This approach set clear expectations with users and management. SQL generation remains a later capability for questions that existing reports cannot answer. ### Small independent phases make an uncertain AI project easier to fund [09:46](https://www.youtube.com/watch?v=LU9KgcZDRfY&t=586s) Bord's main funding approach was a gradual process with visibility, control, and business outcomes throughout. Leadership could see what each phase funded, receive a deliverable, and stop before committing to the next uncertain step. The first phase covered questions, natural language to SQL, and response writing. The second studied metadata and context for a BI agent. Later phases added semantic search, report and data-owner discovery, light pivoting, user roles, and eventually a full agent that could run SQL and join data. Each phase fed the next while remaining useful on its own. ### A modular agent architecture lets each research result become a product [15:57](https://www.youtube.com/watch?v=LU9KgcZDRfY&t=957s) The architecture has a data and metadata layer, then four agents. The metadata agent interprets the question using the catalog and documentation. The retrieval agent finds an approved report. The SQL agent creates a more exact query when no report is suitable, or expands a useful report query instead of starting from nothing. The BI agent turns the resulting information into a business answer. An orchestrator controls the flow, while governance, trust controls, and a contextual interface sit around it. Bord says the agents can be packaged separately, so the team can stop after delivering a useful component or continue toward the full system. ### The incremental work produced measurable value before a full GenBI agent existed [18:43](https://www.youtube.com/watch?v=LU9KgcZDRfY&t=1123s) The report retrieval work addressed about 20% of the BI team's capacity, with a team of 10 people. Bord says the system automated around 80% of that report-finding work, roughly the equivalent of two full-time roles. Metadata research also supported an A/B test in a semantic-layer project. The team compared questions run against a database with good metadata and one without it, showing leadership that metadata enrichment changed model performance. A data-pivoting bot is being tested to let stakeholders change time horizons, views, segmentations, and groupings without asking a person to do it. These results helped leadership keep funding the work quarter over quarter. ### Research gives the company options even if it later buys a vendor product [14:28](https://www.youtube.com/watch?v=LU9KgcZDRfY&t=868s) The staged approach reduces pressure from sunk costs and makes it easier to stop if an outside product becomes a better choice. Bord names Databricks Genie as an example of a third-party GenBI solution that may improve over time. By doing its own research, Northwestern Mutual can build internal benchmarks, understand what good performance looks like, and ask vendors more precise questions. The company also gains metadata and semantic-layer knowledge that remains useful even if it does not finish an end-to-end internal agent. Bord's broader prediction is that GenAI will change software pricing as individual users become more productive, pushing some products from seat-based pricing toward usage or value-based models. ## Notable quotes - "It's basically an agent that helps people answer business questions with data like a business intelligence person would do in real life." (01:23) - "The end users were part of the research process itself." (06:15) - "Each one of this was a six week sprint at the end of which we had a very tangible deliverable coming back to the business that we could decide to productize." (13:31) - "We're just going to deliver you the same asset that you would have gotten anyway just in a much faster much more interactive way." (09:45) - "We already got a lot of value back from this and this is really what allowed our senior leadership team to continuously invest in this project quarter over quarter." (20:28) ## Tools & references mentioned - GPT-3 - GPT-5 - GitHub Copilot - Databricks Genie - Salesforce - Salesforce Data Cloud ## Who should watch - You are trying to fund an uncertain GenAI project inside a large company where leadership needs a stop point before approving the next phase. - Your analytics team spends time finding reports, explaining metadata, or answering repeat questions, and you want to turn that work into a controlled product. - You need an architecture that can start with retrieval and certified data before taking on generated SQL and more complex questions. ## Related talks - [POC to PROD: Hard Lessons from 200+ Enterprise GenAI Deployments](https://aietalks.com/talks/poc-to-prod-hard-lessons-from-200-enterprise-genai-deployments) (Randall Hunt, Caylent, 19:16) - [Anchoring Enterprise GenAI with Knowledge Graphs](https://aietalks.com/talks/anchoring-enterprise-genai-with-knowledge-graphs) (Jonathan Lowe, Pfizer & Stephen Chin, Neo4j, 20:59) - [Small AI Teams with Huge Impact](https://aietalks.com/talks/small-ai-teams-with-huge-impact) (Vikas Paruchuri, Datalab, 17:36) - [Tiny Teams](https://aietalks.com/talks/tiny-teams) (Britney Walker, CRV & Eric Simons, StackBlitz & Sid Bendre, Oliv & Max Broer Herbas, Gum Loop & Grant, Gamma & Vic Paruturi, Data Lab & Alex Duffy, Every, 3:37:25) - [The AI Pivot](https://aietalks.com/talks/the-ai-pivot) (Brittany Walker, CRV & Chris White, Prefect & Bryan Bischof, Hex, 35:16)