# Why Your Product Needs an AI Product Manager, and Why It Should Be You

James Lowe, Incubator for AI, UK Government | AI Engineer World's Fair 2025 | 18:37

Source: https://www.youtube.com/watch?v=xzJdSi2Tsqw
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/why-your-product-needs-an-ai-product-manager-and-why-it-should-be-you
Published: 2025-07-28
Tags: design, evals, product-strategy

## TL;DR
- AI products add uncertainty around feasibility, evaluation, data, safeguards, and how probabilistic behaviour will work for users.
- Teams should evaluate the AI capability early with real users, try a wide range of features, then simplify around what works.
- Technical people can take on the AI product manager mindset because the role depends on understanding both product decisions and AI limits.

## Summary
James Lowe argues that AI products need someone to connect user needs, business constraints, technical feasibility, and the limits of AI. He calls this an AI product manager mindset rather than a fixed team role, and says technically strong people are well placed to take it on. He uses three UK Government projects to explain the approach. Consult failed when the team built around existing language-processing methods before testing their accuracy, then improved after creating evaluations with real and synthetic data. Minute showed the value of trying many features before removing the ones users did not need. Redbox changed direction several times as user demand, Microsoft Copilot Chat, and the Model Context Protocol changed the available options. Lowe is direct about the pace of change: teams must test AI early, simplify after experimentation, and change direction when the original product no longer makes sense.

## Key ideas
### AI product management combines product judgment with AI expertise
[02:11](https://www.youtube.com/watch?v=xzJdSi2Tsqw&t=131s)
Lowe describes product management as balancing business viability, technical feasibility, and user desirability. AI adds further questions: how much experimentation can the business accept, how should the system be evaluated and monitored, and how should a team handle probabilistic behaviour, safeguards, and human review? The AI product manager must also ask whether the proposed system is possible at all. Lowe says existing product skills remain useful, but data and AI proficiency matter more. He presents the role as a mindset that someone on the team must adopt, rather than a job title that every team must add.

### Evaluating the AI capability early can change the product itself
[05:08](https://www.youtube.com/watch?v=xzJdSi2Tsqw&t=308s)
The Consult project began as an attempt to analyze large government consultation responses. The team first built around existing techniques such as BERTopic because there was pressure to deliver. Testing with real users showed inaccurate and inconsistent results that did not meet user needs or the legal threshold. The team returned to the AI capability, collected real data, generated synthetic data, and created an evaluation to optimize against. They tested outputs with users and developed the work into Themefinder, an open-source package. Lowe says this made the output comparable to human work, while being 1,000 times faster and 400 times cheaper. It also revealed where human input mattered.

### Wide feature experiments should end with a simpler product
[08:11](https://www.youtube.com/watch?v=xzJdSi2Tsqw&t=491s)
For Minute, a secure government transcription and summarization tool, the team did not need to invent the underlying transcription service because AWS and Azure already offered options. The uncertainty was how to create a useful experience for different users. The team tried many features with different groups, using AI coding assistance to build them quickly. An early version included templates, agenda input, AI editing, AI chat, speaker-name prediction, and transcript citations. Users found it complicated and often ignored the features. The team then focused on probation services and removed options that group did not need, producing the simpler Justice Transcribe product with Justice AI in the Ministry of Justice.

### Real users can turn a hackathon idea into a different product
[12:36](https://www.youtube.com/watch?v=xzJdSi2Tsqw&t=756s)
Redbox began as a digital version of the paperwork carried by government ministers. User testing showed that people cared most about securely chatting with a large language model, because they knew the value of ChatGPT but could not put work information into it. The second version therefore focused on secure chat for civil servants. After launching it in the Cabinet Office, the team gained thousands of users within weeks. That adoption suggested Redbox could also become the interface for other Incubator for AI tools, including Parlex, which made parliamentary and legislative data easier to access.

### Changing technology can require a product to pivot again
[14:49](https://www.youtube.com/watch?v=xzJdSi2Tsqw&t=889s)
Redbox changed direction when Microsoft announced that Copilot Chat would be free for enterprise Microsoft users, including much of the government, and when Claude's Model Context Protocol provided a standard way to connect tools and data to models. Redbox no longer needed to be the main route to secure model chat or the only route to the team's tools. The team instead invested in using MCP to make its tools and data available through different clients, including Redbox, Copilot Chat, and enterprise offerings from Anthropic and OpenAI. Lowe says Redbox remains useful, but teams must change when the commercial and technical conditions change.

### AI makes familiar product practices faster and more uncertain
[16:53](https://www.youtube.com/watch?v=xzJdSi2Tsqw&t=1013s)
Lowe acknowledges that resolving the largest uncertainties first, listening to users, and testing features are established product practices. He argues that AI changes their practical weight. AI systems need more experimentation and evaluation because their behaviour is probabilistic. AI coding tools make it faster to try features and reduce attachment to features that should be removed. The features themselves are newer, so teams have less prior knowledge about what users will value. The surrounding technology also changes quickly, which makes product pivots more frequent and more urgent.

## Notable quotes
- "An AI product manager has to resolve all of these different areas to find the right path forward." (03:31)
- "Resolve AI uncertainties early on with evaluations and tests with real users." (07:25)
- "Experiment hard and go wide with lots of features." (11:58)
- "You'll have to pivot harder and faster than ever before." (16:08)

## Tools & references mentioned
- Andrew Ng
- Brett Taylor
- Incubator for AI
- Consult
- Themefinder
- Minute
- Justice Transcribe
- Justice AI
- Ministry of Justice
- Redbox
- Evidence House
- Parlex
- AWS
- Azure
- Microsoft Copilot Chat
- Claude
- Model Context Protocol
- ChatGPT
- BBC

## Who should watch
- You are building an AI demo and need a way to decide whether it can become a useful product.
- You work in product management and need to understand how evaluation, probabilistic behaviour, and human review change product decisions.
- You are an engineer or founder who may need to take ownership of product direction as AI capabilities and available platforms change.

## Related talks

- [The AI Pivot](https://aietalks.com/talks/the-ai-pivot) (Brittany Walker, CRV & Chris White, Prefect & Bryan Bischof, Hex, 35:16)
- [The #1 Question Every AI Product Manager Must Answer](https://aietalks.com/talks/the-1-question-every-ai-product-manager-must-answer) (Brian Balfour, 00:56)
- [Shipping Products When You Don't Know What They Can Do](https://aietalks.com/talks/shipping-products-when-you-dont-know-what-they-can-do) (Ben Stein, Teammates, 19:34)
- [Hiring & Building an AI Engineering Team](https://aietalks.com/talks/hiring-building-an-ai-engineering-team) (Dr. Bryan Bischof, Hex, 29:07)
- [Shipping AI That Works: An Evaluation Framework for PMs](https://aietalks.com/talks/shipping-ai-that-works-an-evaluation-framework-for-pms) (Aman Khan, Arize, 1:26:16)