# Shipping Products When You Don't Know What They Can Do

Ben Stein, Teammates | AI Engineer World's Fair 2025 | 19:34

Source: https://www.youtube.com/watch?v=PthmdT92qNg
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/shipping-products-when-you-dont-know-what-they-can-do
Published: 2025-07-28
Tags: design, evals, product-strategy, testing

## TL;DR
- AI products have a wide, unpredictable surface area because their underlying models and user expectations are difficult to define in advance.
- Product teams should specify affordances such as communicating, emailing, and collaborating, then discover which behaviors emerge from those building blocks.
- Evals, prototypes, experiments, and direct customer collaboration give teams practical ways to define and build trust in probabilistic products.

## Summary
Ben Stein describes product management for AI agents as a discipline built around uncertainty. A customer asked whether a Teammates agent could be tagged in a Google Docs comment. The team had never designed that behavior, yet the agent handled it well. Stein argues that this happens because LLM-based products have an unknown foundation and invite users to try almost anything through open-ended interfaces. Product teams should write specs around affordances, such as communicating or collaborating, instead of attempting to enumerate every feature. Evals become both tests and evolving product specifications. Vibe-coded prototypes help teams feel an interaction before committing it to production. Testing also means probing for unexpected behavior and deciding whether a failure is a bug, a feature, or an acceptable probabilistic result. With customers, Stein recommends saying that the future is being invented together, because neither a confident roadmap nor a detailed explanation of current behavior is always honest.

## Key ideas
### AI agents create a product surface that teams cannot fully define
[02:40](https://www.youtube.com/watch?v=PthmdT92qNg&t=160s)
Stein says two conditions make AI product management different. Teams cannot know everything an LLM knows, in the way they understand the interface and limits of a database. At the same time, a free-text interface invites customers to try almost anything beyond the intended use case. That creates a broad surface area on top of a foundation the team does not fully understand. His Google Docs example captures the problem: the team had never built support for tagging an agent in a comment, so Stein could not predict whether the agent would receive the comment, see a link, or respond. The uncertainty was in the shipped product, not just in the roadmap.

### Product specs should describe affordances that agents can compose
[05:50](https://www.youtube.com/watch?v=PthmdT92qNg&t=350s)
A traditional requirement might say that Stacy must read and respond to Google Docs comments. Stein argues that this quickly expands into an unmanageable list of cases: access to the document, direct replies, other people entering the thread, and comments addressed to someone else. The team is not building a general commenting system, and the same questions could appear in Linear, Figma, or LinkedIn. He proposes describing broader affordances, such as communicating, emailing, and collaborating. The agent's planning and workflow should compose those building blocks at runtime. Product teams then give engineers and customers useful capabilities without pretending they can specify every place and interaction in advance.

### Emergent behavior becomes part of the product manager's job
[06:44](https://www.youtube.com/watch?v=PthmdT92qNg&t=404s)
Stein says behavior often works in ways the team did not expect. Product managers therefore need to discover functionality after building the underlying pieces. He compares the pieces to Lego bricks that the engineering team, product, and customers can compose. This changes how teams communicate ideas because a conventional PRD or Figma file may not express a behavior that nobody has imagined yet. The work includes trying combinations, observing what agents do, and deciding which behaviors are useful enough to keep. Stein calls discovering what the product can do a new part of the job, since he could not simply sit in front of a Google Doc and write a complete specification for the agent's behavior.

### Evals can become a living product specification
[08:52](https://www.youtube.com/watch?v=PthmdT92qNg&t=532s)
Stein defines evals as a testing framework for probabilistic AI agents. A deterministic test can check that withdrawing $100 reduces a bank balance by $100. An agent test might ask whether a Slack reply is somewhat snarky and funny without becoming mean. Another LLM can judge whether the response meets that criterion, with a target such as passing 80% of the time. Stein wants product managers to inspect these evals because they show what the software can do and begin to specify what it should do. An eval can also create a shared threshold: a behavior might be acceptable at 90% and blocked from shipping below that level, while an action such as issuing an unsupported refund can require a much stricter standard.

### Vibe-coded prototypes let product teams test the feel of an interaction
[10:58](https://www.youtube.com/watch?v=PthmdT92qNg&t=658s)
Stein says it is hard to describe an agent experience from a blank page. Questions about how many clarifying questions an agent should ask are visceral, and a written requirement can easily get the answer wrong. His team once specified that the teammate should ask many clarifying questions. Users tried it and said, "This sucks." A quick prototype or vibe-coded version let the team experience the interaction and notice the problem. He is clear that this prototype is not a production feature and should not be used to pressure engineers about delivery time. Its purpose is to give the product team a feel for the experience before it becomes a commitment. He uses Claude repeatedly saying "certainly" as another example of an interaction that seems good until repeated use makes it annoying.

### Testing AI products requires searching for behavior nobody specified
[13:00](https://www.youtube.com/watch?v=PthmdT92qNg&t=780s)
For agent products, ordinary happy-path testing does not reveal the full behavior. Stein uses a bar-testing joke to explain the gap: a QA engineer can test many unusual beer orders, while the first customer asks for the bathroom and causes the bar to fail. He applies the same idea to agents by trying unexpected combinations, such as having an agent post LinkedIn comments or react to Spotify tracks. These experiments are ways to discover emergent behavior. Teams must also decide what to report to engineering. Too many emojis may be undesirable without being a bug, because nobody wrote that constraint down. Evals and explicit thresholds help separate a real product failure from a subjective preference.

### Customer trust depends on admitting uncertainty and working together
[16:07](https://www.youtube.com/watch?v=PthmdT92qNg&t=967s)
Stein says the usual product roles break down with probabilistic agents. A future-focused roadmap can sound like witchcraft, while the honest-broker role is difficult when the product team cannot explain exactly how the current system works. His current approach is to tell customers that they are inventing the future together. He presents this as a practical partnership with customers who are willing to explore how agents may change their business, rather than as a promise that everything already works. This gives the customer a truthful account of the product's state while making experimentation part of the relationship. Stein does not claim this is a permanent answer, but says it is what works for him in 2025.

### Model upgrades can change product behavior without product changes
[18:15](https://www.youtube.com/watch?v=PthmdT92qNg&t=1095s)
Stein describes a further source of uncertainty: the agents can become more capable when the underlying models are upgraded. He has seen them begin checking their own work, such as running a query to confirm that a row was inserted correctly, even though his team did not explicitly instruct them to do so. The behavior may be useful, but its arrival changes what the product does. This makes old product practices less reliable because a fixed implementation can acquire new behavior through a model change. Stein still keeps familiar principles such as listening to customers and solving real problems. He argues that the tools and techniques used to define, test, and manage those products need to change.

## Notable quotes
- "I actually don't know what's going to happen." (02:15)
- "It's the affordances that we need to think about, not the individual features." (06:28)
- "Evals are a testing framework for probabilistic AI for agents." (08:53)
- "The answer is because it's so easy to prototype and vibe code something and get the feels." (11:36)
- "We're inventing the future together." (17:26)

## Tools & references mentioned
- Teammates
- Stacy
- Google Workspace
- Gmail
- Slack
- Google Docs
- Google Sheets
- Linear
- Figma
- LinkedIn
- Spotify
- Claude
- LLMs

## Who should watch
- Product managers defining agent behavior when customers can use the product in ways the team did not plan for.
- Engineering and product teams deciding how evals, prototypes, and exploratory testing should influence what ships.
- Founders and customer-facing teams who need to explain an AI product honestly when its behavior is probabilistic and still changing.

## Related talks

- [Build Dynamic Products, and Stop the AI Sideshow](https://aietalks.com/talks/build-dynamic-products-and-stop-the-ai-sideshow) (Eliza Cabrera, Workday & Jeremy Silva, Freeplay, 18:10)
- [Form factors for your new AI coworkers](https://aietalks.com/talks/form-factors-for-your-new-ai-coworkers) (Craig Wattrus, Flatfile, 15:35)
- [Why Your Product Needs an AI Product Manager, and Why It Should Be You](https://aietalks.com/talks/why-your-product-needs-an-ai-product-manager-and-why-it-should-be-you) (James Lowe, Incubator for AI, UK Government, 18:37)
- [Building AI Products That Actually Work](https://aietalks.com/talks/building-ai-products-that-actually-work) (Ben Hylak, Raindrop & Sid Bendre, Oleve, 18:42)
- [AI Agents, Meet Test Driven Development](https://aietalks.com/talks/ai-agents-meet-test-driven-development) (Anita, Vellum, 29:10)