Why A/B Testing for AI Products Is Weird

Supriya Subudhi, Bumble14:44 · Oct 2026 · 1,413 views
Thumbnail for Why A/B Testing for AI Products Is Weird Watch on YouTube
TL;DR
  1. 1

    Survey interest did not predict usage: 78% of users wanted AI photo feedback, but only 10 to 15% used it in the A/B test.

  2. 2

    A positive average result hid poor feedback, such as telling people who were already smiling to smile more, so Bumble read real outputs and evaluated different user groups.

  3. 3

    The feature's result depended on the model, its cost, and where it could legally ship, so an A/B win alone was not enough to justify launch.

Summary

Supriya Subudhi explains why a positive A/B test is only the beginning for generative AI features. Bumble tested AI photo feedback for dating profiles. Although 78% of surveyed users wanted feedback, only 10 to 15% used the feature. The team then found that good aggregate results hid bad individual suggestions, including advice to smile more when users were already smiling. It added guardrails, read real outputs, and used evals across user segments, languages, and countries. Cost created another problem. The best language model was about 10 times more expensive than a simpler approach, and moving to a cheaper model reduced the feature's impact. That showed the original result came from model quality, not only from the feedback concept. Finally, regulation limited the feature to the US. Subudhi closes with five checks covering usage, output quality, total cost, model affordability, and legal markets.

Key ideas
02:38

Surveyed demand does not predict feature usage

Bumble's research found that users often felt unsure whether their dating profiles were good enough. When shown examples of AI support, almost 78% said they would like feedback. In the A/B test, however, only 10 to 15% of users actually used the feature. Subudhi says this gap showed that openness to AI is different from willingness to use an AI feature. Bumble improved adoption by placing feedback inside existing journeys. Users saw an offer while adding pictures, and people who were struggling to find matches were directed back to profile feedback. The team had to measure actual use instead of relying on survey answers.

05:07

Aggregate improvement can hide bad individual outputs

The first results looked positive because more users added pictures and made profile changes after receiving feedback. Looking at individual outputs exposed a problem: the system sometimes told people to smile more when they were already smiling. Subudhi compares this with Spotify's AI DJ, where a few poor song choices may be tolerable, and a budgeting app, where harmful advice can have a much higher cost. Bumble added guardrails against comments on things users cannot control, such as skin color and height. The team also read real outputs through evals and examined differences across genders, languages, and countries rather than judging only the overall result.

07:48

Generative AI costs rise with usage

Bumble initially chose the best language model because the team wanted high-quality, accurate feedback. That model cost about 10 times more than a simpler approach, before adding the cost of running evals. Subudhi contrasts this with a traditional machine-learning algorithm, where most cost can come from training and the cost per additional user may be smaller. With generative AI, a successful feature can increase the bill as more people use it and use it more often. She says teams should decide why they are using AI before choosing a budget. A short-term return may support one spending limit, while a long-term product or brand goal may justify another.

10:30

A cheaper model can change the product result

When Bumble switched to a cheaper model, the cost fell and the feature's impact fell with it because accuracy also declined. Subudhi says this showed that the A/B result came from the model being tested, not simply from the idea of giving users photo feedback. The team needed repeated tests, model comparisons, prompt tests, and feedback tests across different data sets. Evals made those comparisons faster and let the team examine quality alongside cost. The eventual choice was the cheapest option that still produced a good result. That choice required testing the model and the concept separately.

11:49

A successful US test may have no global rollout

Bumble launched the project in the US, then stopped because the feature could not be shipped to other countries in the same form. Subudhi says AI regulation changes across markets and creates financial and legal risks when companies are not compliant. Product managers and engineers therefore need to think about compliance and legal requirements early. One approach is to design for all intended markets from the start. Another is to launch the strongest version in one country, accepting that it may not be available elsewhere. A company can also build different versions for different countries, which adds cost. The A/B result does not settle that decision.

13:51

An AI test needs five checks beyond the dashboard

Subudhi ends with five questions for teams building AI features. Are people using the feature, or only saying they will? Are teams reading real outputs, or only watching averages? Does the feature still pay off after all costs are included? Is a win on the best model affordable? Are the legal markets for launch clear? These questions connect product usage, output quality, model economics, and regulation. Her conclusion is that an A/B test can show a promising result without answering whether the feature is ready to ship, can scale at an acceptable cost, or can reach the users the company wants to serve.

"With generative AI, the more successful your product, the more people that use it, and the more often they use it, the more your bill increases."08:58
Who should watch
  • You are evaluating a generative AI feature and need to decide whether a positive experiment is enough to launch.
  • Your survey results look strong, but actual adoption is low or the quality of individual outputs is uncertain.
  • You need to compare model quality with inference cost and understand which countries can legally receive the feature.