# Does GenAI "belong" to data scientists?

Phil Hetzel, Braintrust | AI Engineer Europe 2026 | 18:54

Source: https://www.youtube.com/watch?v=NKwIX3CiRgU
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/does-genai-belong-to-data-scientists
Published: 2026-05-25
Tags: evals, observability, team-adoption

## TL;DR
- Generative AI is often handed to ML or data science teams because it has "AI" in its name, even though the underlying model and training pipeline already exist.
- Agent development needs product, application, systems, data science, and domain expertise because prompts, context, distributed execution, human annotation, and evaluation all affect the result.
- Data scientists can add value through guardrails, evaluation design, LLM-as-judge analysis, and fine-tuning, while domain experts contribute prompts, context, and judgments about agent behavior.

## Summary
Phil Hetzel argues that traditional enterprises often isolate generative AI inside an ML or data science team because the technology has "AI" in its name. That division misses much of the work involved in building agents. Anthropic, OpenAI, and Mistral already train and deploy the underlying models, so teams usually change agent behavior through prompts and context. Agent systems can also involve distributed infrastructure, product integration, human annotation, and evaluation of whether the agent actually solves the user's problem. Data scientists still have an important role. They can explain model limits, add guardrails, assess LLM judges, create evaluation datasets, and fine-tune open models when needed. Hetzel's answer is a diverse team in which product and systems engineers build the surrounding application, domain experts shape prompts and review traces, and data scientists help create evaluation and observability feedback loops.

## Key ideas
### Traditional enterprises hand agents to existing AI teams by default
[02:56](https://www.youtube.com/watch?v=NKwIX3CiRgU&t=176s)
Hetzel describes a common enterprise pattern: an executive hears that the company needs agents, passes the request down, and an existing ML or data science platform team receives it. The assignment feels natural because generative AI includes the word "AI." AI-native companies take a different path. They often start with a small engineering team that works across product, engineering, and AI rather than preserving separate specialties. Their smaller size also gives each person closer contact with the problem the agent must solve.

### The underlying model pipeline is already provided by model companies
[05:17](https://www.youtube.com/watch?v=NKwIX3CiRgU&t=317s)
Traditional ML work includes collecting data, training and testing a model, checking for overfitting, and deploying the result. Hetzel says Anthropic, OpenAI, and Mistral have already done that process for their foundation models and exposed them through APIs. Application teams still need to run their own evaluations after integrating those APIs. The work has moved toward testing how the model behaves inside a particular product rather than repeating the original training pipeline.

### Prompts and context change agent behavior more often than retraining
[07:12](https://www.youtube.com/watch?v=NKwIX3CiRgU&t=432s)
In traditional ML, teams change a model by adding training data, doing feature engineering, or comparing model changes through A/B tests. With generative AI, teams can often change behavior by changing the inputs, including the prompt and the context supplied to the model. Hetzel calls this context engineering. People who understand real users and the problem being solved may be better placed to make those changes because they have closer contact with the intended use case.

### Agent evaluation covers the product's function, not only model metrics
[09:18](https://www.youtube.com/watch?v=NKwIX3CiRgU&t=558s)
Hetzel gives data scientists a strong case for owning part of agent development. They understand neural networks and can bring disciplined testing and production processes. His objection is that teams can become fixed on precision, recall, and F1 because those metrics are familiar. Agents require evaluation across a wider surface, including whether the system functionally performs the task users need. A technically sound component can still produce a poor experience when placed inside a larger agent workflow.

### Agents create systems work that product engineers already understand
[09:58](https://www.youtube.com/watch?v=NKwIX3CiRgU&t=598s)
Hetzel compares LLMs to APIs, which product engineers already use to retrieve information from other systems and put it into an application. A complex agent may involve a supervisor calling child agents across different infrastructure, with each part calling more systems. That makes agent development a distributed systems problem as well as an AI problem. Product, application, and systems engineers can therefore contribute directly to how the agent runs and how users experience it.

### Domain experts should control prompts and review agent traces
[10:56](https://www.youtube.com/watch?v=NKwIX3CiRgU&t=656s)
Subject matter experts and product managers understand the problem closest to the agent's intended use. Hetzel says they should have substantial influence over the prompts that seed the agent. They can also inspect traces and annotate whether an interaction worked, whether it failed, and why. This human annotation work gives technical teams information that model metrics alone cannot provide. It also keeps the agent tied to the actual domain rather than to an abstract benchmark.

### Data scientists add guardrails and discipline to evaluation
[12:32](https://www.youtube.com/watch?v=NKwIX3CiRgU&t=752s)
Hetzel does not ask data scientists to abandon agent work. He assigns them several useful roles. They can explain that an LLM predicts tokens rather than possessing human-like understanding, which can temper overly aggressive implementations. They can scrutinize LLM-as-judge evaluations because judges are themselves prompts and models. Their experience with labeled datasets and precision, recall, and F1 can help test those evaluators. Fine-tuning an open-source model for a specific use case is another area where their technical background is especially useful.

### The team needs a feedback loop between production and experiments
[14:29](https://www.youtube.com/watch?v=NKwIX3CiRgU&t=869s)
Hetzel's proposed team combines product, application, and systems engineers with data scientists and domain experts. Engineers implement requirements and build the systems around the agent. Domain experts perform human annotation and prompt and context engineering. Data scientists can help implement evaluation and observability pipelines that connect experimentation with production behavior. In the question period, Hetzel explains that production data can continually expand the offline evaluation set, while grounded data helps the team check whether its evaluators are moving closer to human agreement.

## Notable quotes
- "The model's already built." (05:17)
- "When you're analyzing agents, it is far broader of a surface area that you need to be evaluating." (09:37)
- "It makes sense to have a very diverse team when you're building these platforms." (12:18)
- "You need to bring them into the fold if you want to have a very relevant agent to your use case." (15:06)

## Tools & references mentioned
- Braintrust
- Anthropic
- OpenAI
- Mistral
- Databricks
- Slalom Consulting
- precision
- recall
- F1
- LLM as judge

## Who should watch
- You work in a traditional ML or data science team that has been asked to build agents and want to understand which responsibilities should move beyond that group.
- You are a product, application, or systems engineer building an agent and need a clear case for owning parts of the work.
- Your team has domain experts who understand the users but are not involved in prompts, trace review, or evaluation.

## Related talks

- [Why Agent Engineering](https://aietalks.com/talks/why-agent-engineering) (Shawn Wang, Latent.Space, 11:45)
- [Lessons from building GenAI based applications](https://aietalks.com/talks/lessons-from-building-genai-based-applications) (Juan Peredo, 33:13)
- [The Build-Operate Divide: Bridging Product Vision and AI Operational Reality](https://aietalks.com/talks/the-build-operate-divide-bridging-product-vision-and-ai-operational-reality) (Chris Hernandez, Chime & Jeremy Silva, Freeplay, 12:50)
- [AI Leadership](https://aietalks.com/talks/ai-leadership) (Alex Lieberman, Morning Brew and 10X & Kath Korevec, Google Labs & Katelyn Lesse, Anthropic & Michele Catasta, Replit & Lisa Orr, Zapier & Steve Yegge, Sourcegraph and AMP & Gene Kim, IT Revolution & Bill Chen & Brian Fioca, OpenAI & Martin Harrysson & Natasha Maniar, McKinsey & Yegor Denisov-Blanch, Stanford & Itamar Friedman, Qodo & Olive Song, MiniMax & Asaf Bord, Northwestern Mutual & Lei Zhang, Bloomberg & Samir Mody, The Browser Company & Max Kanat-Alexander, Capital One & Arman Hezarkhani, 10X & Justin Reock, DX & Dan Shipper, Every & Mel Lutzky, Graphite, 8:16:05)
- [Scaling Agents for Gen AI Products](https://aietalks.com/talks/scaling-agents-for-gen-ai-products) (Anju Kambadur, Bloomberg, 19:38)