# The AI Pivot

Brittany Walker, CRV & Chris White, Prefect & Bryan Bischof, Hex | AI Engineer Summit 2023 | 35:16

Source: https://www.youtube.com/watch?v=bDQn1w0idXs
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/the-ai-pivot
Published: 2023-11-07
Tags: data-pipelines, evals, observability, product-strategy

## TL;DR
- Chris White treats AI systems as expensive, brittle, non-deterministic data APIs that need the same workflow and observability discipline as other production data systems.
- Bryan Bischof argues that AI features should reduce friction inside an existing product, with evaluation tied closely to production and user behavior.
- Both speakers warn that promising early prototypes can hide a long period of engineering work, testing, data preparation, and failure handling before production use is dependable.

## Summary
Brittany Walker moderates a discussion with Chris White of Prefect and Bryan Bischof of Hex about adding AI to established startup products. White describes AI workloads as data APIs that need orchestration, monitoring, debugging, and careful handling of failure modes. Bischof focuses on making Hex easier to use, while keeping Magic as an augmentation of the existing product rather than a separate destination. The panel covers when to build or buy infrastructure, why Hex built its own evaluation system, and why neither company wanted to build a vector database. Both speakers describe the gap between an impressive prototype and a dependable product. They discuss private betas, opt-in releases for security-conscious customers, typed outputs, binary evaluations, and retrieval improvements for reducing hallucinations. Their closing advice is practical: design around machine-to-machine use instead of adding chat everywhere, and expect the production work to include a great deal of ordinary data engineering.

## Key ideas
### AI investment can follow the data and workflow problems a company already understands
[01:53](https://www.youtube.com/watch?v=bDQn1w0idXs&t=113s)
Chris White says Prefect had two reasons to invest in AI. Its users are data engineers and data scientists, and many AI applications involve data work such as semantic search, retrieval, and summarization. The company wanted to understand how customers were putting those systems into production, including data movement between systems such as vector databases. White also sees a production AI application as "a remote API that is expensive brittle and non-deterministic," which makes it a data API from an orchestration perspective. Prefect also created the open source Marvin repository so the team could learn from users and keep up with the area in public.

### AI features should remove friction inside the product's existing workflow
[03:27](https://www.youtube.com/watch?v=bDQn1w0idXs&t=207s)
Bryan Bischof says Hex's AI team is focused on making the product feel magical by reducing friction in data science work. He gives the example of an explode feature that breaks a long SQL query made from common table expressions into chained cells. The feature is simple, but it solves a problem he had wanted fixed for years. For Hex, the question is whether a feature helps users do their work more easily and gives them more room for creative and business-focused thinking. Bischof also says a data platform that ignores AI risks making its product feel outdated as these capabilities become expected.

### Small teams and direct user contact can produce useful AI learning quickly
[04:46](https://www.youtube.com/watch?v=bDQn1w0idXs&t=286s)
Prefect kept its AI effort small. White says the team began with one engineer, built a simple prototype, then grew to a couple of people while remaining around five engineers despite having a roadmap that could occupy twenty. He calls ruthless prioritization an advantage. Prefect also uses Marvin to speak directly with users experimenting with language models. At Hex, Bischof says his team began with one person and expanded carefully. These separate efforts let the companies learn and ship without moving large parts of the organization before they had evidence about what users wanted.

### Evaluation should be close to production and tied to a clear user task
[10:20](https://www.youtube.com/watch?v=bDQn1w0idXs&t=620s)
Bischof says evaluation was one of the first things he asked about when joining Hex. He wanted evaluations close to production, using production where possible, because deep platform integration is needed and a SaaS company may struggle to move quickly if it depends on an outside system. Hex built its own evaluation tooling. Bischof says teams should define what success means for a specific task, then reduce the result to a clear human judgment such as yes or no. He rejects the idea that complicated tasks cannot be evaluated this way and says teams should keep searching for a workable evaluation.

### The right infrastructure boundary depends on where the complexity sits
[11:21](https://www.youtube.com/watch?v=bDQn1w0idXs&t=681s)
Hex chose to build its evaluation system because it needed deep hooks into the product, but it chose LanceDB for vector retrieval rather than building a vector database. Bischof had used vector tools before and did not want to take on that setup and maintenance burden. Hex built a custom retrieval implementation around LanceDB for its use case. Prefect also did not build a vector database, and White mentions experiments with Chroma and LanceDB that were not yet in production. For White, build-versus-buy decisions look like ordinary choices about whether the existing developer tooling can make a component observable and monitorable.

### Early AI prototypes create a misleading sense of how close production is
[16:04](https://www.youtube.com/watch?v=bDQn1w0idXs&t=964s)
Both speakers describe enthusiasm as an internal challenge. White says engineers initially wanted AI features to do everything, including monitoring themselves, before the team narrowed its focus to removing user friction. Bischof describes a different version of the same problem: the first experiments work for a few cases very quickly, while the long tail of corner cases takes much longer than expected. The distance between "we got this to work for a few cases" and a system that is dependable is especially large for AI features. Hex responds by turning user requests into explicit expectations and checking whether a new data scientist with little context could use the feature successfully.

### Product boundaries matter when AI could become its own destination
[19:22](https://www.youtube.com/watch?v=bDQn1w0idXs&t=1162s)
Prefect created Marvin as a separate open source repository because it could increase learning without requiring a full product commitment. If the project failed to attract attention, the cost was limited. If it succeeded, the team could meet people working on AI and adjacent areas. Hex made the opposite product choice with Magic. Bischof says Magic is an augmentation of Hex, not its own product. The team killed a prototype called Crystal Ball because, although it was capable and well designed, it pulled users away from the main Hex experience. The decision kept AI features available to every Hex user within the same platform.

### Typed outputs and failure visibility make AI easier to connect to software
[31:16](https://www.youtube.com/watch?v=bDQn1w0idXs&t=1876s)
Asked what agent builders should know, White returns to failure modes. He recommends wiring in observability early, capturing unusual inputs and outputs, and making it possible to debug runaway processes. He also mentions caching, while warning that it has tradeoffs. White is especially interested in machine-to-machine tooling and typed outputs, including function calling, because typed handshakes make backend integration and testing easier. In Prefect's own work, moving from untyped prompt-and-string interactions to guaranteed typed outputs made tests easier to write. Bischof adds that Hex reduced hallucinations through more careful retrieval-augmented generation rather than relying on a generic fix.

### Production AI includes ordinary data engineering work
[33:56](https://www.youtube.com/watch?v=bDQn1w0idXs&t=2036s)
The panel closes with two blunt predictions. White says teams should stop building chat interfaces everywhere. He sees chat as a product pattern and AI as a tool, and wants more attention on machine-to-machine interfaces that developers can use directly. Bischof says the work ahead will often be boring. The capabilities may be exciting, but the path to dependable machine learning has always included tedious work. He expects substantial data engineering before AI features work well in production. Their advice is grounded in the gap between a compelling demo and a system that can be tested, monitored, and used repeatedly.

## Notable quotes
- "When I look at AI going into production I see basically a remote API that is expensive brittle and non-deterministic and that's just a data API to me." (02:46)
- "Magic is not a product, magic is an augmentation of our product." (21:41)
- "I definitely stop building chat interfaces." (33:56)
- "A lot of the work that's in front of you as you're building out AI capabilities is going to be incredibly boring." (34:18)

## Tools & references mentioned
- Prefect
- Hex
- Marvin
- CRV
- LanceDB
- Chroma
- Rivet
- Ironclad
- Weights & Biases
- Pinecone
- Hugging Face
- Crystal Ball
- Magic
- OpenAI
- GitHub

## Who should watch
- You are adding language-model features to a product that already has customers and need a way to choose scope without creating a separate AI product.
- Your prototype works in a few examples, but you need guidance on evaluation, rollout, observability, typed outputs, and the long tail of failures.
- You are deciding which AI infrastructure to build yourself and which parts to adopt, especially for evaluation, retrieval, and backend workflows.

## Related talks

- [Build Dynamic Products, and Stop the AI Sideshow](https://aietalks.com/talks/build-dynamic-products-and-stop-the-ai-sideshow) (Eliza Cabrera, Workday & Jeremy Silva, Freeplay, 18:10)
- [Building AI Products That Actually Work](https://aietalks.com/talks/building-ai-products-that-actually-work) (Ben Hylak, Raindrop & Sid Bendre, Oleve, 18:42)
- [The Build-Operate Divide: Bridging Product Vision and AI Operational Reality](https://aietalks.com/talks/the-build-operate-divide-bridging-product-vision-and-ai-operational-reality) (Chris Hernandez, Chime & Jeremy Silva, Freeplay, 12:50)
- [Why Your Product Needs an AI Product Manager, and Why It Should Be You](https://aietalks.com/talks/why-your-product-needs-an-ai-product-manager-and-why-it-should-be-you) (James Lowe, Incubator for AI, UK Government, 18:37)
- [Hiring & Building an AI Engineering Team](https://aietalks.com/talks/hiring-building-an-ai-engineering-team) (Dr. Bryan Bischof, Hex, 29:07)