# AI Platform Engineering

Patrick Debois | AI Engineer World's Fair 2024 | 28:18

Source: https://www.youtube.com/watch?v=5qNXdLbEdew
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/ai-platform-engineering
Published: 2024-12-31
Tags: developer-experience, enterprise, evals, observability

## TL;DR
- AI platform teams centralize model access, data connectors, versioning, observability, caching, feedback, and access control so product teams do not rebuild them.
- Platform adoption requires prototyping tools, training, local development environments, documentation, and feedback from product owners, alongside infrastructure.
- AI changes software work by increasing generation and review, so teams need evaluations, monitoring, governance, and failure preparation before they scale applications.

## Summary
Patrick Debois applies lessons from DevOps and Team Topologies to the spread of generative AI inside companies. He describes a familiar progression: one pilot team experiments, several teams adopt the approach, and shared infrastructure eventually moves into a platform team. An AI platform can provide model access, vector databases, data connectors, model versioning, proxies, observability, evaluations, caching, feedback collection, and guardrails. Infrastructure alone is insufficient. Teams also need prototyping tools, training, documentation, local development environments, and help finding worthwhile use cases. Debois is candid about the problems: models change, frameworks move quickly, testing is difficult, and engineers can lose understanding when AI writes more code. He recommends combining centralized rules with team-specific guardrails. AI platform work should connect with cloud operations, security, developer experience, and data platform teams. He suggests this structure for organisations scaling beyond a small number of teams, rather than for a ten-person company.

## Key ideas
### AI adoption usually moves from a pilot team into a platform
[01:47](https://www.youtube.com/watch?v=5qNXdLbEdew&t=107s)
Debois describes a pattern he has seen with Cloud, DevOps, and other new practices. One team starts as a pilot, two or three more teams follow, and the organisation extracts reusable lessons. Those lessons become an abstraction layer so other teams can move faster without understanding the whole underlying system. He applies the same pattern to generative AI. Early work brought engineers into data science teams because data scientists understood models while engineers understood production systems. As applications became more about integration, the engineering share grew, the approach spread to more teams, and shared capabilities moved into a platform.

### An AI platform centralizes the infrastructure that teams would otherwise duplicate
[07:52](https://www.youtube.com/watch?v=5qNXdLbEdew&t=472s)
The platform can give teams access to approved models and help the company choose model and cloud providers. It can provide vector databases, embeddings, and connectors to company data sources for retrieval-augmented generation. Debois also describes model version control and a shared repository, plus a proxy that controls who can access which models. Production systems need prompt tracing, monitoring for data quality, and evaluations that run continuously so teams notice model changes or unusual user behaviour. Caching and richer feedback collection can also be shared. Feedback might include inline editing rather than only thumbs up or thumbs down.

### Enablement has to help teams experiment and find real use cases
[12:14](https://www.youtube.com/watch?v=5qNXdLbEdew&t=734s)
Once infrastructure exists, the platform team has to help its customers use it. Debois recommends prototyping tools, secure access to company data, frameworks for learning, documentation, and local development environments. Product owners should be included because they can experiment and identify useful applications in their domain. He warns against demanding that every part of a product use generative AI before the company understands the use case. A focus on training or fine-tuning can delay production, while premature cost optimisation can distract from proving the business case. Engineers also need help interpreting end-user feedback, which often requires product knowledge.

### Changing models and fast-moving frameworks make ordinary engineering practices harder
[15:37](https://www.youtube.com/watch?v=5qNXdLbEdew&t=937s)
Debois says engineers often do not know which model to choose, how to access data, or how to test an AI application. Frameworks change quickly, so teams may abandon them and build at lower layers themselves. Over two years, his organisation used eight different language models to improve quality. Each change meant rewriting prompts and manually checking the application, which created a large cost and gave no guarantee that the result still worked. He argues that evaluations and testing must exist before refactoring. Exact checks are straightforward, but semantic checks may require another model, and the overall testing problem remains unresolved.

### AI-assisted coding increases review work and can reduce situational awareness
[18:51](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1131s)
Debois describes coding copilots as a way to help developers become comfortable with AI, but he does not present the productivity effect as simple. More code was produced, review times increased, and AI-generated pull requests were larger. The person using AI shifts from producing work toward managing and reviewing it. That can create a gap in understanding when the developer no longer has the domain model or experience behind the generated code. He connects this to the 'ironies of AI Automation': faster generation can coexist with slower review and weaker awareness of what the system is doing.

### Enablement must include testing, monitoring, and preparation for failure
[21:09](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1269s)
Debois compares AI adoption with the development of DevOps practices. Automation led to continuous integration and delivery, more tests, monitoring, observability, design for failure, prediction, and eventually chaos engineering. AI teams can choose fewer controls based on their risk tolerance, but the platform should reduce the cognitive load when something fails. Enablement therefore includes more than making developers enthusiastic about AI. It gives them ways to evaluate changes, see what happens in production, and practise responding to failures.

### Governance combines central rules with controls owned by individual teams
[22:15](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1335s)
The governance workflow should cover awareness, training choices, licences, model origins, risk levels, prompt injection, guardrails, personally identifiable information, metrics, and alerts. Debois warns that services which record everything on a developer's screen can create privacy concerns while promising productivity gains. He also says an open model is not automatically open in every relevant sense, so teams should examine its origin. In the question period, he explains that central governance teams should apply generic guardrails to all models, while product teams add rules for their own use cases. This gives teams self-service without forcing every team to duplicate common controls.

### AI platform teams should work with the other platform groups
[24:34](https://www.youtube.com/watch?v=5qNXdLbEdew&t=1474s)
Debois places AI platform infrastructure alongside cloud operations, security operations, developer experience, and data platform teams. These groups can share knowledge about permissions, governance, cloud vendors, and performance. He also mentions an experience crew that works with feature teams to make AI behaviour and user experience consistent in the product. The organisation should avoid this structure as premature optimisation for a ten-person company. When a company is scaling to ten or more teams, he considers the pattern more appropriate because the platform groups can collaborate instead of each product team solving the same operational and security problems.

## Notable quotes
- "The name does not matter that much in the beginning." (03:28)
- "You don't want every team to build this service because it's quite expensive." (11:29)
- "If the AI doesn't work just use more AI." (18:07)
- "We started designing for failure." (21:31)
- "I would not recommend to do this for a 10 person company." (25:25)

## Tools & references mentioned
- Team Topologies
- DevOps
- DevOps Topologies
- Ironies of Automation
- Ironies of Generative AI
- Kubernetes
- OpenAI
- GitHub Copilot
- RAG
- RAG Ops
- CloudOps
- SecOps
- Developer Experience
- Data Platform
- AI Platform

## Who should watch
- You are forming an AI platform team after several product teams have started building generative AI applications.
- Your teams keep duplicating model access, data connectors, prompt tracing, evaluations, feedback systems, or guardrails.
- You are introducing AI coding tools and need to account for larger pull requests, longer reviews, changing models, and loss of system understanding.

## Related talks

- [AI Leadership](https://aietalks.com/talks/ai-leadership) (Alex Lieberman, Morning Brew and 10X & Kath Korevec, Google Labs & Katelyn Lesse, Anthropic & Michele Catasta, Replit & Lisa Orr, Zapier & Steve Yegge, Sourcegraph and AMP & Gene Kim, IT Revolution & Bill Chen & Brian Fioca, OpenAI & Martin Harrysson & Natasha Maniar, McKinsey & Yegor Denisov-Blanch, Stanford & Itamar Friedman, Qodo & Olive Song, MiniMax & Asaf Bord, Northwestern Mutual & Lei Zhang, Bloomberg & Samir Mody, The Browser Company & Max Kanat-Alexander, Capital One & Arman Hezarkhani, 10X & Justin Reock, DX & Dan Shipper, Every & Mel Lutzky, Graphite, 8:16:05)
- [Hiring & Building an AI Engineering Team](https://aietalks.com/talks/hiring-building-an-ai-engineering-team) (Dr. Bryan Bischof, Hex, 29:07)
- [The Build-Operate Divide: Bridging Product Vision and AI Operational Reality](https://aietalks.com/talks/the-build-operate-divide-bridging-product-vision-and-ai-operational-reality) (Chris Hernandez, Chime & Jeremy Silva, Freeplay, 12:50)
- [Leadership in AI Assisted Engineering](https://aietalks.com/talks/leadership-in-ai-assisted-engineering) (Justin Reock, DX, 18:11)
- [The AI Pivot](https://aietalks.com/talks/the-ai-pivot) (Brittany Walker, CRV & Chris White, Prefect & Bryan Bischof, Hex, 35:16)