# On Engineering AI Systems that Endure the Bitter Lesson

Omar Khattab, Databricks | AI Engineer World's Fair 2025 | 19:12

Source: https://www.youtube.com/watch?v=qdmxApz3EJI
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/on-engineering-ai-systems-that-endure-the-bitter-lesson
Published: 2025-08-06
Tags: evals, prompt-engineering, reliability

## TL;DR
- AI engineering should define the task, tools, control flow, and evaluation rather than hard-code the details of search and learning.
- Premature optimization happens when engineers code at a lower level of abstraction than they can justify, such as embedding model-specific prompting tricks in the system.
- A durable AI system separates its specification from changing models, inference strategies, and learning algorithms, which is the design goal behind DSPy.

## Summary
Omar Khattab applies Rich Sutton's bitter lesson to AI software engineering. Scaling search and learning is useful for building intelligence, but software exists to make systems reliable, controllable, understandable, and scalable. Engineers therefore need to define what the system should do while leaving lower-level model behavior open to change. Khattab argues that prompts are a poor programming abstraction because they mix the task definition with model-specific wording, inference strategies, formatting instructions, and accidental tricks. He recommends separating natural-language specifications, evaluations, and code. Evaluations define the behavior that matters across model changes. Code controls tools, information flow, and reliable composition. DSPy provides a programming model built around signatures, allowing these application-level definitions to be decoupled from interchangeable language models, inference methods, and optimizers. His advice is to avoid hand-engineering below the abstraction level the problem justifies, then invest in system-specific structure and evaluations.

## Key ideas
### AI software must account for models that change every week
[00:35](https://www.youtube.com/watch?v=qdmxApz3EJI&t=35s)
Khattab says AI software engineering has unusual moving parts. New language models appear constantly, with different trade-offs in quality, cost, speed, and application fit. Providers may change a model behind the same API name, and the latest model may have new quirks that require new prompting guidance. Researchers also release new learning algorithms, prompt optimizers, search methods, inference strategies, and agent frameworks at a rapid pace. Engineers who are doing a reasonable job may still be scrambling every week to keep up. This makes it unsafe to tie an application too closely to the current model or technique.

### The bitter lesson applies to intelligence, while software engineering defines the problem
[03:42](https://www.youtube.com/watch?v=qdmxApz3EJI&t=222s)
Khattab summarizes Rich Sutton's bitter lesson as the observation that domain-specific methods often stop scaling and lose to general methods that scale search and learning. He says this is the right advice for researchers building agents that must learn quickly in new environments. AI software has a different purpose. People build software because existing intelligence is unreliable and does not solve the required problems in a controllable way. Software should provide reliable, robust, controllable, scalable systems that people can reason about. Engineering therefore has to specify what the system is learning and searching for, rather than trying to hand-code the details of search and learning.

### Premature optimization means coding below a justified abstraction level
[06:57](https://www.youtube.com/watch?v=qdmxApz3EJI&t=417s)
Khattab connects the bitter lesson to the software idea that premature optimization is the root of all evil. Domain knowledge and human design are not inherently harmful. They become harmful when engineers use them too early in ways that constrain a system based on a poor understanding of the problem. He illustrates this with code that computes a square root through machine-specific bit manipulation. Such code may be correct for an old representation but fail on a different architecture, and a normal square-root operation might become faster on newer hardware. His rule is to express the more general operation first, then move to a lower abstraction only after showing that the higher one is insufficient.

### Tight coupling makes machine-learning systems age badly
[10:42](https://www.youtube.com/watch?v=qdmxApz3EJI&t=642s)
Khattab argues that applied machine learning and prompt engineering often rewrite a whole system around the latest technique. He points to a 2006 modular multilingual question-answering paper whose architecture resembles modern multi-agent systems, with an execution manager, question analyzers, and retrieval strategies over multiple corpora. In ordinary software, a reasonably designed system could often move to new hardware or an operating system and continue working. Machine-learning systems do not get that benefit when their architecture is expressed through model-specific assumptions. The software needs abstractions that preserve the system design while allowing its lower-level components to change.

### Prompts mix the task with accidental model-specific instructions
[12:12](https://www.youtube.com/watch?v=qdmxApz3EJI&t=732s)
Khattab calls a prompt a poor abstraction for programming because it is a stringly typed canvas with little explicit structure. A prompt can combine the task definition with examples, wording tricks, formatting directions, persona instructions, and an inference-time strategy such as telling the model it is an agent. The engineer cannot easily distinguish the fundamental behavior from a temporary trick that happened to work for one model. Instructions such as producing XML, answering in JSON, or acting as a particular expert are often implementation details rather than the actual task. Khattab says prompts can be useful for managing an agent, but they are a poor basis for engineering an AI system.

### A durable system separates specifications, evaluations, and code
[14:28](https://www.youtube.com/watch?v=qdmxApz3EJI&t=868s)
Khattab recommends separation of concerns. Natural-language descriptions should express the parts of a specification that cannot be stated another way, but they should not carry every model-tuning adjustment. Evaluations state what the engineer actually cares about and remain meaningful when the model changes. He says systems need both evaluations and instructions because learning behavior from data is harder than following instructions. Code defines tools, structure, information flow, privacy boundaries, and function composition. Neural networks do not reliably perform composition, while software functions provide that reliability by construction. A useful programming canvas combines these elements without entangling them with a particular model.

### DSPy decouples application definitions from changing AI techniques
[17:11](https://www.youtube.com/watch?v=qdmxApz3EJI&t=1031s)
Khattab's second takeaway is to invest in definitions specific to the application and decouple them from lower-level pieces that expire quickly. DSPy provides this separation through a first-class concept called signatures. The application can express what its components should do while adapters and optimizers handle changing language models and learning or search methods. Khattab says the system should allow engineers to swap a chain of thought approach for an agent or another inference strategy without rewriting the application. He recommends investing in signatures, essential control flow, tools, and evaluation, then swapping models and optimizers as they improve.

## Notable quotes
- "We build software not because we lack AGI but because we want reliable, robust, controllable scalable systems." (05:57)
- "Premature optimization is what is happening if and only if you're hard coding stuff at a lower level of abstraction than you can justify." (10:08)
- "A prompt is a horrible abstraction for programming and this needs to be fixed ASAP." (12:12)
- "Your job as an engineer is to invest in your actual system design." (14:28)
- "Invest in essential control flow and tools and invest in evaluating on by hand and ride the wave of swappable models." (18:46)

## Tools & references mentioned
- Rich Sutton
- The Bitter Lesson
- DSPy
- Databricks
- MIT EECS
- ColBERT
- Koushik Sen
- Prompt optimization
- Reinforcement learning

## Who should watch
- You are building an LLM application whose prompts contain task instructions, formatting rules, agent behavior, and model-specific tricks in one large string.
- Your team keeps rewriting application code whenever a new model, prompting method, agent framework, or inference strategy appears.
- You want a concrete way to separate application behavior and evaluation from models, tools, control flow, and optimization methods.

## Related talks

- [The Unreasonable Effectiveness of Separating the Task from the Model](https://aietalks.com/talks/the-unreasonable-effectiveness-of-separating-the-task-from-the-model) (Maxime Rivest, DSPy & Isaac Miller, DSPy; cmpnd, 17:11)
- [Why (Senior) Engineers Struggle to Build AI Agents](https://aietalks.com/talks/why-senior-engineers-struggle-to-build-ai-agents) (Philipp Schmid, Google DeepMind, 10:40)
- [The Many Ends of Programming](https://aietalks.com/talks/the-many-ends-of-programming) (Ray Myers, All Hands AI, 27:12)
- [AI Engineer Summit 2023, Day 1 Livestream](https://aietalks.com/talks/ai-engineer-summit-2023-day-1-livestream) (Benjamin Dunphy, Software 3.0 LLC & swyx, Latent.Space & Smol.ai & Amjad Masad & Michele Catasta, Replit & Toran Bruce Richards, AutoGPT & Simón Fishman & Logan Kilpatrick, OpenAI & Flo Crivello, Lindy & Barr Yaron, Amplify & Sasha Sheng & Harrison Chase, LangChain & Shreya Rajpal, Guardrails AI & Eugene Yan, Amazon & Linus Lee, Notion & Brittany Walker, CRV & Chris White, Prefect & Bryan Bischof, Hex, 4:50:00)
- [Build AI Systems for Discernment, Not Approval](https://aietalks.com/talks/build-ai-systems-for-discernment-not-approval) (Angel Ortmann Lee, Duolingo, 25:53)