# AI Engineering Without Borders

swyx | AI Engineer World's Fair 2024 | 10:32

Source: https://www.youtube.com/watch?v=eDr0m6R7fI4
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/ai-engineering-without-borders
Published: 2024-10-30
Tags: engineering-culture, product-strategy

## TL;DR
- AI engineering should account for the real constraints and changing laws of AI systems, rather than relying on arbitrary team or conference categories.
- Context windows are growing, model capabilities are getting cheaper, and engineers should understand these trends before making long-term technical bets.
- AI engineers should cross boundaries, question consensus, and turn powerful but unpredictable systems into useful tools for people.

## Summary
swyx argues that borders are human-made constraints, while AI systems routinely cross them through multilingual learning, multimodality, and unclear boundaries between memorization and hallucination. He applies this idea to AI engineering, whose categories, roles, and conference tracks are useful but arbitrary. He proposes thinking about AI engineering as software engineering that encounters more real-world constraints, including facts about human behavior, changing model costs, context windows, and inference speeds. Engineers should understand which conditions are stable and which are temporary trends. He also places AI engineering between safety-focused and acceleration-focused philosophies, adding utility as a second dimension. His advice is to disagree with inherited categories and assumptions, move between disciplines, and treat powerful AI systems as forces that can be shaped into practical tools.

## Key ideas
### AI crosses boundaries that people treat as fixed
[00:13](https://www.youtube.com/watch?v=eDr0m6R7fI4&t=13s)
swyx begins with borders as a human-made constraint. People may need the right paperwork to cross a line, while AI systems cross boundaries easily. A language model trained mostly on English can learn other languages as a side effect. Llama can become a visual language model through post-training, and AI systems do not naturally separate hallucination from information memorized in a world model. Copyright is another boundary that AI does not automatically respect. These examples frame AI as a technology that does not fit neatly inside human categories.

### AI engineering has several useful definitions, but none is final
[01:20](https://www.youtube.com/watch?v=eDr0m6R7fI4&t=80s)
He describes AI engineering as an API boundary, a set of roles, or a job description. He previously grouped roles into the AI-enhanced engineer, the AI products engineer, and the non-human agentic AI engineer. Elicit had also discussed different roles on the Latent Space podcast. These definitions can help teams decide where responsibilities begin and end, but swyx does not think they settle the field. Saying that every person simply has a different opinion avoids the harder question of whether there are shared facts about the work.

### Conference tracks and team boundaries are made up
[03:01](https://www.youtube.com/watch?v=eDr0m6R7fI4&t=181s)
The 2023 conference included tracks on RAG, code generation, agents, and multimodality, and the same subjects appeared again. As AI engineers become more capable, they face additional concerns such as moving to open models, building evaluations, scaling inference, deploying to the Fortune 500, and managing AI strategy. swyx says the categories are arbitrary because he created them. Agents and code generation, or RAG and open models, could have been grouped differently. Engineers should know how the areas connect instead of treating each track as an isolated discipline.

### AI engineering sits between software engineering and real-world engineering
[05:58](https://www.youtube.com/watch?v=eDr0m6R7fI4&t=358s)
Definitions of software engineering from IEEE and Google Developers focus on designing, implementing, testing, documenting, and managing software over its lifecycle. Definitions of engineering from IEEE and the National Association of Engineers also involve natural science and benefit to humanity. swyx proposes that AI engineering lies between these ideas. It still includes software engineering, but AI systems bring more real-world constraints into the work. He uses this distinction to ask what stable laws or recurring conditions AI engineers should understand.

### Some AI constraints are stable, while others change quickly
[06:47](https://www.youtube.com/watch?v=eDr0m6R7fI4&t=407s)
swyx separates constants from contingent facts. When designing for people, engineers should account for humans speaking at about 80 words per minute and reading at about 200 words per minute. A contingent fact can become a temporary baseline, such as Apple Intelligence's advertised local inference speed of 30 tokens per second. These conditions are not physical laws, and they move with market and technical forces. Engineers should understand which trends are likely to continue so they do not make bets that overwhelming evidence will soon invalidate.

### Context and intelligence costs are moving targets
[07:51](https://www.youtube.com/watch?v=eDr0m6R7fI4&t=471s)
He points to context windows as one major trend. A year earlier, Mosaic's MPT-7B had a 60,000 to 70,000 token context window with substantial loss, while people at the conference had trained million-token windows. Claude 3.5 also showed progress in using the available context, so context utilization matters alongside context length. The cost of intelligence is falling as well. swyx cites a 99.55% decline over two years for GPT-3-level intelligence and estimates that GPT-4-level intelligence had become 80% to 90% cheaper through models such as Llama 3 and newer systems.

### AI engineering combines safety with a strong focus on utility
[08:47](https://www.youtube.com/watch?v=eDr0m6R7fI4&t=527s)
swyx compares AI engineering with EA and EAC, presenting it as a possible middle position between safety and acceleration. He argues that this single dimension does not describe the difference well enough. A second dimension is utility. AI engineers, in his formulation, are utility-focused: they see available systems and want to use them to benefit humanity. This does not remove safety concerns, but it gives practical use a central place in how the field decides what to build.

### Good AI engineers cross boundaries and challenge consensus
[09:17](https://www.youtube.com/watch?v=eDr0m6R7fI4&t=557s)
His message is to disrespect borders, question personal dogma, and disagree with lazy consensus, other people's assumptions, and one's own conclusions. He wants disagreement to be productive rather than passive. His closing image is of someone who looks at a Shoggoth and imagines mass rapid transit, meaning a person who sees a force of nature and wants to turn it into a useful tool. He encourages engineers to move between conference tracks, friend groups, disciplines, and modalities because work usually keeps those groups apart.

## Notable quotes
- "AI is a border disrespect er." (00:31)
- "And it's arbitrary." (04:52)
- "We are utility maxis above all else." (09:02)
- "Try to disagree. Disagree more." (09:17)
- "AI engineers are the kind of person that looks at Shoggoth and sees, instead of a monster that cannot be tamed, they want to turn them into mass rapid transit." (09:34)

## Tools & references mentioned
- Llama
- Latent Space
- Elicit
- Raza Habib
- IEEE
- Google Developers
- National Association of Engineers
- Mosaic
- MPT-7B
- Anthropic
- Claude 3.5
- Greg
- Apple Intelligence
- GPT-3
- GPT-4
- Llama 3
- Shoggoth
- EA
- EAC

## Who should watch
- You are defining AI engineering roles or team boundaries and want a framework that is less tied to arbitrary job titles.
- You are choosing model, context, inference, or deployment strategies and need to account for fast-moving technical trends.
- You work in a specialized AI area and want a direct argument for crossing into adjacent disciplines and challenging settled assumptions.

## Related talks

- [Why Agent Engineering](https://aietalks.com/talks/why-agent-engineering) (Shawn Wang, Latent.Space, 11:45)
- [Leadership in AI Assisted Engineering](https://aietalks.com/talks/leadership-in-ai-assisted-engineering) (Justin Reock, DX, 18:11)
- [Structuring a Modern AI Team](https://aietalks.com/talks/structuring-a-modern-ai-team) (Denys Linkov, Wisedocs, 17:40)
- [AI Platform Engineering](https://aietalks.com/talks/ai-platform-engineering) (Patrick Debois, 28:18)
- [The Infinite Software Crisis](https://aietalks.com/talks/the-infinite-software-crisis) (Jake Nations, Netflix, 18:57)