# Open Challenges for AI Engineering

Simon Willison, Datasette | AI Engineer World's Fair 2024 | 18:49

Source: https://www.youtube.com/watch?v=eTTMUWP5B0s
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/open-challenges-for-ai-engineering
Published: 2024-07-17
Tags: benchmarks, open-models, privacy, security

## TL;DR
- GPT-4-class models now have serious competition, including strong open models and inexpensive alternatives.
- AI tools are power-user tools because their behavior depends on details that providers often do not document.
- AI engineering has to address trust, prompt injection, and unreviewed generated content before these systems can be used responsibly.

## Summary
Simon Willison argues that AI engineering has moved past the period when GPT-4 had no serious competition. Models from Anthropic, Google, Meta, Nvidia, Cohere, Alibaba, and DeepSeek now occupy similar or lower-cost parts of the market, while some GPT-4-class access is free to consumers. The harder problems are practical and social. ChatGPT can handle long searchable PDFs, but its treatment of tables, diagrams, and files depends on hidden implementation details. AI products also face a trust crisis because users often cannot tell how their data is handled. Willison shows how a markdown image can silently exfiltrate private data through a chatbot, and argues that prompt injection must be understood by engineers. He defines slop as unrequested, unreviewed AI-generated content. His answer is for builders to establish responsible usage patterns and help other people develop the judgment to review and take responsibility for generated material.

## Key ideas
### GPT-4 no longer has the field to itself
[00:00](https://www.youtube.com/watch?v=eTTMUWP5B0s&t=0s)
Willison starts with the change in model competition since GPT-4 launched. For about a year, GPT-4-class systems were clearly ahead of other language models. His updated comparison groups Gemini 1.5 Pro, GPT-4o, and Claude 3.5 Sonnet among the strongest models, while Claude 3 Haiku and Gemini 1.5 Flash offer much lower-cost options that can still do substantial work. He says GPT-3.5 Turbo is now both relatively expensive and poor by comparison. The practical result is that engineers have more viable models to choose from instead of defaulting to OpenAI.

### Benchmark scores do not measure how models feel to use
[02:48](https://www.youtube.com/watch?v=eTTMUWP5B0s&t=168s)
The MMLU benchmark is popular because many model providers report results on it, but Willison says it mainly tests broad trivia knowledge. He gives a question about Type II-A supernovae and points out that most AI applications do not require that kind of knowledge. For a more practical comparison, he uses the Chatbot Arena, where people anonymously compare model responses to the same prompts and rankings are calculated with an Elo-style system. That ranking places Claude alongside GPT-4-class models and shows openly licensed models such as Llama 3 70B competing near the top.

### GPT-4-class capability is becoming a commodity
[10:33](https://www.youtube.com/watch?v=eTTMUWP5B0s&t=633s)
Willison says the GPT-4 barrier has been broken. Several organizations now compete in the same broad quality range, and he expects these systems to become faster and cheaper as competition continues. He points out that Llama 3 70B can fit on a hard drive and run on his Mac. GPT-4o and Claude 3.5 Sonnet are effectively free for consumers at the time of the talk, so more people can experience both the impressive and unreliable sides of advanced models. Access is becoming less of a constraint, while knowing how to use the systems remains difficult.

### AI applications require hidden operational knowledge
[07:53](https://www.youtube.com/watch?v=eTTMUWP5B0s&t=473s)
Willison uses PDF uploads to show why ChatGPT is harder to use than it appears. A PDF needs searchable text to work reliably, and long files may be handled through an internal search process rather than being placed directly into the prompt. Tables and diagrams may be processed incorrectly, while screenshots of those elements work better through GPT vision. ChatGPT may also use Code Interpreter and packages such as fpdf, pdf2image, or pypdf. Willison learned this by scraping the package list with GitHub Actions because OpenAI does not publish enough detail about the feature. He compares this experience to mastering Excel, where basic use is easy but expert use takes years.

### AI products have a serious trust problem
[10:17](https://www.youtube.com/watch?v=eTTMUWP5B0s&t=617s)
Willison describes public reactions to AI features added by Dropbox and Slack. Users feared that their private data was being used to train models because of confusing terms and default settings. He says neither company was training models on customer data, and both had agreements that prevented OpenAI from training on the data sent to it. That explanation is hard to communicate after users have already lost confidence. Anthropic made a direct statement that it had not used customer or user-submitted data to train its generative models. Willison also points out that model providers still face criticism for training on unlicensed web scrapes.

### Rendering untrusted markdown can leak private data
[12:46](https://www.youtube.com/watch?v=eTTMUWP5B0s&t=766s)
Willison presents a prompt-injection example he calls the markdown image exfiltration bug. If a chatbot can access private information and render markdown images, an attacker can ask it to create an image URL whose query string contains a secret. Loading the image sends that secret to the attacker's server, and the image can be invisible. He says this pattern appeared in ChatGPT, Google Bard, Writer.com, Amazon Q, Google NotebookLM, and GitHub Copilot Chat. His direct engineering advice is not to render markdown images in this way. The repeated mistake shows why chatbot features need security review around untrusted instructions and output.

### Language models are gullible because they follow information from any source
[14:19](https://www.youtube.com/watch?v=eTTMUWP5B0s&t=859s)
Prompt injection can produce security failures, but it can also create strange answers. Willison describes a retrieval application that answered a question about the meaning of life in the voice of a witty gerbil. The source documentation contained an example prompt telling a model to pretend to be a witty gerbil and discuss snacks. Semantic search found that passage as the closest match, and the model followed it. Willison says models are designed to believe information supplied by their users, but they can also accept instructions embedded in retrieved material. Systems that make decisions from unverified context can therefore be manipulated.

### Slop is unrequested and unreviewed generated content
[15:31](https://www.youtube.com/watch?v=eTTMUWP5B0s&t=931s)
Willison defines slop as AI-generated content that is both unrequested and unreviewed. Asking Claude for information is not slop, and publishing AI-assisted writing is acceptable when a person checks it. Publishing whatever comes out of a prompt without review is different because it adds unreliable material to the internet. He compares the term to spam, which helped make unwanted marketing messages recognizable as bad behavior. His concern is accountability. A person can attach their reputation to published work and verify that it deserves to exist. A language model cannot take responsibility for the quality or truth of its output.

## Notable quotes
- "The GPT-4 barrier has been decimated." (10:33)
- "Tools like ChatGPT generally they're power user tools, they reward power users." (09:46)
- "The key thing here problem here is that LLMs are gullible." (15:15)
- "My definition of slop is this: this is anything that is AI generated content that is both unrequested and unreviewed." (15:31)
- "We as humans can do that and so if you're using a language model to help you publish, great text, fantastic, provided you're reviewing that text." (16:50)

## Tools & references mentioned
- Datasette
- SQLite
- GPT-4
- GPT-4o
- Claude 3.5 Sonnet
- Claude 3 Haiku
- Gemini 1.5 Pro
- Gemini 1.5 Flash
- GPT-3.5 Turbo
- MMLU
- Chatbot Arena
- Llama 3 70B
- Nvidia
- Cohere
- Alibaba
- DeepSeek AI
- Ethan Mollick
- Dropbox
- Slack
- OpenAI
- Anthropic
- GitHub Copilot Chat
- Google Bard
- Writer.com
- Amazon Q
- Google NotebookLM
- GitHub Actions

## Who should watch
- You are choosing models for an application and want quality, cost, and open licensing to be part of the decision.
- You are building a chatbot that retrieves documents, renders markdown, or handles private data and need concrete prompt-injection failure modes to test.
- You publish AI-assisted writing or product content and need a clear standard for when review and human responsibility are required.

## Related talks

- [Open Questions for AI Engineering](https://aietalks.com/talks/open-questions-for-ai-engineering) (Simon Willison, Independent open source developer, 24:33)
- [Keynotes & CodeGen Track](https://aietalks.com/talks/keynotes-codegen-track) (Benjamin Duny, AI Engineer & Simon Willison, Independent & Steven Hood & Justine Tunney, Mozilla & Jamie Turner, Convex & Tanmai Gopal, Hasura & Kevin Van Gundy, Codium & Nicholas Schlapfer, Hyperspace & Adi Adani & Akilesh Gupta, Harness Care & Greg Brockman, High High Labs & Maxime Labonne, Kagi & Afarin Khan, Timescale & Rahul Panda, GitHub Next & Michael Truell, Cursor & Quinn Slack, Sourcegraph & Morgante Pel, Grit & Gunjun Patel, Palo Alto Networks & SaaS Nita, Emergence AI & Changlu Chen, Midjourney & Eiso Kant, Spreadsheets Are All You Need & Jerry Liu, LlamaIndex & Swix, Small AI & Paul Henry, BotDojo & Joselyn Matthews, Pinecone, 8:38:54)
- [Local Models: Trust, Control, Optimization](https://aietalks.com/talks/local-models-trust-control-optimization) (Carter Abdallah, NVIDIA & Vincent Weisser, Prime Intellect & Lucas Atkins, Arcee AI & Chris Alexiuk, NVIDIA, 43:21)
- [The Wild World of AI: 6 Months That Changed Everything](https://aietalks.com/talks/the-wild-world-of-ai-6-months-that-changed-everything) (Simon Wilson, 01:57)
- [Trends Across the AI Frontier](https://aietalks.com/talks/trends-across-the-ai-frontier) (George Cameron, ArtificialAnalysis.ai, 17:52)