# Keynotes & CodeGen Track

Benjamin Duny, AI Engineer & Simon Willison, Independent & Steven Hood & Justine Tunney, Mozilla & Jamie Turner, Convex & Tanmai Gopal, Hasura & Kevin Van Gundy, Codium & Nicholas Schlapfer, Hyperspace & Adi Adani & Akilesh Gupta, Harness Care & Greg Brockman, High High Labs & Maxime Labonne, Kagi & Afarin Khan, Timescale & Rahul Panda, GitHub Next & Michael Truell, Cursor & Quinn Slack, Sourcegraph & Morgante Pel, Grit & Gunjun Patel, Palo Alto Networks & SaaS Nita, Emergence AI & Changlu Chen, Midjourney & Eiso Kant, Spreadsheets Are All You Need & Jerry Liu, LlamaIndex & Swix, Small AI & Paul Henry, BotDojo & Joselyn Matthews, Pinecone | AI Engineer World's Fair 2024 | 8:38:54

Source: https://www.youtube.com/watch?v=5zE2sMka620
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/keynotes-codegen-track
Published: 2024-06-27
Tags: code-generation, inference, open-source, rag

## TL;DR
- GPT-4-class models became widely available and competitively priced, but Simon Willison argues that using them well still requires substantial experience and careful judgment.
- Local AI can run on ordinary CPUs, and Mozilla's llama.cpp and llama file work focuses on making private, portable inference faster without requiring a cloud connection.
- AI coding tools are moving from autocomplete toward repository-wide edits, production workflows, and constrained agents, while developers still need control, tests, retrieval, and review.

## Summary
This compilation covers the opening keynotes and CodeGen sessions from AI Engineer World's Fair 2024. Simon Willison describes the end of the GPT-4 barrier, the rise of capable open models, and the practical difficulties of using chatbots, including PDF handling, prompt injection, privacy confusion, and unreviewed AI-generated content. Mozilla's Steven Hood and Justine Tunney present llama file, portable local inference, and CPU performance work. Other talks cover reactive backends for AI workflows, access-controlled data tools, email bots, query-aware retrieval, Postgres vector search, and generated educational videos. The CodeGen track then examines GitHub Next's next-edit suggestions and Copilot Workspace, Codium's retrieval system, Cursor's AI-native editor, Grit's repository-scale migrations, and CI-based security review. Later keynotes discuss multi-agent systems, dynamic interfaces, model internals in Excel, knowledge assistants, and the boundaries of AI engineering.

## Key ideas
### GPT-4-class models became a commodity, but model benchmarks still miss practical quality
[33:33](https://www.youtube.com/watch?v=5zE2sMka620&t=2013s)
Simon Willison says GPT-4 had no serious competition for about 12 months, then several models reached what he calls GPT-4 class. He groups Gemini 1.5 Pro, GPT-4o, and Claude 3.5 Sonnet among the strongest models, while Claude 3 Haiku and Gemini 1.5 Flash offer much lower-cost options. He criticizes MMLU because questions such as identifying the correct statement about a Type IIa supernova measure trivia knowledge rather than usefulness. For practical comparison, he prefers the LM Arena, where users compare anonymous model responses and rankings emerge from those votes. Open models including Llama 3 70B, Command R+, and models from Alibaba and DeepSeek also appear near the top.

### Language-model tools reward experience and expose users to hidden failure modes
[41:21](https://www.youtube.com/watch?v=5zE2sMka620&t=2481s)
Willison argues that ChatGPT is a power-user tool even though its interface looks simple. A PDF must be searchable, long PDFs are handled differently from short ones, and tables or diagrams can be processed incorrectly unless users upload screenshots instead. He learned about Code Interpreter's PDF capabilities by scraping its installed packages with GitHub Actions because the details were not documented. He compares this to Excel, where basic use is easy but mastery takes years. The same experience gap appears in prompt injection. A chatbot that renders Markdown images can leak private data through an image URL, and Willison says he has seen the same mistake in ChatGPT, Google, Writer, Amazon Q, NotebookLM, and GitHub Copilot Chat.

### Responsible AI use requires accountability, privacy clarity, and review
[48:21](https://www.youtube.com/watch?v=5zE2sMka620&t=2901s)
Willison defines slop as AI-generated content that is both unrequested and unreviewed. He accepts AI assistance when a person checks the result and takes responsibility for publishing it, including when someone uses a model to improve English writing. He also describes an AI trust crisis around Dropbox and Slack, where confusing wording and default settings led users to believe private data was being used to train models, even though the companies were not training models on that data. His warning is that language models cannot stake their reputation on an answer. Engineers need to establish patterns for responsible use and help other users understand where the systems work and where they fail.

### Local inference can be portable, private, and fast enough on CPUs
[52:58](https://www.youtube.com/watch?v=5zE2sMka620&t=3178s)
Steven Hood and Justine Tunney present llama file, a Mozilla open-source project that packages model weights and a program into a single executable. The same file can run across operating systems, CPU architectures, and GPU architectures, using whatever hardware is available. They argue that dependence on GPUs creates cost, supply, and energy problems, while CPUs are widespread and often have substantial RAM. Their work with llama.cpp produced reported speed increases ranging from 30% to 500%, depending on the CPU, model, and weights. Tunney explains an outer-loop unrolling technique for matrix multiplication, CPU implementations of GPU-style synchronization, and a demo where summarization becomes much faster. Mozilla also announces SQLite vector search and a $100,000 non-dilutive accelerator for local AI projects.

### AI applications need data access and backend workflows that update with the application
[1:10:48](https://www.youtube.com/watch?v=5zE2sMka620&t=4248s)
Jamie Turner describes Convex as a reactive backend that tracks data flow and dependencies across queries and mutations. This lets server-side actions participate in the same update model as the application. His example chains speech recognition, summarization, embeddings, and related-note lookup, with results arriving in the interface as each step completes. Tanmai Gopal makes a related argument for exposing live data and business logic as tools to an LLM. His examples include calendar analysis, Salesforce deal diagnosis, support-ticket context, and product delivery questions. He proposes a unified query language, shared authorization rules based on data and session properties, and Python execution so the model can plan retrieval across several systems.

### Repository-scale coding needs richer retrieval than a single embedding lookup
[2:45:04](https://www.youtube.com/watch?v=5zE2sMka620&t=9904s)
Kevin Van Gundy says Codium's code-generation quality depends on retrieving context from design-system components, nearby forms, style guides, dependencies, and documentation. He argues that long context is slow and expensive, fine-tuning needs continual updates and separate models, and ordinary embeddings struggle to reason over multiple relevant files. Codium evaluates recall of the top 50 retrieved items against production-like ground truth built from pull requests and changed files. Its M Query system makes parallel model calls over repository items instead of relying only on vector distance, then combines signals such as active files, directories, and recent commits. The company trains its own models and infrastructure so it can spend more compute on retrieval while still streaming results quickly.

### AI coding agents work better when developers retain control and the system validates changes
[2:26:36](https://www.youtube.com/watch?v=5zE2sMka620&t=8796s)
Rahul Panda presents GitHub Next's exploration process, where ideas move through internal dogfooding, company testing, technical previews, and possible productization. Next Edit Suggestions predict edits beyond the current cursor, such as updating a method definition, arguments, and documentation after a parameter changes. Copilot Workspace expands from code completion to an inner loop of specification, planning, implementation, and verification. It generates a proposal from an issue, lets the developer edit the plan, runs commands in an integrated terminal, and provides a preview. Michael Truell describes Cursor's similar direction: an AI-native editor with next-edit prediction, Command-K code changes, repository context, and constrained background agents. Both talks keep the programmer in charge.

### Reliable agents need static analysis, checkpoints, tests, and controlled scope
[5:33:59](https://www.youtube.com/watch?v=5zE2sMka620&t=20039s)
Morgante Pel says Grit focuses on raising the ceiling for experienced engineers by coordinating changes across many repositories. Its migration workflow indexes semantic and structural relationships, plans changes, delegates file edits to agents, and opens pull requests for review. GritQL combines syntax, semantic similarity, and import-graph analysis to find the right code, such as logs that should move to OpenTelemetry. Pel stresses that generated code must go through compilers, type checkers, linters, and tests. Since large builds can take many minutes, Grit uses language-server-style incremental checks and Firecracker snapshots to fork several precomputed environments. Gunjun Patel presents a related CI workflow, or 'ghost pilot', that improves comments, establishes unit-test baselines, debates security findings with several role-based agents, proposes fixes, and leaves the final decision to a human reviewer.

## Notable quotes
- "Tools like ChatGPT genuinely, they're power user tools. They reward power users." (43:33)
- "The key thing is it's not just the big folks who can solve these problems. It's individuals and small groups working together in open source." (53:58)
- "Developers are in control, not the other way around." (4:06:43)
- "Compilers rock." (5:44:04)
- "If you're building a product and you are not using it every single day, it is not going to make it there." (5:24:35)

## Tools & references mentioned
- GPT-4
- GPT-4o
- GPT-3.5 Turbo
- Gemini 1.5 Pro
- Gemini 1.5 Flash
- Claude 3.5 Sonnet
- Claude 3 Haiku
- Llama 3 70B
- Command R+
- MMLU
- LM Arena
- Microsoft Bing
- Anthropic
- OpenAI
- Dropbox
- Slack
- GitHub Copilot Chat
- llama file
- llama.cpp
- Cosmopolitan
- TinyBLAS
- Hugging Face
- Mozilla Builders
- SQLite Vec
- Convex
- Hasura
- Hypermode
- Hyperspace
- Puppeteer
- Beautiful Soup
- Math Matrix Movies
- Manim
- Cairo
- FFmpeg
- Azure TTS
- Harness Care
- Postmark
- FastAPI
- Celery
- OpenAI Assistants API
- Kagi
- PGVector
- PGVector Scale
- Pinecone
- Streaming DiskANN
- Statistical Binary Quantization
- GitHub Next
- Copilot Workspace
- Codium
- M Query
- Cursor
- Copilot Plus+
- Command-K
- Sourcegraph Cody
- Grit
- GritQL
- OpenTelemetry
- Firecracker
- Emergence AI
- Agent E
- LlamaIndex
- LlamaParse
- Llama Agents
- GPT-2
- Anthropic Golden Gate Claude
- Sparse autoencoders
- Activation steering
- Representation engineering
- RWKV
- Smalltalk
- Andrew Ng
- Ethan Mollick
- Karina Win
- Peter GEV
- Johan Rehberger
- Grant Sanderson
- 3Blue1Brown

## Who should watch
- You are evaluating which model, retrieval, or agent architecture to use in a production application and want practical failure cases rather than benchmark scores alone.
- Your team is building code assistants and needs ideas for repository context, edit prediction, validation loops, or multi-repository migrations.
- You want private local inference, CPU optimization, or open-source alternatives to hosted AI services.

## Related talks

- [Open Questions for AI Engineering](https://aietalks.com/talks/open-questions-for-ai-engineering) (Simon Willison, Independent open source developer, 24:33)
- [Open Challenges for AI Engineering](https://aietalks.com/talks/open-challenges-for-ai-engineering) (Simon Willison, Datasette, 18:49)
- [Llamafile: Bringing AI to the Masses with Fast CPU Inference](https://aietalks.com/talks/llamafile-bringing-ai-to-the-masses-with-fast-cpu-inference) (Stephen Hood & Justine Tunney, Mozilla, 17:25)
- [AI Engineer Summit 2023, Day 2 Livestream](https://aietalks.com/talks/ai-engineer-summit-2023-day-2-livestream) (Mario Rodriguez, GitHub & Dedy Kredo, CodiumAI & Matt Welsh, Fixie.ai & Amelia Wattenberger, Adept & Samantha Whitmore & Jason Yuan, New Computer & Joseph Nelson, Roboflow & Hassan El Mghari, Vercel & Paul Copplestone, Supabase & Daniel Rosenwasser, Microsoft & Jason Liu, Fivesixseven & Anton Troynikov, Chroma & Jerry Liu, LlamaIndex & Mithun Hunsur, Ambient & Abi Aryan, O'Reilly & Simon Willison, Datasette & Benjamin Dunphy, Software 3.0 LLC & swyx, Latent.Space & Smol.ai, 7:30:56)
- [AI Engineer Summit 2023, Day 1 Livestream](https://aietalks.com/talks/ai-engineer-summit-2023-day-1-livestream) (Benjamin Dunphy, Software 3.0 LLC & swyx, Latent.Space & Smol.ai & Amjad Masad & Michele Catasta, Replit & Toran Bruce Richards, AutoGPT & Simón Fishman & Logan Kilpatrick, OpenAI & Flo Crivello, Lindy & Barr Yaron, Amplify & Sasha Sheng & Harrison Chase, LangChain & Shreya Rajpal, Guardrails AI & Eugene Yan, Amazon & Linus Lee, Notion & Brittany Walker, CRV & Chris White, Prefect & Bryan Bischof, Hex, 4:50:00)