# Keynotes & Multimodality Track

Benjamin Duny & Simon Sturmer, AI Engineer & Chris Lattner, Modular & Auntie Aart & Mike Chambers, AWS & Alex Albert, Anthropic & Harrison Chase, LangChain & Rob Chandra, Substrate & Vic Kapati, M87 Labs & Ben Hilac, Dawn & Karan Goel, Cartesia & Chang She, LanceDB & Noah Schnapp, Character.AI & Steph Duenas & Quinn Dombrowski, Daily & Roman Huet, OpenAI & Brian Bischof, Charles Fry, Hamza Hussein, Jason Liu, Shreya Shankar & Eugene Yan & Thomas Dohmke, GitHub | AI Engineer World's Fair 2024 | 8:34:14

Source: https://www.youtube.com/watch?v=vaIiNZoXymg
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/keynotes-multimodality-track
Published: 2024-06-28
Tags: design, evals, inference, multimodal

## TL;DR
- AI applications need control over data, cost, hardware, and model behavior, which is driving new inference stacks such as Modular's Max and Mojo.
- Multimodal products work better when they combine models with explicit structure, evaluation, and familiar interfaces instead of putting everything into an unstructured chat.
- Real-time voice and vision systems are limited by latency, data pipelines, and orchestration, while smaller specialized models can make deployment cheaper and more private.

## Summary
This compilation covers the AI Engineer World's Fair 2024 keynotes and multimodality track. Chris Lattner argues that fragmented inference frameworks make it hard to move research into affordable production systems, and presents Modular's Max framework and Mojo language. AWS shows a path from software development to AI engineering through model selection, Amazon Q, Bedrock, the Converse API, and agents. Anthropic presents Claude 3.5 Sonnet, Artifacts, Projects, tool use, and model steering research. Harrison Chase explains why production agents need custom cognitive architectures, persistence, human approval, and debugging tools through LangGraph Cloud and Studio. The multimodality talks cover Substrate's graph-based inference, MoonDream's small open vision model, product design for unbounded interfaces, state space models for long real-time contexts, training data infrastructure, education, and voice agents. Later talks from OpenAI, GitHub, and other builders connect multimodal models with product design, developer workflows, evals, and production operations.

## Key ideas
### Inference fragmentation keeps AI prototypes from becoming economical products
[21:57](https://www.youtube.com/watch?v=vaIiNZoXymg&t=1317s)
Chris Lattner says hosted endpoints make it easy to prototype, but teams may need to keep data under their control, integrate company security, customize models, reduce inference costs, or target unusual hardware. The stack has shifted from PyTorch and TensorFlow toward ONNX, TensorRT, and many model-specific frameworks, leaving engineers to switch technologies across deployments. Modular's Max aims to combine these layers into one inference-focused stack that works with Python and existing PyTorch models. Lattner says Max can use native APIs for advanced developers while offering compiler and runtime features to others. He gives an example of an INT4 and INT8 quantization release that he says is 5x faster than llama.cpp on cloud CPUs.

### Models become more useful when product teams redesign the interface around them
[1:00:21](https://www.youtube.com/watch?v=vaIiNZoXymg&t=3621s)
Alex Albert compares AI product design with the early electrical and mobile revolutions. Replacing a steam engine with an electric motor did not change a factory until factories were redesigned around electricity. He says many AI products are still adding a star button or chat box to an existing product. Claude 3.5 Sonnet, which he describes as stronger than Claude 3 Opus, supports coding, retrieval, tool use, and vision tasks. Anthropic's Artifacts separates generated work from the chat so users can edit and iterate on SVGs, React sites, and other outputs. Projects adds shared context from codebases, style guides, transcripts, and prior work. Albert's point is that structure around the model creates product experiences that feel designed for AI.

### Production agents need explicit cognitive architectures and infrastructure around them
[1:14:17](https://www.youtube.com/watch?v=vaIiNZoXymg&t=4457s)
Harrison Chase traces the move from the ReAct paper, LangChain, and AutoGPT to systems designed for production. Generic agent architectures were often unreliable, so companies built custom cognitive architectures that encode how their agents should behave. LangGraph is deliberately low-level and controllable, with persistence, streaming, and human-in-the-loop interaction. LangGraph Cloud adds an agent API, background runs, configurable graph instances, cron jobs, handling for overlapping messages, and breakpoints for approval before tool calls. Users can also edit an earlier step and resume the run. LangGraph Studio displays the graph, streams steps and tokens, and supports editing, replay, and breakpoints. Chase describes the framework as the application-specific logic, with cloud infrastructure handling repeated operational work.

### Small vision models can win by narrowing their goal and using better data
[2:26:04](https://www.youtube.com/watch?v=vaIiNZoXymg&t=8764s)
Vic Kapati presents MoonDream, an open-source vision-language model with fewer than two billion parameters. It focuses on understanding images, answering questions, captioning, locating objects, and counting, rather than general world knowledge or solving mathematical problems. Kapati combines Google's SigLIP vision encoder with Microsoft's Phi-1.5 text model and says the latest release uses about 35 million images. He stresses that training data mattered more than swapping model backbones. Synthetic data can be useful, but blindly asking a powerful model such as GPT-4 to generate captions can add hallucinated details. MoonDream's pipeline introduced noise, capitalization problems, distractor questions, and other variations so its training distribution better matched real queries. Kapati expects smaller models to be useful where video cost, privacy, or latency make large hosted models impractical.

### Real-time multimodal systems need model compression and low-latency orchestration
[3:04:32](https://www.youtube.com/watch?v=vaIiNZoXymg&t=11072s)
Karan Goel describes applications that continuously process audio, video, and sensor streams, including voice assistants, on-device systems, generated worlds, and robotics. These workloads differ from batch intelligence because they must respond instantly. He argues that transformer attention becomes expensive as multimodal context grows, since the model retains and revisits a long history. State space models instead update an internal memory as tokens arrive and discard the input tokens, giving them a streaming interface and more linear scaling with context length. Goel says this compression can help with long-context tasks such as understanding a day of security-camera footage. Cartesia is building real-time foundation models around this idea, including a low-latency voice generation model that can run in the data center or on devices.

### Voice agents are systems problems where latency is felt by the user
[5:30:34](https://www.youtube.com/watch?v=vaIiNZoXymg&t=19834s)
Quinn Dombrowski says a useful voice agent combines audio capture, echo cancellation, network transport, transcription, turn detection, model inference, tool calls, and speech generation. Users notice latency in conversation much more than throughput. He gives roughly half a second as an important human conversational target, while noting that a typical voice pipeline can reach 600 to 700 milliseconds even with optimistic component timings. Audio processing on a Mac can consume 30 to 40 milliseconds before application software begins. His practical recommendation is to colocate as much of the pipeline as possible in one compute container instead of making repeated hosted-service calls. Daily's open-source Pipecat framework supports conversational bots, speech translation, voice-controlled agents, and real-time vision workflows.

### Multimodal data systems must support filtering, shuffling, and large binary objects together
[5:21:01](https://www.youtube.com/watch?v=vaIiNZoXymg&t=19261s)
Chang She and Noah describe training data as a managed product rather than an opaque input file. Character uses analytics, clustering, retrieval, quality scoring, synthetic data, and human labeling to understand what belongs in training sets. Distributed training commonly filters rows, shuffles them, and streams text, images, or video into GPUs. The speakers argue that many existing formats handle only some of these needs well. Lance is designed for fast scans, random access, and large blobs, with versioning, schema changes without copying the original dataset, time travel, vector indexes, scalar indexes, and full-text search. This lets teams use one table for SQL analysis, embedding search, retrieval, and model training. Noah also emphasizes that materializing datasets separately from training jobs makes iteration less error-prone.

### Evals and user data are the mechanism for improving uncertain AI systems
[7:31:28](https://www.youtube.com/watch?v=vaIiNZoXymg&t=27088s)
The authors of the production lessons talk argue that teams should build what model providers will not, collect user interactions, and improve continuously. Evals are treated as objectives for the product, not as decorative metrics. Simple assertions can check fields such as title, price, table names, or expected columns, while model-based evaluators can score properties such as relevance and factual consistency. LLM judges are quick to prototype, but fine-tuned classifiers can be faster and easier to align once enough labeled examples exist. The speakers advise looking at production data regularly, grouping failures into recognizable slices, pinning model versions, and tracking code, prompt, and model changes. Guardrails can check toxicity, personal information, copyright, language, and hallucination. Their warning is direct: a demo can be easy to build while turning it into a product takes sustained measurement and operational work.

### Multimodal interfaces should give users structure, familiarity, and room to revise
[3:05:49](https://www.youtube.com/watch?v=vaIiNZoXymg&t=11149s)
Ben Hilac draws on his work designing visionOS to explain how products can handle open-ended input without becoming confusing. Unbounded products create many 'what if' situations, so designers need to add structure. He recommends putting important functions in front of users quickly, establishing hierarchy, and using familiar patterns. He applies those ideas to AI products such as Dot, Perplexity, v0, Clay, and Claude Artifacts. Hilac criticizes interfaces that place temporary controls inside a chat stream, where they drift away as the conversation grows. He prefers moving the structured work area outside the conversation and preserving versions so users can revise safely. He expects future products to replace fragile prompt engineering with ranked presets and developer-defined personalization, with analytics becoming more useful than a single universal correctness score.

## Notable quotes
- "The fragmentation slows down getting the research and the Innovations coming into gen into your products." (24:13)
- "The training data is the biggest needle mover in terms of model performance." (2:26:17)
- "Latency is mostly what bites us here." (5:58:47)
- "The model is actually not your moat for almost no one in this audience." (7:36:19)
- "Without evals you can't make progress quickly." (7:52:41)

## Tools & references mentioned
- Modular
- Max
- Mojo
- PyTorch
- TensorFlow
- ONNX
- TensorRT
- llama.cpp
- Amazon Q Developer
- Amazon Bedrock
- Converse API
- Claude 3.5 Sonnet
- Claude 3 Opus
- Artifacts
- Projects
- LangGraph
- LangGraph Cloud
- LangGraph Studio
- MoonDream
- SigLIP
- Phi-1.5
- Cartesia
- LanceDB
- Lance format
- Pipecat
- OpenAI
- GPT-4
- GPT-4o
- Sora
- Voice Engine
- GitHub Copilot
- GitHub Copilot Workspace
- Character.AI
- Dawn
- Vision Pro
- visionOS
- ReAct paper
- AutoGPT
- AI Dungeon
- MLflow
- Toyota Production System
- Kaizen
- Deepgram
- Llama 3
- Hugging Face
- Scratch
- Cognimates

## Who should watch
- You are moving from an AI prototype to a production system and need to choose between hosted APIs, custom inference, or a mixed stack.
- Your product accepts images, audio, video, or open-ended natural-language input, and you need interface patterns that keep users oriented.
- You are building agents or voice systems and need practical guidance on persistence, human approval, evaluation, data pipelines, and latency.

## Related talks

- [See, Hear, Speak, Draw](https://aietalks.com/talks/see-hear-speak-draw) (Logan Kilpatrick & Simón Fishman, OpenAI, 18:43)
- [From Text to Vision to Voice: Exploring Multimodality with OpenAI](https://aietalks.com/talks/from-text-to-vision-to-voice-exploring-multimodality-with-openai) (Romain Huet, OpenAI, 23:39)
- [The Era of Unbounded Products: Designing for Multimodal IO](https://aietalks.com/talks/the-era-of-unbounded-products-designing-for-multimodal-io) (Ben Hylak, Dawn, 20:32)
- [Multi model multimodal and multi agent innovations in Azure AI](https://aietalks.com/talks/multi-model-multimodal-and-multi-agent-innovations-in-azure-ai) (Cedric Vidal, Microsoft, 28:56)
- [The Multimodal Future of Education](https://aietalks.com/talks/the-multimodal-future-of-education) (Stefania Druga, Google Gemini team, 20:05)