Language models work best as language interfaces to existing software and services, with retrieval supplying the context they lack.
2
Structured outputs let language models connect to software through schemas, function calls, retries, and grammar-based sampling.
3
AI engineering requires shipping into production, monitoring real use, and building evaluations from observed failures.
Summary
Charles Frye argues that language model inference is only the interior of an AI product. The surrounding engineering includes user interfaces, retrieval, tool use, agents, monitoring, and evaluation. He describes language user interfaces as a new way to control existing systems, then explains why retrieval augmented generation is mainly an information retrieval problem. For tool use, he focuses on forcing text models to produce structured outputs, including JSON schemas and grammar-constrained generation. Agents add memory, skills, tools, and decision procedures, but reliability remains difficult. Frye's practical advice is to ship early and learn from production data. Monitoring should cover user behavior, latency, throughput, errors, and cost. Evaluation is difficult for open-ended text, so teams should use reference answers, comparisons, human feedback, decomposition, and real user behavior whenever possible. The talk is candid about how incomplete the engineering process still is.
Language models are becoming language user interfaces
Frye describes current AI products as language user interfaces, or LUIs, by analogy with graphical user interfaces. A terminal required users to learn a special language, while a GUI let people apply visual intuition. Language models offer a similar change by letting users say what they want in ordinary language. He points to query assistants, Alexa, ELIZA, and Ask Jeeves as earlier attempts at this idea. Honeycomb's assistant, for example, lets someone ask to see slow requests or latency by status code instead of writing a query. Frye expects language interfaces to eventually shape products in the way graphical interfaces changed computers.
RAG is mainly an information retrieval system around a model
Frye calls retrieval augmented generation chatbots the starter project for language user interfaces. A model may have absorbed broad internet knowledge, but it lacks information about a particular person, company, or application. The application therefore collects and indexes context, searches for relevant information, and inserts the retrieved material into the prompt. Vector search became popular because embeddings were readily available, but Frye says the lasting pattern is information retrieval itself. Systems may combine vector search with keyword search, metadata extraction, and direct filtering. Pinecone, Chroma, Elasticsearch, Redis, and Postgres appear in his discussion. The difficult work is often ordinary database and search engineering.
Retrieval and fine-tuning solve different problems, within limits
Frye accepts the common rule that fine-tuning is mainly for style while retrieval is mainly for information, especially for low-rank fine-tuning on small datasets. He explains that such fine-tuning can reorder or emphasize computations already present in the model, such as making a Homer Simpson simulation more important for a specialized chatbot. That rule stops applying when training uses a much larger corpus, such as textbooks, because the model can learn more knowledge. He also discusses knowledge graphs. A graph can sometimes be represented in Postgres, and graph sharding is difficult because arbitrary links do not divide cleanly. For language applications, however, the database may only need to act as external memory at a moderate scale.
Structured outputs connect language models to software
Language models produce unstructured text, while software systems need data with a defined shape. Frye presents structured outputs as a way to connect model output to other systems and make models more useful as interfaces. Developers can ask for JSON through prompting, fine-tune models for tool use, retry invalid responses, or constrain generation with a grammar. Grammar-based sampling sets the probability of invalid tokens to zero, which can enforce JSON, code, or another format. OpenAI's function calling uses JSON Schema so a model can describe calls to many tools through a common format. Frye also points out that the function can be fictional. A schema for a spam classification function can force the model to emit structured classification data without any real function call.
Agents combine model knowledge with memory and decision procedures
Frye says agents are closer to the popular idea of artificial intelligence because they can retain memories, use tools, and act over time. He discusses Generative Agents in a simulated environment and Voyager, which writes JavaScript routines to operate in Minecraft, accumulate skills, and create a curriculum. These systems combine the world knowledge and common sense of language models with ideas from older cognitive architectures. The cited work on cognitive architectures for language agents includes episodic, semantic, and procedural memory, internal actions, and explicit decision procedures. Frye sees this combination as a way to organize the many existing agent designs. He is also clear that reliability remains the largest problem, and that the field has many competing techniques.
AI products improve through production use rather than finished tests
Frye recommends shipping to learn. He connects this to Andre Karpathy's idea of a data engine or data flywheel: collect data from real use, find problems, and use those problems to improve the system. In traditional software, teams often write tests before production. With machine learning systems, production use can reveal the cases that should become tests. Frye gives his Discord bot as an example. Users asked recurring meta-questions, including whether the bot received feedback or whether its dataset included its source code. Those observations led him to add special handling. Monitoring user behavior therefore produces product information as well as bug reports.
Observability requires enough evidence to fix model failures
Monitoring should cover user behavior, latency quantiles, throughput, errors, performance, and cost. Frye warns that the 99th percentile matters because users make many requests in sequence, so a low fraction of slow requests can still affect a session. Inference providers need API rates and errors, while self-hosted systems also need compute utilization. He distinguishes monitoring from observability. Observability asks whether the team can work out what is happening inside a system from its external outputs and logs. This matters for language models because engineers cannot step through the internal layers of a model with a live debugger. Tools he names include Datadog, Sentry, New Relic, Honeycomb, OpenTelemetry, Weights & Biases, Fiddler, Arize, Gantry, LangSmith, and Langfuse.
Evaluation starts with the strongest available source of truth
Frye says evaluation is hard because teams may not know whether an open-ended answer is right or whether a change improved it. He recommends first checking whether a task has a correct answer, a reference answer, a previous system answer, or human feedback. Those allow ordinary metrics, reference matching, comparisons, or feedback checks. If none exists, teams enter a less structured evaluation problem. Elicit's method of iterated decomposition addresses this by splitting an end-to-end task when a failure can be isolated in a simpler subtask. Production observations can gradually become regression tests. User behavior can reveal preferences without forcing people to fill out forms. Frye also says language models can annotate textual data at lower cost than crowd workers in some settings, with GPT-3.5 Turbo comparable to a median crowd worker and GPT-4 comparable to a 90th-percentile crowd worker.
"The dominant ideology right now is that you should ship to learn rather than learning to ship."41:26
Who should watch
You are building a chatbot or assistant that needs private documents, application data, or reliable connections to external tools.
Your model works in a demo, but you lack a plan for monitoring latency, cost, user behavior, and production failures.
You need a practical way to evaluate open-ended model output and turn real failures into regression tests.
