LLM applications should build around product expertise, user value, and continuous improvement rather than treating the model as the moat.
2
Teams need evals and data literacy to move past an initial demo, and these skills can be learned through four to six weeks of deliberate practice.
3
Production LLM systems need simple evals, data inspection, pinned model versions, and guardrails around the model.
Summary
The talk collects lessons from six authors who combined their experience into an O'Reilly article on applied LLMs. Bryan Bischof argues that most teams should treat models like replaceable SaaS products and build in areas where they have product expertise. The durable work is creating a product that solves a job, maintains a quality bar as models change, and improves through user data and evals. Jason Liu and Hamel Husain focus on team design. They warn against buying tools or hiring an ML engineer before a company has data-generating products, and argue that the broad AI engineer role needs clearer expectations. Shreya Shankar and Eugene Yan then discuss practical evaluation and monitoring. They recommend breaking tasks into simple assertions, inspecting production data regularly, pinning model versions, and using guardrails for issues such as toxicity, personal information, unexpected language, and hallucinations. The talk is direct about the gap between an impressive demo and a product that can be maintained.
Bryan Bischof says most teams should not spend their effort training a custom model. They should use product expertise, an existing product, or a narrow market niche, then build what model providers cannot build for every customer. He compares models to SaaS products and says teams should drop them quickly when a competitor is clearly better. Model benchmarks do not prove product value: an 87% score on Spider SQL does not mean a system automates all data requests, and a human-evaluation score does not explain how to sell an application. The product still needs to solve a job and improve the user's experience.
Evals and data create a continuous improvement loop
Bischof describes evals and data as the center of a cycle that lets teams improve LLM applications. The same pattern appeared in MLOps, DevOps, the Lean Startup movement, and the Toyota production system's idea of Kaizen. Evals define what the system should do, while production data shows whether it delivers value. He warns that teams can become absorbed in building eval frameworks, calculating concept drift, and collecting tools while forgetting to ship something users can use. His practical advice is to put a beta in front of people, inspect real interactions, and use human review. Binary feedback is a useful starting point, while user requests can reveal product-market-fit opportunities.
Falling model costs make future products worth designing early
The speakers argue that teams can plan for the future by studying how capability and cost have changed. Bischof describes an order-of-magnitude cost decrease roughly every 12 to 18 months across several model generations, including Da Vinci, Text-DaVinci-002, GPT-4, and Claude 3. He gives a video-game example: in 2023, running a game whose non-player characters were powered by a chatbot cost about $625 per hour. If costs fell by two orders of magnitude over about 30 months, the experience could approach the inflation-adjusted $6-per-hour cost of playing Pac-Man in 1980. The product may not work economically today, but a team can design and test it before the economics change.
Buying more tools does not replace product and engineering processes
Jason Liu uses deliberately bad advice to criticize tool-driven development. If a retrieval application fails, buying another vector database or updating the embedding model will not by itself create product-market fit. He also mocks the idea of hiring an expensive ML engineer immediately to avoid a large API bill, then assigning them infrastructure problems before the company has a useful data-generating product. Changing tools, processes, and decision frameworks cannot substitute for learning what customers will pay for. The underlying point is that teams need to develop expertise and a repeatable way to make decisions instead of assuming the next tool will solve the business problem.
The AI engineer role needs clearer skills and timing
Hamel Husain says the AI engineer role is often overscoped and underspecified. The common skill diagram emphasizes chains, agents, tooling, and infrastructure, while leaving out evals and data. That may be enough to move from zero to one quickly with tools and vibe checks, but without measurement a team cannot make steady progress. Husain and Liu say four to six weeks of deliberate practice can produce useful skill in writing evals and working with data. They recommend matching hiring to product maturity: application development can get an early product off the ground, platform and data engineering can capture what happens, and an ML engineer should come later, after the company has data. They also compare this problem with the earlier data scientist role, which split into more specific jobs.
Simple assertions are a strong first evaluation method
Shreya Shankar recommends making evaluation tasks smaller and easier to check. For product-attribute extraction, a team can test the title, price, and rating separately. For summarization, it can evaluate dimensions such as factual consistency, relevance, and information density instead of treating the summary as one vague object. Natural-language-to-SQL systems can be checked for the expected table and columns. Assertions will not cover every quality question, but they can answer many concrete ones with little complexity. For harder criteria, teams can train a classifier or reward model, especially when the evaluation is naturally a classification or regression problem.
LLM judges are easy to start with but costly to align
Shankar says LLM-as-a-judge systems are easy to prototype because a team can write a prompt and provide examples of good and bad outputs. The difficulty is aligning the judge with a company's exact criteria. With a few hundred to a few thousand samples, she says it may be practical to fine-tune a smaller evaluator for more precise checks. A judge that uses chain-of-thought can take five to eight seconds, while a simple classifier or reward model may take about 10 milliseconds. Judges also need maintenance: their examples or prompts must stay aligned with the definition of good and bad, and fine-tuned validators need new production data to avoid drift. The right choice depends on resources, evaluation volume, and how established the product is.
Production systems need regular data review and guardrails
Eugene Yan says teams should inspect LLM outputs regularly rather than only during initial development. One startup created a Slack channel for live agent outputs, then moved to daily batch review after a couple of weeks. Useful analysis groups failures into identifiable slices, such as a source, keyword, phrase, or topic. Traces should include the relevant Git commit, model version, and prompt version, and API model versions should be pinned because behavior can change. Yan also recommends basic guardrails for toxicity, personally identifiable information, copyright, unexpected language, and hallucinations. Reference-free checks can compare a summary with its source document by testing entailment or contradiction, avoiding the need to collect an ideal answer for every case.
"The model is actually not your moat for almost no one in this audience."02:45
Who should watch
You are building an LLM product and need to decide where your effort belongs when model providers keep improving their APIs.
Your team has shipped a demo but lacks a way to evaluate outputs, inspect failures, or decide which role to hire next.
You are choosing between an LLM judge, a fine-tuned evaluator, and simpler production checks.
