Companies are already getting revenue and cost savings from LLM products in production, including Filevine, which launched six products and roughly doubled its revenue.
2
Successful teams rely more on generalist product engineers and domain experts than on researchers focused on model training.
3
Evaluation needs to run from prototyping through production, with user feedback, focused automated checks, human review, regression tests, and detailed logs.
Summary
Raza Habib compares companies that have put LLM products into production and found measurable business value. He argues that the applications themselves are usually simple pipelines made from a model, prompt, data selection strategy, and tools. The difficult work is improving each part. Successful teams include generalist engineers, domain experts who know what good output looks like, and enough machine learning knowledge to design representative tests. Evaluation starts during prototyping, continues through regression testing, and includes production monitoring. Habib recommends collecting user actions, issue reports, votes, and edits, then combining those signals with focused LLM judges, code-based metrics, and human review. He also describes tooling that lets teams collaborate on prompts, replay logged runs, debug agent traces, and turn failures into test cases. His examples include Filevine, Duolingo, Ironclad, Fathom, GitHub Copilot, Hex, and Vanta.
LLM products are already producing measurable business value
Habib argues that LLM work has moved beyond experiments and promised future value. He points to Filevine, a legal-sector customer operating in a sensitive regulated industry. The company launched six products in the previous year and roughly doubled its revenue. Habib uses this example to show that teams no longer need to ask whether generative AI can deliver value in principle. Some companies have already tied production products to revenue growth or cost savings. The example also matters because Filevine operates in a domain where accuracy and sensitivity might appear to make adoption harder.
Most LLM applications are simple pipelines with four parts
Habib reduces an LLM application to four components that can be chained in different ways: a base model, a prompt template, a data selection strategy, and optional tools such as function calls. Data may come from retrieval or an API. The hard part is making each component work well, rather than managing an inherently complicated architecture. He uses GitHub Copilot as an example. Its system uses a model fine-tuned for latency, selects relevant code from recently touched files, and places that code into context. Habib expects some current chains to become simpler as models improve at tool selection.
Teams need generalist engineers and domain experts more than model trainers
The teams Habib has seen succeed tend to contain generalist full-stack product engineers who understand prompting and models without focusing on training models. They also give domain experts a central role. These people can help write prompts, define evaluations, and judge outputs because they understand the work the product must support. Habib still recommends having someone with machine learning or data science knowledge who can teach the team about ideas such as representative test sets. He does not think teams generally need PhDs or people experienced in hardcore model training.
Domain experts can directly build parts of the product
Habib says LLMs let subject-matter experts contribute more directly to implementation. At Duolingo, linguists reportedly did much of the prompt engineering because they knew what good language output looked like. At Filevine, legal professionals helped prompt models, with those natural-language prompts becoming effectively production code. Fathom shows why this matters for meeting summaries. A useful summary depends on its audience and context, so a salesperson, a product manager, and an engineer may need different outputs. Habib says one Fathom product manager handled most of the prompting for different meeting-summary types.
Evaluation should begin during prototyping and continue in production
Habib treats evaluation as part of the product specification because it defines what good output means. During prototyping, teams should evolve their evaluation criteria alongside the application, often by putting a rough internal prototype in front of people quickly. As the product approaches production, those criteria can become more formal tests. In production, teams need monitoring and enough detail to investigate failures. A good evaluation setup also makes it safer to change prompts or switch models because the team can check for regressions. He says the best companies evaluate at every stage rather than adding evaluation after the application is built.
User feedback is valuable, but it needs several forms and does not replace testing
Habib says users know what is right, especially for subjective tasks such as summarization and question answering. GitHub Copilot collects signals including whether a suggestion was accepted and whether the suggested code remained in the codebase over time. Other applications use actions, issue reports, direct votes, and corrections or edits. These signals can guide later improvements, but they are lower volume and unavailable during development. Teams therefore need a separate evaluation scorecard for testing before release. Habib recommends designing feedback into the application from the start instead of treating it as an afterthought.
Evaluation works better when subjective goals are split into small checks
Habib warns that a broad question such as whether a piece of writing is good produces noisy results from an LLM judge. Teams should break subjective criteria into specific questions that can be tested independently. Examples include whether the tone suits a child or whether text contains five required points. Code-based measures such as precision, recall, and latency still matter. Human evaluation remains part of the best teams' process, especially for high-stakes products such as Vanta's regulated work. Since product quality involves trade-offs between factors such as cost and helpfulness, Habib recommends a scorecard rather than one overall metric.
Logging and replay help teams make agents production-ready
Habib recommends tooling that supports collaboration with domain experts, evaluation during prototyping and monitoring, and comprehensive logging. Teams should capture inputs and outputs at each stage, replay runs, and turn production failures into regression test cases. He cites Rivet, an open-source library built by Ironclad. Ironclad nearly abandoned agents after adding more function calls caused the system to fail, then recovered after an engineer built logging and rerunning tools that exposed the traces. Habib also describes Notion's ability to find an AI run from production, rerun it, and modify it. In his view, this debugging workflow helped move agent systems into production.
"You want to be able to take data points from your logs and put them into test sets of edge cases or things that you want to make sure that you succeed on in the future."16:50
Who should watch
You are building an LLM product and need a practical way to organize engineers, subject-matter experts, and machine learning specialists.
Your team is changing prompts or models without a reliable regression suite and needs to connect evaluation to product requirements.
You are working on RAG or agents in production and need better logging, replay, user feedback, and failure analysis.
