Production LLM applications need tracing and logging so prompt changes can be connected to regressions in user-facing behavior.
2
Teams should iterate on prompts and build evaluation pipelines before spending resources on fine-tuning.
3
LLM judges can reduce evaluation costs, but they still need human review, custom criteria, bias checks, and ongoing iteration.
Summary
Alex Volkov presents LLM application evaluation as a mock courtroom. His cases cover common mistakes: shipping a hackathon project without tracing, fine-tuning before trying prompt iteration, ignoring new model releases, and relying only on programmatic tests for a customer-facing chatbot. He explains the evaluation process as collecting representative inputs and correctness criteria, running models against them, and scoring the results. He describes three approaches: programmatic checks, human review, and LLM judges. Programmatic evaluation works well for numbers, assertions, and executable code, while human review remains useful during development. LLM judges can process large amounts of conversational output more cheaply, but they need human-defined criteria, bias checks, and repeated validation. Volkov demonstrates these ideas with Weights & Biases Weave, including tracing, version comparison, custom criteria, and evaluation results for the Open WebUI project.
Production LLM features need traces and logs before prompt changes begin
Volkov contrasts a fun hackathon project with a paid production application. After Daniel's demo gains attention and customers request features, he repeatedly changes the prompt and deploys on a Friday. Existing users report that older features no longer work, but Daniel cannot identify the cause because he did not trace or log the application. Volkov's verdict is "charged with no Trace left behind." He says that anything valuable in production needs complete tracing and logging. He then describes Weights & Biases Weave, which can start with a Python decorator and track user interactions, call stacks, prompts, parameters, inputs, outputs, and code versions.
Sasha downloads Llama 3 and fine-tunes it on company data after seeing a 6% improvement on internal benchmarks. Volkov changes the verdict after asking whether she tried prompt iteration first. He says she skipped prompt experiments, a retrieval pipeline, and other ways to improve the application before using expensive GPU time. Fine-tuning can be useful later, but teams can often go far with prompt methods such as chain-of-thought prompting, flow engineering, and mixtures of agents. His advice is to iterate on prompts before fine-tuning, then consider fine-tuning once the earlier methods have been explored.
Model news changes fast enough to affect engineering choices
Morgan keeps using Llama 2 7B after deciding he cannot follow AI news. Volkov lists several releases from the recent quarter, including Claude Sonnet 3.5, Llama 3, GPT-4o, Gemini Flash, Project Astra, and Apple Intelligence. His point is practical: an engineer who stops tracking developments may miss models that change what is possible or economical for an application. He also acknowledges that following every release is difficult for people with other work. His proposed solution is his weekly ThursdAI show and podcast, which summarizes AI news so listeners can spend less time following it themselves.
Programmatic, human, and LLM-judge evaluations have different uses
Volkov defines an evaluation workflow as collecting user inputs and answers or correctness criteria, running a model on each example, and scoring the result. He describes programmatic checks as a good fit for numerical outputs, assertions, and executable code. They are easier to scale and cheaper, but do not cover every kind of conversation. Human review helps teams evaluate applications while they develop and iterate on prompts, though reviewing thousands of chats can be expensive and repetitive. LLM judges can summarize and assess many conversations at lower cost, but they still need humans to define criteria, check for bias, and improve the judging prompts.
LLM judges must be validated instead of trusted automatically
Volkov warns that the language models available in 2024 are not perfect judges. He recommends keeping humans involved, creating and checking evaluation criteria, testing for bias, and iterating on the judge's system prompts and examples. He cites the paper "Validate Your Validators" by Shankar as a related talk and idea. Off-the-shelf criteria may not fit a particular product, so teams should create criteria for their own business and application. His phrase "validate your validators" captures the operational requirement: the evaluator itself needs testing before its scores guide product decisions.
Custom criteria make evaluation specific to the application
Volkov uses a conference talk as an example of custom evaluation criteria. A talk might be judged on whether it is memorable, educational, helpful, funny, original, clear, articulate, and not too promotional. The same method applies to an LLM application. Instead of relying on generic measures, a team can define the qualities that matter for its users and product. In the Open WebUI example, Chris uses criteria such as contrast, relevance, and polish. Weave lets him inspect individual examples, compare model versions, and see the generated outputs alongside evaluation results.
Continuous evaluation makes model and prompt changes easier to inspect
The strongest case in the mock trial describes a pipeline with tracing and all three evaluation layers. Maxim uses Weave to add tracing, then iterates on an evaluation pipeline and continuously evaluates experiments. That setup makes it possible to change system prompts and detect regressions. Volkov presents the Open WebUI project as a concrete example. The application streams HTML generated by an LLM, while Weave traces calls and records evaluations. Chris can compare model versions, inspect individual examples, and review the output against application-specific criteria. Evaluation is therefore tied to the actual behavior of the product rather than a single benchmark score.
"These LM judges are great but still need iteration and yes humans in the loop to create criteria check for biases iterate on system prompts examples and much much more."12:28
Who should watch
You have an LLM feature in production and prompt changes can break behavior without an obvious way to find the cause.
You are deciding whether to improve prompts, fine-tune a model, or build an evaluation pipeline first.
Your team uses LLM-as-judge scores and needs a practical process for human review, custom criteria, and bias checks.
