Offline evals fail when simulated users are more polite and helpful than real customers.
2
LLM judges become useful when they score binary, task-specific criteria tied to product decisions.
3
Production traces, human labels, and error analysis should continually update the dataset, judge, prompts, context, and harness.
Summary
Nick Ung and Akshay Sharma describe Lyft's evaluation pipeline for customer-support agents. They separate development, offline testing, production tracing, online grading, and human error analysis. Their first simulator used a frontier model as the customer, which produced complete and polite requests and gave the agent a pass rate above 90 percent. Fine-tuning the user model on Lyft transcripts and adding personas such as refund seekers and AI skeptics made the simulations harder and closer to production. The speakers also explain how to build LLM judges around binary task outcomes, calibrate them against roughly 100 human-labeled examples, and measure precision and recall. Traces and annotation queues turn production failures into new offline cases. Lyft is moving toward a config-driven harness with reusable primitives, parallel execution, and integration into local development, pre-commit checks, and CI/CD. The talk is practical about the work required to make eval scores affect shipping decisions.
An agent needs a launch gate before it reaches live users
Lyft treats agent development much like machine-learning model development. After the team builds the context, tools, graph, and system prompt, it runs a rigorous offline evaluation before launch. The evaluation uses simulated multi-turn conversations, a grader, and explicit criteria that determine whether the agent can ship. Once the agent is live, tracing records the execution and the context used, while online graders and human review expose failure modes. The point is to avoid using live customers as test data and to create a feedback path from production back to development.
A synthetic user can make an eval look much easier than production
Lyft's first offline simulator used a frontier LLM to role-play the customer. The model behaved like a helpful assistant, giving complete explanations and patiently describing problems. That produced a pass or accuracy rate above 90 percent, which the speakers regarded as too good to trust. Real Lyft customers are often impatient, frustrated, or unwilling to explain the issue clearly. A simulator built from overly polite prompts tests whether the agent handles ideal requests, so its score says little about difficult production conversations.
Real transcripts and user personas make simulations harder in useful ways
Lyft fine-tuned an LLM on user verbatims so it would produce requests closer to the language seen in production. The score went down after this change, but that was the desired outcome because the evaluation became more realistic. The team also defines intent, world state, and behavior for each scenario. Personas include a long-time customer frustrated with the earnings system, a user who wants to bypass the AI and escalate, a refund seeker, and an AI skeptic. These personas give the simulated customer a specific reason and style of interaction.
Evaluation criteria should describe a task outcome that engineers can act on
Akshay Sharma argues that generic metrics such as helpfulness, naturalness, toxicity, or conciseness are useful as baselines but poor core metrics when they produce unexplained scores. Lyft instead works with domain experts to define whether a specific task passed or failed. One education rubric checks whether the agent tried to educate the user too many times when escalation was appropriate, or escalated before giving education a fair chance. A binary result is easier to calibrate, connect to a product decision, and use for error analysis.
An LLM judge should be tested like a binary classifier
Lyft hand-labels around 100 examples with pass or fail outcomes, then separates the examples into training, development, and validation sets. A few training examples inform the judge prompt. The team iterates on the prompt and harness using the development set, then checks the final judge on held-out examples to avoid overfitting. Precision and recall against human labels provide an actual report of judge quality. This process turns an evaluator from an arbitrary score generator into a component with measurable agreement to a defined ground truth.
The evaluation standard changes as the team studies more failures
Akshay says evaluation criteria cannot be fixed entirely before the team examines model behavior. New traces can change the team's understanding of what good performance means, so the evaluator and the model need to be developed together. The team reviews raw traces, identifies specific failure modes, and keeps only metrics that can change a decision. It then forms a new premise, reevaluates the examples, and repeats the process on a regular cadence. Evaluation is therefore a continuing review loop rather than a one-time audit.
Production traces turn failures into new offline tests
Tracing captures the full graph execution, including which nodes ran, what the model saw, which tools were called, token usage, and latency. Tools such as LangSmith and Langfuse can provide a place to inspect these traces. Annotation queues give domain experts a simpler interface for labeling examples without reading raw JSON. The labeled traces can then enter offline datasets or help calculate precision and recall for judges. This connects observed production failures to the test suite and gives future evaluations more relevant cases.
Eval insights can improve the model, the context, or the harness
Nick Ung separates continual improvement into model learning, context learning, and harness learning. Model learning changes the underlying weights through post-training or custom model training. Context learning changes documents, stored user memories, and tool outputs. Harness learning changes system prompts, tool schemas, routing, control flow, and retries. Lyft's error analysis informs changes across these areas. The team is also considering reward modeling and reinforcement learning from the real user signals it has collected about agent performance.
A config-driven harness makes repeated evaluation part of development
Lyft's planned harness stores evaluation configuration in editable YAML so analysts, data scientists, and engineers can contribute. It defines reusable primitives such as tasks, datasets, personas, model adapters, and evaluators. Parallelism is needed for suites containing thousands or tens of thousands of examples. The same configuration can run locally after a prompt change, in a pre-commit hook, or in CI/CD as a regression and acceptance test. The harness is meant to make evaluation repeatable across agents and sub-agents instead of leaving it in scattered notebooks and scripts.
"If you have an eval that's too easy, that doesn't give you any real production insights into how your AI agent is actually going to perform."15:56
Who should watch
You are shipping a user-facing agent whose offline pass rate looks strong, but production keeps revealing failures that your test set missed.
Your team uses generic LLM-judge scores and needs a method for tying evaluation results to concrete product or business decisions.
You are building an evaluation platform and need patterns for tracing, human annotation, reusable datasets, and CI/CD regression checks.
