DoorDash treats evals as a cross-functional process that includes strategy and operations, product, annotation teams, and engineering.
2
Stable APIs let non-engineers build annotation interfaces for their own workflows with coding agents instead of waiting for a platform-specific UI.
3
Teams continuously trace, sample, annotate, create golden datasets, calibrate judges, and monitor quality, with prompt ownership varying by team.
Summary
DoorDash's GenAI platform team began with the assumption that evals were an engineering concern. Different product groups needed different forms of evaluation, including session-level judgments, scaled human review, and trajectory-based evaluation. The team therefore moved from UI-first to API-first, then toward workflow-first tools that let strategy and operations staff run work themselves. The talk describes an eval process shared across roles. Strategy and operations set the quality bar, product managers turn it into rubrics and workflows, operations run annotations, and engineering provides APIs, telemetry, datasets, and judges. The platform separates a telemetry layer from a workflow layer. Stable APIs also let operators use coding agents to create annotation UIs for menus, images, and manual testing. A self-serve interface lets teams calibrate judge prompts, compare original and optimized prompts, and choose models. Prompt ownership differs across teams. The speakers report lower annotation costs and faster iteration after making the process self-serve.
Evals became the fourth pillar of the GenAI platform
DoorDash's GenAI platform team began by helping product teams balance accuracy, latency, and cost across models and agents. Its building blocks included an LLM gateway for switching models, an agent gateway for connecting tools and agents, and open-weights model hosting. The speakers describe evals as the fourth pillar. Different teams needed different forms of judgment. A discovery and shopping assistant needed session-level quality judgments, personalization needed scaled human judgment, and multi-agent systems needed trajectory-based evals. The platform had to support these needs while giving domain experts a direct role.
The platform moved from UI-first to API-first to workflow-first
DoorDash first built UIs so strategy and operations staff, product managers, and labeling partners could contribute without engineering help. It then added an API-first approach so engineers could build their own systems without being blocked by the central platform. With coding agents, the team moved toward workflow-first access, allowing strategy and operations staff and product managers to run operations themselves. The change followed the needs of the people doing the work rather than trying to predict every interface in advance.
Quality depends on several teams owning different parts of the process
The speakers describe evals as a cross-functional effort that adds domain knowledge to AI quality. Strategy and operations set priorities and the quality bar. Product teams translate those requirements into rubrics and workflows. Operations teams run annotations. Engineering provides APIs, telemetry, datasets, and judges. The platform combines these contributions into a process for shipping AI products. This arrangement puts quality decisions close to the people who understand the product use case, while engineering supplies the infrastructure and measurement tools.
The evaluation loop turns traces into human-reviewed data and calibrated judges
DoorDash's loop starts with tracing sessions and viewing what agents or language models produced. Teams sample the traces down to a small set that people can actually inspect. They annotate that sample with domain expertise, review the results, create golden datasets, calibrate judges against those datasets, and monitor the system over time. The speakers present this as a cycle that teams repeat rather than a one-time test. Human annotations improve the data used to measure or calibrate automated judgments.
Telemetry and workflow are separate surfaces with different users
The platform has a telemetry layer containing traces, scores, and observations. Users can access that layer through MCP, an SDK, and APIs. Its workflow layer is where strategy and operations and product teams set annotation tasks, review golden datasets, create judges, and calibrate them. The speakers also describe stable APIs behind the scores and datasets, with UIs built on top. This separation lets the platform team maintain shared data and access patterns while teams operate through workflows suited to their jobs.
Stable APIs let operators build annotation UIs for specific tasks
The platform team found it difficult to build a dedicated interface for every annotation use case. Instead, it gave operators access to APIs and coding agents. Strategy and operations teams could use tools such as Codex or Claude Code to vibe code their own annotation UIs. Examples included image annotation, manual testing, and reviewing a restaurant menu. The underlying patterns were similar even when the tasks differed. The speakers show this as a way to put workflow creation in the hands of operators, rather than making them wait for the platform team.
Judge calibration became a self-serve and reviewable workflow
Teams begin with a judge prompt and a definition of what they want to measure, then establish baseline scores by running the judge on traces. DoorDash uses the DSPy library for prompt optimization. After the optimization loop, a team can promote the resulting prompt as its LLM judge. A self-serve UI lets a product manager or operator set configuration, choose a model such as Gemini, Claude, or an OpenAI model, and run calibration without repeated engineering support. The UI also shows the original system prompt beside the calibrated prompt, so users can inspect the change before trusting it.
Prompt ownership varies because teams are still designing their operating model
Swaroop says some DoorDash teams have strategy and operations own the judge prompt, while others give ownership to a product manager or engineering. The platform supports these arrangements instead of forcing one organizational model. The speakers connect that flexibility to the fact that teams are still learning how to divide responsibilities. They describe the organization itself as evolving alongside the eval process. A self-serve platform reduces the need for every team to depend on the GenAI platform team.
"In some teams you have seen the strategy and operations folks own the prompt, you have seen some teams where the product manager owns the prompt, you have seen some teams where engineering owns the prompt."13:02
Who should watch
You are building an eval platform for several product groups with different data, judgment, or annotation needs.
Your operations or product teams need custom annotation workflows, but engineering cannot build a separate UI for every use case.
You want product teams to calibrate LLM judges themselves while keeping the prompt changes visible and reviewable.
