Uber's food enhancement agent must improve weak food photos while preserving the dish, merchant branding, and marketplace diversity.
2
A closed feedback loop compares production outputs with human labels, diagnoses mismatches, updates agent configuration, and ships only versions that pass offline guardrails.
3
The system combines routing metrics, iterative image QA, pairwise comparisons, publish-ready checks, and marketplace signals such as conversion.
Summary
Soumya Gupta and Jai Chopra describe Uber's production system for improving food photography from smaller independent Uber Eats merchants. The system first understands and routes each image, skips photos that should remain unchanged, and sends selected images through an editing loop with QA feedback. Its evals protect faithfulness, completeness, realism, brand identity, and marketplace diversity. Human-labeled data provides the initial reference set, while sampled production data detects drift and triggers automatic diagnosis and configuration updates. The speakers show how reward hacking can produce large pixel changes without a meaningful improvement, or cause the model to become overly conservative. Multiple QA gates reduce the chance of bad images reaching customers. Once the system is live, thumbs-up and thumbs-down feedback, merchant comments, internal testing, and marketplace metrics feed back into diagnosis. The approach treats evals as an operating loop that keeps every agent aligned as data and failure modes change.
Food photo enhancement has to preserve trust and marketplace diversity
Uber's smaller independent merchants often lack the time, knowledge, or budget for professional food photography. The agent therefore has to improve weak images without making them look artificial. The speakers say it must stay faithful to the original image, preserve each merchant's brand and packaging, and avoid making every dish look the same. A single prompt applied across the marketplace could collapse its visual diversity. The system also has to handle a long-tail distribution of image quality, from poor sharpness and composition to user-generated photos. Their stated goals include selective quality improvement, global marketplace optimization, safe shipping, continuous learning, and cost-efficient operation.
The agent balances creative editing with explicit safety limits
The speakers place their design between a brittle rules-based system and an unconstrained creative agent. Rules offer control but do not scale across the full marketplace. A highly agentic system can adapt to many images, but it needs safety guardrails. The production flow uses an image-understanding and routing agent, an image-editing agent, a QA agent, and final post-processing checks. The editor can receive QA feedback, revise the image, and try again. If it continues to fail after a set number of loops, the image is not published. Every stage is logged in a flat structure so engineers, product managers, and other teams can inspect individual cases or aggregate failures.
Routing is evaluated like a classifier with recall as the main guardrail
The router combines the image, text description, and metadata into a structured representation. It then applies pass and fail criteria to decide whether to enhance the image or keep the original. The team evaluates this decision with a confusion matrix and measures precision and recall. Soumya Gupta says recall is the routing guardrail because a bad image should not slip through without enhancement. False positives also matter: sending an already good cheeseburger photo to the editor costs compute and could degrade the image. In a more complex router, different branches may use different models, latency targets, and costs, so evaluation can use an n-by-n routing matrix instead of a simple two-by-two matrix.
Human labels establish the first model and its offline guardrails
The initial reference point is a representative dataset labeled by humans. The sample covers different geographies, dish types, image-quality types, and other cuts of the marketplace. Labelers receive objective guidelines to reduce subjective variation and noise. The team tunes the agent against this golden dataset, checks its guardrail metrics, and ships only when those metrics pass. The examples show why routing errors matter. A high-quality image may be mistakenly sent for enhancement, creating cost and degradation risk. A photo of six chicken wings may be approved even though the dish description says eight pieces. Enhancing that image could cause the model to hallucinate two extra wings, creating a faithfulness failure.
Production sampling turns static evaluation into automatic tuning
Offline evaluation does not cover every failure that appears in production, so Uber samples production data at a regular cadence and sends it to human labelers using the same guidelines. The system compares agent output with those labels. When it finds a mismatch, a diagnosis agent localizes the problem and triggers an auto-tuning pipeline. A prompt optimizer contains a reflect agent, which examines mismatches and systemic data issues, and a synthesize agent, which updates the agent configuration. The revised version is benchmarked against the golden dataset. It enters the agent store only if it passes, and the production system can roll back quickly if needed. The workflow is configuration-driven and does not require a human to approve each update.
Iterative enhancement uses QA feedback and pass at K
For each selected image, the system generates an image-specific prompt from the description and routing directives. It enhances the image, then sends it through a multidimensional QA gate that checks properties such as plating, faithfulness, and color. Failed feedback is returned to prompt generation for another edit. The image is published if it passes within the allowed iterations; otherwise the system accepts a coverage hit and leaves it unenhanced. The speakers call the metric pass at K, meaning the pass rate by the Kth iteration. Their sweet-potato-fries example fails first because the portion size and plating are unrealistic, then passes after the second iteration.
Pairwise evaluation exposes reward hacking and model failure modes
Jai Chopra explains that the system compares the input and output images and asks whether the output is better according to criteria agreed with product, design, policy, and legal teams. The result can be yes, no, or unsure. Examples include adding shrimp that was absent from the source, removing sauce from sushi, and covering sauce with a plate. One edit changes the raw pixels substantially while producing no meaningful improvement, which the speakers identify as reward hacking. Another edit starts creatively, receives negative QA feedback, and then overcorrects into a generic ceramic bowl. When the model cannot verify details such as the count of wontons, the system chooses uncertainty and rejects the output in production.
Several feedback loops converge through a shared diagnoser
The model loop handles drift against the offline human-labeled dataset, but the production system also gathers internal testing and live user feedback. Thumbs-up and thumbs-down signals, free-form merchant comments, and feedback from design and product teams can all enter the diagnosis process. The diagnoser determines which agent or agents need changes and routes each one to the relevant configuration update. Flagged examples are replayed, metrics are benchmarked, and the new configuration is pushed only after it passes. At the marketplace level, the team tracks health and quality measures such as adding to cart, conversion, and completed orders. These results can be sliced by geography, device type, and dish type to find segments where the system improves or needs tuning.
"You want to start with your logging cuz if you don't start with it, you have nothing to optimize for, let alone set up a self-learning loop."06:46
Who should watch
You are building a narrow-domain multimodal agent and need an evaluation plan before taking it to production.
Your system changes over time, and offline benchmarks are no longer catching the failures that appear in live data.
You need to combine human labels, model-based QA, user feedback, and business metrics without giving any one signal unchecked control.
