# Multimodal Collaborative Agents for Next-Gen Commerce Nidhi Kaushik Vyas, Google DeepMind | AI Engineer World's Fair 2026 | 21:08 Source: https://www.youtube.com/watch?v=AhQpRalYlyg Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/multimodal-collaborative-agents-for-next-gen-commerce Published: 2026-09-01 Tags: agents, evals, multimodal, planning ## TL;DR - Shopping agents need to help users turn a vague feeling into a usable goal instead of assuming they already know the right search terms. - The agent should choose its next question by information gain, such as asking for room width before discussing furniture that may not fit. - Visual preference boards, adaptive response formats, and stage-specific autoraters help the system discover intent and return useful results. ## Summary Nidhi Kaushik Vyas presents a framework for commerce agents that work with incomplete, subjective intent. She argues that users often arrive with a vibe rather than a well-formed query, so the agent must build a working state from conversation history, personal context, hard constraints, and reference images. It should attach confidence to inferred preferences and identify information that needs real-time refresh, such as inventory. The agent then finds the intent gap and chooses the next question by expected information gain. For subjective preferences, a visual board can work better than text. Once the system understands the user's constraints, it should choose a response format that fits the task, such as a summary, comparison table, or product imagery. Vyas also describes autoraters for fact retention, calibration, counterfactual sensitivity, question utility, format accuracy, data fidelity, and user actionability. She is candid that agent-to-agent commerce is still early, and that users may prefer direct involvement during discovery. ## Key ideas ### Shopping agents must close the articulation gap [01:08](https://www.youtube.com/watch?v=AhQpRalYlyg&t=68s) Vyas says many current agents behave like wrappers around a search bar. They assume the user already has a well-defined intent, the right keywords, and a clear idea of what they want. Real shoppers often arrive with a fuzzy feeling or a vibe. The agent must help them understand and express their preferences, show possible directions, and move toward recommendations that fit their constraints. She grounds the discussion in commerce because the pattern is easy to see there, while saying it also applies to areas such as finance and education. ### Discovery builds a working state from mixed signals [02:03](https://www.youtube.com/watch?v=AhQpRalYlyg&t=123s) The discovery phase gathers past conversations, the current query, personal context, reference links, and reference images. In the living-room example, the agent records the budget and other hard constraints, then infers softer preferences from an inspiration image. It might identify a style that appears relevant and attach a confidence score to that inference. The state also records variables that need to be checked in real time. Inventory is one example, since returning stale availability makes the answer useless. The agent therefore remembers both what it knows and how certain it is. ### Autoraters test whether the working state changes for the right reasons [06:33](https://www.youtube.com/watch?v=AhQpRalYlyg&t=393s) Vyas describes several checks for the discovery state. One tests whether facts from the context were retained correctly. Another checks whether confidence calibration stays within an allowed error bound. A counterfactual test changes part of the query and checks that the extracted constraints change when they should, while unrelated constraints remain stable. These tests measure sensitivity in both directions. The same evaluation mindset continues through the rest of the loop, rather than treating the final answer as the only place where quality matters. ### The next question should resolve the most important blocker [07:27](https://www.youtube.com/watch?v=AhQpRalYlyg&t=447s) After identifying known facts, the agent finds the intent gap, the unknown variables that prevent a useful answer. It does not need to ask about every unknown immediately. Instead, it compares possible moves and prioritizes the one with the highest information gain. For the living-room example, room width may be the best question because furniture that does not fit makes later recommendations irrelevant. Vyas says evaluation should check whether the agent finds the blockers, avoids asking too many questions, and chooses questions that elicit useful preferences. ### Constraints need a temporary bridge into the product ontology [10:10](https://www.youtube.com/watch?v=AhQpRalYlyg&t=610s) When the agent explores a style preference, it must map that user constraint to the product catalog and the ontology in its knowledge base. This temporary bridge lets retrieval connect the conversation to merchant metadata. The mapping can be created in real time from the current constraint and other information already gathered. Vyas says merchant domain knowledge matters here because merchants understand how their products and attributes should map to the constraints an agent may receive. ### Visual boards can express subjective preferences better than text [11:04](https://www.youtube.com/watch?v=AhQpRalYlyg&t=664s) For a subjective preference such as room style, the agent may decide that a text question is a poor elicitation method. It can instead assemble a visual preference board using the reference image, past conversation, known constraints, and the temporary product-ontology mapping. The user can react to styles that speak to them, giving the agent a shared visual language for an otherwise difficult preference. Hovers and clicks become micro-signals that update the confidence model. Vyas evaluates this stage through hidden-preference discovery, turn efficiency, and format selection accuracy. ### The response format is part of the agent's intelligence [12:57](https://www.youtube.com/watch?v=AhQpRalYlyg&t=777s) Once the agent understands the user's preferences, budget, and relevant product dimensions, it still has to decide how to present the result. A policy or review question may need a summary or bulleted list. Comparing two products may call for a table of trade-offs across the dimensions the user cares about. Style exploration may work better with visual references and product inspiration photos. Vyas evaluates whether the chosen format fits the query, whether the data is accurate, and whether the response gives the user enough confidence to take the next action. ### The whole loop needs evaluation that grows with the system [16:38](https://www.youtube.com/watch?v=AhQpRalYlyg&t=998s) Vyas closes with four practical directions. The product should accept fuzzy intent. It should show and ask, since visuals and comparisons can reveal preferences faster than repeated textual questions. It should shape the answer around the user's task. It should grade every stage with autoraters. She describes autoraters as an evolving system that starts simply and grows as the agent grows. In questions, she says merchant ontologies provide important domain intelligence, while the agent currently controls the response format across merchants. ## Notable quotes - "However, there is quite a huge articulation gap." (01:28) - "So, in this case I mean one could argue that maybe finding out the room width is the best next move for the agent because if the products that the agent is recommending doesn't fit into the room then again it's a moot point." (08:41) - "So, do focus on making sure that the presentation format is ideal for the user being able to find the right information." (16:39) - "Users really like to be more involved in the process of choosing or even exploring the different possibilities." (20:09) ## Tools & references mentioned - Google DeepMind - UCP - MCP ## Who should watch - You are building a shopping or discovery agent that gets vague requests instead of clean product queries. - Your system asks users many questions and you need a way to choose the next one or use visual choices instead. - You need evaluation methods for state extraction, confidence, question selection, response format, and factual accuracy.