Agent systems collect traces and eval results, but those signals often stop before they can change future retrieval.
2
Utility-ranked memory combines semantic similarity with the history of whether a memory helped or hurt an outcome.
3
The SQL agent demo shows a failed search changing a later tool trajectory without changing the existing prompt.
Summary
Sonam Pankaj argues that production agents need to learn from task outcomes at runtime. Current systems collect tool calls, model completions, exceptions, and pass or fail judgments, but that information often remains in observability dashboards and eval suites. Retrieval then continues to select memories by similarity, even when similar memories have not helped in the past. Her proposed utility-ranked memory re-ranks memories using both task similarity and outcome history. Memories store reasoning about what to do, rather than only user facts or conversation history. Pankaj presents Agent RX, a runtime layer that updates retrieval without retraining, fine-tuning, or manual prompt edits. In the SQL demo, an unsuccessful search for a gaming mouse produces feedback. A later run changes its search behavior and finds a related wireless mouse. She is also direct about cold-start, noisy review, utility drift, and scaling limits.
Pankaj defines an agent as an LLM that can reason, invoke tools, interact with the real world, and retrieve memory. The loop is incomplete when it cannot learn from what worked and what failed. In the basic ReAct-style flow, the agent runs a task through retrieval and tool execution until it finishes, but the outcome does not automatically change the next run. As a result, an agent can repeat the same failure. Pankaj says the missing system must consume traces, absorb eval results, and turn both into guidance for future retrieval.
Production infrastructure records tool calls, model completions, and exceptions. An eval suite judges whether the final answer passed or failed. Pankaj says those results are not reflected in the context, skills, or actions of the agent. The agent therefore has no access to why yesterday's run succeeded or failed. Engineers must manually inspect the results, rewrite prompts, redeploy, change the model, restructure the harness, or fine-tune a model. Her retrieval boundary is the gap between recorded experience and the context used on the next run.
Similarity alone retrieves memories that sound relevant
Pankaj describes common memory systems as stores for user preferences, profiles, conversation history, and long-lived personalization. Their retrieval signal is embedding similarity, so a memory is selected because it resembles the current query. That does not tell the system whether the memory helped complete a task. She contrasts this with production learning, where a useful memory should gain influence after successful use and lose influence after a failed outcome. The goal is to make retrieval sensitive to results rather than only to wording or subject matter.
Utility-ranked memory combines similarity with outcome history
Pankaj introduces a utility score that weights similarity by how useful a memory has been for executing tasks. The ranking uses the history of past pieces of memory and their outcomes. Agent RX applies this at runtime, so the agent can improve without retraining, fine-tuning, or manual prompt engineering. A memory that resembles the current task can still rank poorly if it has repeatedly hurt outcomes. A less obvious memory can rank higher when its earlier uses led to successful execution. Outcome becomes part of retrieval re-ranking.
The system treats memory as reasoning rather than as isolated facts. Pankaj gives a refund example: a useful memory would tell a support bot to check the settlement before issuing a refund, so the customer is not paid twice. A preference such as a user's preferred theme does not help with that decision. The context is updated for the task at hand, which addresses the problem of stuffing large amounts of loosely related information into every prompt. The memory describes what to do in a situation and carries the history of whether that guidance worked.
Pankaj says the system can turn accumulated memory into skills after enough reviews. She gives a product SQL agent example in which an obsolete column remains in the system prompt even though it is no longer useful. An updated skill can carry the newer understanding forward without editing the existing prompt. She describes this as a runtime process in which the agent repeatedly uses an updated skill. The talk presents skills as a way to preserve learned reasoning once the system has gathered enough evidence.
In the demo, the SQL agent initially receives a request for a gaming mouse and retrieves zero memories. It says that it cannot find one. Pankaj submits the failed input, response, trajectory, and tool quality as feedback. On a later run, the agent searches for a wireless mouse, which exists in the database, and returns a related result. The earlier trajectory made one product search that returned an empty product list. The later trajectory still calls the product search tool, but the query produces a result. Pankaj attributes this change to memories ranked by utility.
The approach has a cold-start and review-quality problem
Pankaj lists limits to utility-ranked retrieval. At cold start, the system is doing pure semantic search because it has not accumulated enough reviews. Similar memories can drift in usefulness over time, and noisy labels can make utility noisy. A lambda hyperparameter controls credit and re-ranking. She says the system reduces several of these problems, but cold start remains unresolved. The approach therefore depends on collecting outcome feedback before its ranking can become more informed than ordinary semantic retrieval.
"You do not retrieve by keyword, you do retrieve by semantic similarity to the current task weighted by whether those memories have historically helped or hurt the execution or the outcome."05:10
Who should watch
You are building an agent that repeats failures even though your observability and eval systems record them.
Your memory layer retrieves semantically similar examples but does not know which examples produced good outcomes.
You are evaluating runtime learning for a tool-using agent and want to see how feedback can change a later SQL search.
