Agents should be shipped early, traced extensively, and improved by mining the resulting data.
2
Open models can handle trace judging at much lower cost after harness engineering gives them enough guidance.
3
Teams should use fast harness changes first, then fine-tune when prompt and orchestration changes reach their performance ceiling.
Summary
Vivek Trivedy argues that improving an agent starts with operating it in the real world and saving everything it does. Tool calls, messages, API calls, and command-line actions create traces that can be mined for failures, user reactions, long-context problems, counterfactual model comparisons, and training data. LangChain uses agents to inspect traces from other agents, since people cannot read millions of long interactions themselves. Trivedy connects observability with continual learning because both depend on recording actions and feeding the resulting information back into the agent. He describes a cost-conscious path from frontier models to open models, using harness engineering before fine-tuning. He also argues that pass or fail evals give agents too little information, while traces provide denser feedback. Over time, agents will need updates to their training data, harnesses, and memory. Append-only logs will not work for long-lived systems.
Agent improvement starts with shipping and collecting traces
Trivedy's recipe begins with putting an agent into the real world, where it can operate in an environment and receive feedback. Every run produces data through tool calls, output messages, API calls, and CLIs. Teams should store that information before deciding how to improve the system. They can then mine gigabytes or terabytes of traces, curate useful examples, and run experiments against the behavior seen before. A new prompt, tool, orchestration method, or loop should be judged by whether it improves the agent on the evidence already collected.
Observability and continual learning depend on the same trace data
Trivedy says observability and continual learning are tightly connected for agents. An agent acts in an environment, records what happened, and then needs to update its knowledge or behavior in response to feedback. He compares this with people doing things in the world, thinking about the results, and changing what they know. A company working on continual learning therefore needs traces. A company with traces can attempt continual learning over its agents. The trace is the record that connects action, feedback, and later updates.
Agent behavior is harder to reason about than ordinary code
A programmer can inspect a Python code block, follow its functions, and form a rough model of how it works. Agent behavior is spread across prompts, tools, skills, hooks, middleware, and sometimes other agents in a swarm. A prompt change can also behave differently in different domains. Trivedy gives medical and legal work as examples where the same change may have different effects. As systems trade determinism for autonomy, teams need tools that let them understand behavior from actual operation rather than trying to predict every interaction from configuration alone.
Agents can mine other agents' traces for failures and counterfactuals
LangChain centralizes trace data in projects organized around one agent or many agents, then sends agents to inspect those traces. The mining questions can target good and bad interactions, including cases where users were happy or upset. Teams can ask whether a coding agent becomes less capable after its first or second compaction, or whether it stays effective. Traces can also support counterfactual comparisons, such as testing what a different model would have done on the same task. This exposes behavior at the fine-grained level users actually experience.
Long traces require external querying and cost control
The amount of agent-generated data will keep growing, while reading it with another model can become expensive. The cost depends on input token price, the number of traces, and the average trace size. Very long coding-agent interactions may not fit into another model's context at all. Trivedy says systems should treat the trace as an external object and query into it instead of placing the whole interaction in context. This turns trace mining into a systems problem involving efficient retrieval and analysis, rather than a simple prompt that feeds all data to an evaluator.
Open models can replace frontier models for some trace judging
LangChain starts with a frontier model to establish whether a task is possible, then checks whether a cheaper open model can perform the same job. In work with Harvey on a legal benchmark, Trivedy says an open model roughly matched the trace-judging capability of Opus at one or two orders of magnitude lower cost. The team tried different models and used harness engineering based on the traces. Some of the gap came from prompting and guidance, while the frontier model was still more capable. The goal was to find the minimum intelligence needed for each task.
Fine-tuning becomes useful after harness changes reach a ceiling
Harness engineering gives fast feedback and can improve an agent through prompts, orchestration, and related system changes. Trivedy says teams eventually reach a point where further prompt tweaks produce little additional intelligence. At that point, they can fine-tune a base model for a narrow, domain-specific set of tasks. Customers often care about the tasks their users actually perform rather than the full range of possible tasks. Trivedy also describes an economic shift from token costs to hardware costs. For high-inference workloads, running a cluster can be cheaper than paying for every token, and the cluster can be shut down when demand falls.
Dense feedback makes traces useful for agent improvement
A benchmark that returns only pass or fail gives an agent little information about what to change next. Trivedy uses Terminal Bench as an example of a result that is ultimately just a number. Traces contain more detailed evidence about the actions that led to that result, so an agent can inspect them and propose a better next step. He connects this with auto research: an agent does something, reads the result and trace, proposes an experiment, and applies a fix. His advice is to collect feedback quickly, exhaust harness engineering first, fine-tune to break through its ceiling, and return to harness changes afterward.
Long-lived agents need changing memory, harnesses, and training data
Trivedy says continual learning will update an agent across three areas. The first is training data collected from agents acting in environments. The second is the harness, which evolves as models are trained in particular settings and as tasks change. The third is memory. Humans do not keep an append-only log of every experience, and an agent operating for years or decades cannot simply store everything in one growing file and search it. Trivedy expects sleep-time compute and dreaming-like processing to read traces over an agent's life and update its state.
"We humans are really good at remembering stuff over time. But we are not append only logs of information."17:54
Who should watch
You have an agent in production and need a practical way to turn its traces into fixes, evals, or training data.
Your trace volume is too large for people to inspect and your long interactions no longer fit cleanly into model context.
You are deciding whether to keep changing the harness, fine-tune a model, or move evaluation work to a cheaper open model.
