An agent harness controls memory, tools, perception, context assembly, and the agent loop around a largely fixed language model.
2
Long-term memory should separate episodes, procedures, and semantic facts instead of pushing everything into a limited context window.
3
A bank can use a custom harness to flag suspicious transactions autonomously, then leave final inspection and action to a human.
Summary
Nacho Martinez explains how to build an agent harness around a language model for an anti-money-laundering system. The example is a medium-sized bank whose agent reviews transactions, identifies suspicious patterns, and prepares findings for human inspection. Martinez focuses on the parts engineers can control: the agent loop, memory, context assembly, tools, skills, storage, retrieval, and instruction hierarchy. He separates short-term context from episodic, procedural, and semantic long-term memory. He also compares summarization with compaction, describes vector retrieval for tools and skills, and explains how previous runs can become reusable skills. His advice is practical: start with a small memory design, choose files and databases according to the access pattern, and build a custom harness for the problem at hand. The workshop demonstrates these ideas with a fake bank database, an Oracle Cloud model, transaction analysis, tool calls, and a visual context window.
The harness is where engineers can change an agent's behaviour
Martinez divides an agent system into the application, model, infrastructure, compute, and data layers. Engineers have less control over a third-party model, so he focuses on the data layer and its surrounding machinery. The harness contains the tool-use, memory, and perception scaffolding. Changing its configuration can make the same frontier model perform better or worse on a specific task. The parts he names include storage, encoding and retrieval, memory design, the semantic layer, the agent loop, and context engineering. He argues that these choices determine what the agent can access and how it works over time.
An AML agent can flag transactions while leaving decisions to a human
The workshop uses a medium-sized bank with suspicious transactions grouped into five types. Martinez gives an example of a $1 million deposit followed by $50,000 ATM withdrawals over three days. The agent reviews transactions continuously and flags cases for bank staff, reducing the time people spend searching through them. In the demonstrated application, fake transactions arrive continuously, an agent running a Grok model on Oracle Cloud marks some as suspicious, and the interface shows the database, context window, evidence searches, tool calls, and suspicious activity reports. The agent makes analysis and recommendations while a human inspects the results.
Short-term context and long-term memory have different jobs
The active context window is short-term memory because most of its contents disappear when the conversation ends. Martinez divides long-term memory into episodic, procedural, and semantic categories. Episodic memory contains conversations, tool logs, and summaries of previous runs. Procedural memory contains skills and workflows. Semantic memory contains facts such as a user's identity, preferences, or preferred framework, including information that was not stated directly in the current exchange. He advises engineers to decide which categories their application needs rather than adopting a fixed architecture.
Keeping the context window small requires summarization or compaction
Martinez says that larger attention matrices perform worse, even when a model advertises a very large context limit. Context summarization creates a shorter summary and starts the next interaction from that summary. Compaction stores chunks of the conversation in a file or database with an identifier, allowing the agent to retrieve a chunk later when needed. These approaches reduce the amount of history included in every request. He presents context engineering as a way to keep the working context focused instead of placing every conversation, tool, and skill in it.
The agent loop assembles context and controls the work
The agent loop drives the other components. Its context assembly step can pull in skills, tools, earlier conversation, files, and database records. The model reasons about what to do, makes an API call, and continues through more iterations. In the workshop harness, the loop has a maximum of eight iterations and 60 seconds of reasoning per iteration, although Martinez says these limits are configurable. The loop also decides which tools and skills to retrieve and when to dispatch them. This makes the loop an engineering control point rather than a fixed part of the model.
Toolbox and skillbox retrieval prevent capabilities from consuming the whole prompt
Martinez says that loading hundreds of skills into every request would quickly consume the context window. His pattern stores vector embeddings for tools and skills. When the agent needs a capability, it performs similarity search, retrieves the relevant items, uses them, and then removes them from the active context. In the workshop, separate vector stores hold tools and skills, with a default of three retrieved items from each search. A reranker could then select the most relevant item from those candidates. This lets the harness expose many capabilities without placing all of them in every request.
Previous work can become reusable skills without changing model weights
Skill distillation turns useful conversations and agent runs into reusable instructions. Martinez suggests creating a skill file and adding it to the existing skillbox. A front-end skill, for example, could preserve a user's preferred colour style and other working preferences. When the agent later retrieves skills through embeddings, it can find that prior guidance from related wording. He connects this to continual learning for a frozen frontier model. The model's parameters stay unchanged, while the harness improves through nonparametric memory. He also mentions representational memory through embeddings and parametric methods such as fine-tuning and low-rank adaptation.
A small memory design is a better starting point than many abstractions
During questions, Martinez recommends starting with only short-term and long-term memory. If something in the current context is worth retaining, the agent can promote it to long-term storage as raw text. Engineers can later vectorize that material and add a recall step between observing and reasoning. The agent retrieves a memory when its similarity to the current query is close enough and discards it when it is not relevant. He gives this advice because extra memory categories and abstractions add complexity before the team knows which information the application needs.
Custom harnesses should be shaped around the problem
Martinez does not suggest rebuilding a general-purpose coding agent for every project. He recommends starting from the chat completions API and attaching a smaller, problem-specific harness. A financial agent may need database access, transaction analysis, and a particular memory design. Mathematical agents that run without a human may instead need automatic compaction, automatic summarization, and near-infinite patience. His workshop agent has a simplified eight-step limit, while other tasks may need different loop and context policies. Starting from scratch can remove most of the features that a particular use case does not need.
"The harness is actually one of the most interesting topics in AI nowadays because these harnesses actually control everything that the AI agent can do."10:51
Who should watch
You are building an agent that must run for long periods and need a concrete way to manage context, tools, and memory.
Your application handles financial or operational data and needs autonomous analysis followed by human review.
You are deciding whether to build a focused harness instead of adopting a general-purpose agent framework.