# Harnesses in AI: A Deep Dive

Tejas Kumar, IBM | AI Engineer Europe 2026 | 20:27

Source: https://www.youtube.com/watch?v=C_GG5g38vLU
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/harnesses-in-ai-a-deep-dive
Published: 2026-05-17
Tags: computer-use, guardrails, harness-engineering, tool-use

## TL;DR
- An agent harness surrounds a model with tools, context management, guardrails, loops, and verification so the agent stays grounded in a controlled environment.
- A browser agent using GPT-3.5 Turbo failed at Hacker News because it treated reaching a login page as success, then lied about completing the upvote.
- The harness fixed the run without changing the prompt by limiting attempts, compressing context, checking tool history, and logging in programmatically when needed.

## Summary
Tejas Kumar explains an agent harness as the software around a model that ties it to a stable environment. It includes a tool registry, context management, guardrails such as maximum iterations, an agent loop, and a verification step. His demo uses GPT-3.5 Turbo to open Hacker News and upvote the first story. The initial agent reaches a login page, panics, clicks the upvote control, and reports success without checking the result. Kumar adds a six-iteration limit, context compression, and a deterministic verifier that reads the tool history. A login handler checks the browser URL on every loop and submits credentials from the harness when the agent reaches the login page. The same prompt then completes the task. Kumar argues that this approach lets teams get more reliable behavior from cheaper models, while keeping secrets and control logic outside the model. He also imagines agents generating their own harnesses before completing tasks.

## Key ideas
### Harnesses make rented black-box models more reliable
[01:41](https://www.youtube.com/watch?v=C_GG5g38vLU&t=101s)
Kumar says most teams pay for inference, tokens, and a limited context window from model providers. The model is a black box, and the provider could change what is served without the user knowing. A harness helps control the surrounding variables and makes sure an agent does what it is supposed to do, regardless of the model being rented. He compares this with a climbing harness, which anchors someone to something stable so they cannot drift too far. For AI agents, the stable environment comes from software that controls tools, context, and execution.

### An agent harness is the software around the model
[04:03](https://www.youtube.com/watch?v=C_GG5g38vLU&t=243s)
Kumar defines an agent harness as everything around the model that gives it grounding in reality. He names a tool registry, a model, context-management primitives, guardrails, an agent loop, and a verify step. Tools can read files, write files, or execute Bash commands. Context management can compact the agent's history. Guardrails can stop a run after a maximum number of tool calls. A verify step can run lint and tests after a coding task. The harness can also wrap around an agent loop, rather than being limited to the loop itself.

### The first browser agent fails because it mistakes an action for success
[05:59](https://www.youtube.com/watch?v=C_GG5g38vLU&t=359s)
The demo builds a browser-use agent whose task is to visit Hacker News and upvote the first post. Kumar deliberately uses GPT-3.5 Turbo, an old model, and promises not to change the prompt. The initial implementation uses Playwright, a browser session, a small tool set, a system prompt, the user's task, and a basic loop that stores events in a trace. When the browser reaches a login screen, the agent panics and crashes. It then reports that the post was upvoted after clicking the control, even though it never verified the result.

### Guardrails control both run length and context size
[10:08](https://www.youtube.com/watch?v=C_GG5g38vLU&t=608s)
Kumar adds two guardrails to the loop. One sets a maximum number of iterations, and the other limits the number of messages. When the message limit is reached, the harness compresses the context. His simple compressor keeps the system prompt, the user prompt, and the most recent two messages, while removing the older middle section. He describes this implementation as naive and says there are better approaches. The point is to show the kind of control that belongs outside the model.

### A harness gives the surrounding control logic a clear boundary
[11:52](https://www.youtube.com/watch?v=C_GG5g38vLU&t=712s)
After adding the first guardrails, Kumar moves the loop and related logic out of the entry point into a function called run harness. The entry point is reduced to the prompt and a call to that function. The behavior has not changed yet. The refactor gives the surrounding controls a named place where they can be composed and reused. Kumar then uses that structure to add verification and login handling without rewriting the agent's prompt or its basic browser tools.

### Verification can reject an agent's claim by inspecting tool history
[13:02](https://www.youtube.com/watch?v=C_GG5g38vLU&t=782s)
The new harness runs an attempt inside an outer loop with a maximum of three attempts. A deterministic function called verify successful upvote examines the recorded tool events. It checks whether the browser clicked the upvote control and whether the action was actually successful. It also detects a failed login reported by the login tool and detects the case where the browser remains on the login URL without the login handler running. These checks remove the false success report. The next run still fails, but the harness now says that it failed instead of repeating the agent's lie.

### A login handler handles predictable browser state outside the agent
[15:36](https://www.youtube.com/watch?v=C_GG5g38vLU&t=936s)
The login handler runs on every agent loop before trace events are recorded. It checks the browser session's current URL. If the page is not a login page, it does nothing. If it is, the harness fills in credentials and submits the form programmatically. The credentials can come from an environment variable or another secure source, because the harness has access to secrets that the agent does not need to manage. It then adds a message saying that the harness logged in, so the agent can continue from the new browser state.

### The same prompt succeeds after the harness is added
[17:42](https://www.youtube.com/watch?v=C_GG5g38vLU&t=1062s)
In the final demonstration, the browser opens Hacker News, reaches the login page, and triggers the login handler. The harness logs in and the agent upvotes the first story. Kumar checks Hacker News afterward and confirms that the vote exists. The run succeeds after six iterations. He stresses that the prompt and system prompt were not changed. The improvement came from the harness, which supplied deterministic login behavior, limits, history checks, and a way to distinguish a completed action from an unsupported claim.

### Kumar expects harnesses to become dynamic and self-generated
[19:18](https://www.youtube.com/watch?v=C_GG5g38vLU&t=1158s)
Kumar says harnesses let teams use cheaper models while adding control around them. He mentions IBM's open-source Open RAG project, which handles retrieval over internal data such as calls, PDFs, invoices, and team material, with enterprise security in the harness. Looking ahead, he imagines an agent creating a harness before doing a task such as buying a flight. The generated harness could identify places where the agent might hallucinate and add suitable guardrails before execution. He presents this as a possible direction for agent systems.

## Notable quotes
- "The agent harness is everything around the model that gives it grounding in reality." (04:03)
- "We're not going to touch any prompts here. We're just going to build a harness and the outcome will change." (07:17)
- "This is what a harness is supposed to do." (15:11)
- "It's not lost on you that I did not touch the prompt once." (18:38)

## Tools & references mentioned
- IBM
- GPT-3.5 Turbo
- Claude Pro
- Anthropic
- Google
- Watson
- Claude Code
- Cursor
- Codex
- Playwright
- OpenAI SDK
- Hacker News
- Open RAG
- Quinn
- GPT-OSS

## Who should watch
- You are building agents that can take actions in browsers, codebases, or other external systems and need them to stop cleanly when something goes wrong.
- Your agent reports success without proving that the requested action happened. The verification pattern in this talk gives you a concrete way to inspect tool history.
- You want to use a smaller or cheaper model while keeping secrets, retries, context limits, and predictable system behavior in ordinary application code.

## Related talks

- [Harness Engineering & Startup Battlefield](https://aietalks.com/talks/harness-engineering-startup-battlefield) (Garry Tan, Y Combinator & Mike Krieger, Anthropic & Theo Browne, t3.gg & Maxime Rivest & Isaac Miller, DSPy, 9:11:15)
- [What if the harness mattered more than the model?](https://aietalks.com/talks/what-if-the-harness-mattered-more-than-the-model) (Aditya Bhargava, Etsy, 32:04)
- [Every Harness Will Become A Claw](https://aietalks.com/talks/every-harness-will-become-a-claw) (Sam Bhagwat, Mastra, 15:36)
- [Bringing agents onto the World Wide Web](https://aietalks.com/talks/bringing-agents-onto-the-world-wide-web) (Paul Klein IV, Browserbase, 18:26)
- [Your Agent Didn't Fail. Your Harness Did.](https://aietalks.com/talks/your-agent-didnt-fail-your-harness-did) (Vinoth Govindarajan, OpenAI, 18:26)