# What if the harness mattered more than the model?

Aditya Bhargava, Etsy | AI Engineer World's Fair 2026 | 32:04

Source: https://www.youtube.com/watch?v=2e9ANoOEn28
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/what-if-the-harness-mattered-more-than-the-model
Published: 2026-07-07
Tags: agents, guardrails, harness-engineering, multi-agent, tool-use

## TL;DR
- A harness can change the same model's score from 52.4% to 76.2% on Harness Bench, with the harness as the only variable.
- Agent safety comes from constraining tools with interrupts, handlers, and partial function application rather than approving every action manually.
- A coding agent improves through a sequence of tools, safety controls, feedback loops, sub-agents, and measured self-optimization.

## Summary
Aditya Bhargava argues that agent development should put more effort into harnesses, the code around a model that gives it tools, controls its actions, and manages its loop. He points to Harness Bench, where the same model and task setup scored between 52.4% and 76.2% when only the harness changed. He then demonstrates this idea with Agency, a language he has been building for agents. A coding agent starts with only a model, then gains file tools, human approval through interrupts, restricted access through partial function application, and a reason-act-observe feedback loop that runs tests and fixes failures. Bhargava then adds sub-agents to separate unrelated capabilities and an optimizer to improve prompts against a measurable goal. His argument is that stronger harnesses could make local open-source models useful for tasks that currently depend on proprietary models.

## Key ideas
### Harness design can change performance substantially even when the model stays the same
[01:37](https://www.youtube.com/watch?v=2e9ANoOEn28&t=97s)
Bhargava cites Harness Bench, a benchmark with 106 tasks that evaluates the same models and setup inside different harnesses. The scores range from 52.4% to 76.2%, a difference of more than 20 percentage points when only the harness changes. He says the harness matters more for weaker models. That creates a path for improving a local model through software around it, rather than relying only on companies that train the most capable proprietary models. His larger proposal is to build harnesses good enough to bring cutting-edge performance to open-source models that can run locally.

### Building a serious harness may require language-level support
[03:19](https://www.youtube.com/watch?v=2e9ANoOEn28&t=199s)
Bhargava says existing tools and frameworks did not provide the capabilities he wanted, so he concluded that a language-level solution was needed. He has spent much of the previous six months building Agency, a language for building agents. Its design follows Alan Kay's idea that simple things should be simple and complex things should be possible. Common agent primitives, such as defining tools, should require little code. More advanced behavior should remain possible without forcing developers into a fixed framework. In the talk, Agency provides the harness while the model remains the same across the examples.

### Tools give an agent capabilities, but unrestricted tools are unsafe
[09:35](https://www.youtube.com/watch?v=2e9ANoOEn28&t=575s)
The first coding agent has only a prompt asking for a bug fix, so it cannot read or write the target file. Bhargava then adds tools. In Agency, any function can become a tool, and the language creates a JSON schema from the function and uses its docstring as the tool description. For the coding example, he gives the agent read and write functions. Agency refuses to let those functions access arbitrary files by default because that would be unsafe. Its standard library functions that mutate code, perform destructive actions, or read sensitive data raise an interrupt first.

### Interrupts add human approval, but approval for every action is slow
[12:12](https://www.youtube.com/watch?v=2e9ANoOEn28&t=732s)
Agency's interrupt and handler features let a harness pause before a sensitive action and ask a person to approve it. Bhargava shows the agent asking whether it should read a file and then asking again before writing to it. A handler receives information about the interrupt, asks for input with the built-in input function, and approves the tool call if the user agrees. This makes the file-editing agent safer, but it also makes it slow because a person has to approve each action. The next design problem is giving the agent autonomy without giving it unrestricted access to the file system.

### Partial function application constrains an autonomous agent without repeated approval
[15:18](https://www.youtube.com/watch?v=2e9ANoOEn28&t=918s)
Bhargava uses partial function application to lock the directory argument of the read and write tools to the demo directory. The model can provide a file name, but it cannot change the directory or even see that the directory argument exists. The agent can therefore act without asking for permission on every file operation, while its access remains limited to the intended directory. This version reads the code and produces the correct fix, but it only asks whether the user wants the file updated. It still does not complete the edit on its own.

### A reason-act-observe loop lets the agent verify and repair its own work
[17:48](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1068s)
The next harness adds a feedback loop based on the ReAct pattern, which Bhargava expands as reason and act. The agent reasons about what to inspect, acts by reading files and running tests, observes the test results, and reasons again from any failure. In the median example, it reads the code and tests, runs the tests, sees them fail, writes the correction, and runs the tests again. The second run confirms success. This is the first version that both changes the code and checks that the change works, rather than merely suggesting a solution.

### Sub-agents separate unrelated tools and add capabilities without bloating context
[21:38](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1298s)
Bhargava's sub-agent example asks the system to fix the failing median test and research Jensen's inequality for medians through Wikipedia. The main agent receives two sub-agents as tools: a coding agent with file tools and a Wikipedia agent with a search tool. They run in parallel, with one modifying the file while the other searches. In Agency, a sub-agent is just another function, so it uses the same model as a tool and does not need a separate framework concept. Bhargava says this grouping helps because agents often fail when they have too many unrelated tools and concepts in their context.

### Self-optimization replaces prompt guesswork with measured improvement
[25:25](https://www.youtube.com/watch?v=2e9ANoOEn28&t=1525s)
Agency includes optimizers, including the Japa optimizer shown in the talk. Developers mark variables such as prompts with an optimize modifier, then provide a goal. In the example, the starting prompts contain little information, while the goal says to fix median.py using test-driven development. The optimizer first establishes a baseline, then tries to improve the objective by rewriting the prompt. Bhargava shows a run where the objective is 0.2 at baseline and is achieved in the first iteration. He presents this as a way to measure and improve a harness systematically instead of repeatedly guessing and checking.

## Notable quotes
- "If a good harness can compensate, and can make a weaker model perform better, then you can build our own harnesses, and that's something that any of us can do, and we don't have to depend on paid models." (02:56)
- "Partial function application is a really great way to constrain the capabilities of your agent." (16:53)
- "Sub agents are cool because they let you add new capabilities without bloating context." (24:24)
- "The reason this self-optimization is so great is because we're not guessing and checking. We're systematically measuring and improving, which is a big leap forward." (27:39)

## Tools & references mentioned
- Etsy
- Grokking Algorithms
- ducktyped.org
- Harness Bench
- Agency
- Claude Code
- Opus
- Alan Kay
- ReAct
- Jensen's inequality
- Wikipedia
- Japa optimizer
- agencylang.com

## Who should watch
- You are building agents that need to edit files or take actions, and you need a practical way to limit what their tools can do.
- Your agent often produces plausible changes without checking whether they work, so you want to add a test-driven feedback loop.
- You want to compare harness improvements against local open-source models instead of assuming that a larger proprietary model is the only route to better performance.

## Related talks

- [Beyond the Harness: A Journey Towards Adaptive Engineering](https://aietalks.com/talks/beyond-the-harness-a-journey-towards-adaptive-engineering) (Rajiv Chandegra, Annicha Labs, 37:01)
- [Every Harness Will Become A Claw](https://aietalks.com/talks/every-harness-will-become-a-claw) (Sam Bhagwat, Mastra, 15:36)
- [Harnesses in AI: A Deep Dive](https://aietalks.com/talks/harnesses-in-ai-a-deep-dive) (Tejas Kumar, IBM, 20:27)
- [Your Agent Didn't Fail. Your Harness Did.](https://aietalks.com/talks/your-agent-didnt-fail-your-harness-did) (Vinoth Govindarajan, OpenAI, 18:26)
- [Harness Engineering & Startup Battlefield](https://aietalks.com/talks/harness-engineering-startup-battlefield) (Garry Tan, Y Combinator & Mike Krieger, Anthropic & Theo Browne, t3.gg & Maxime Rivest & Isaac Miller, DSPy, 9:11:15)