# Your coding agent doesn't always follow your rules

Talha Sheikh, Checkout.com | AI Engineer Europe 2026 | 10:08

Source: https://www.youtube.com/watch?v=MpZzWMdmQCE
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/your-coding-agent-doesnt-always-follow-your-rules
Published: 2026-07-08
Tags: agents, coding-agents, cost, evals, testing

## TL;DR
- Coding agents can report that a task is complete while still failing requirements when someone runs the result.
- A harness adds deterministic checks after an agent session and can send failures back to the agent for another attempt.
- Verification rules can reduce dependence on large models, but the verification layer itself becomes the work that needs investment.

## Summary
Talha Sheikh describes the gap between an agent completing a coding task and completing it according to the developer's requirements. His first response was Vector V1, a program called through Claude hooks after a session ends. It reads a config file containing test cases, checks the agent's output, and sends failures back for another attempt. Sheikh argues that stronger models, better specifications, MCP servers, and sub-agents improve capability without guaranteeing reliability. He then broadens the idea from a personal tool into a language-agnostic enforcement pattern that can run during conversations, before commits, inside multi-agent workflows, and during asynchronous work. He points to similar approaches from Anthropic, OpenAI, Cudo, and WorkOS. His practical conclusion is that teams should spend more effort designing verification systems. Those systems may allow smaller models to produce acceptable results, although the checks themselves add time and engineering work.

## Key ideas
### A completed agent task still needs to be checked by a person
[00:07](https://www.youtube.com/watch?v=MpZzWMdmQCE&t=7s)
Sheikh starts with a familiar workflow: he gives Claude Code a feature, watches it split the work into subtasks and sub-agents, then receives a final "Task completed" message. When he runs the result, something is often missing or broken. He keeps asking the agent to fix one more issue. Even a detailed specification does not remove this problem. With little or no instruction, there is still something he has to correct. The agent can say it is done, but the developer remains responsible for checking whether it really works and whether it was implemented in the intended way.

### A harness can make agent output subject to deterministic checks
[01:04](https://www.youtube.com/watch?v=MpZzWMdmQCE&t=64s)
Sheikh says he wanted to play Cyberpunk on his Xbox while Claude Code worked without needing to supervise every result. He built Vector V1 as an enforcement layer and connected it through Claude hooks. When Claude finishes a session, the hook calls the Vector program. A config file defines the test cases, and the program checks the result. If a test fails, the system tells Claude what failed and asks it to try again. In his example, the first run has failures, the agent retries, and the later output passes the checks.

### More capable models do not automatically become more reliable
[03:03](https://www.youtube.com/watch?v=MpZzWMdmQCE&t=183s)
After an Anthropic engineer told Sheikh that a future agent or model might make enforcement unnecessary, he reconsidered the purpose of his work. His distinction is between capability and reliability. A new model may be able to do more, but that does not mean it will follow requirements more consistently. Better specifications, MCP servers, sub-agents, and added context can improve the instructions given to an agent. They still do not provide verification. The developer needs a separate way to test whether the requested result was actually produced.

### Guardrails can make smaller models practical for some tasks
[04:18](https://www.youtube.com/watch?v=MpZzWMdmQCE&t=258s)
Sheikh argues that a system with enough guardrails could use a smaller model, such as Haiku or an open-source model, because the checks constrain the output. A frontier Opus model may complete a task at a higher cost. Adding Vector and a small number of guardrails can reduce that cost, while investing more time in the harness could reduce it further. He also mentions asynchronous tasks as another option. The tradeoff is that savings in model usage require engineering effort in the verification system.

### Enforcement should be a shared pattern rather than one team's private tool
[05:09](https://www.youtube.com/watch?v=MpZzWMdmQCE&t=309s)
When Sheikh discussed Vector with more people, he found that Anthropic, his company, Facebook, and Meta were all building their own enforcement systems. A rule that makes sense for his product may not make sense for another team's product, so a personal implementation is difficult to share. He reframed the work as a general pattern. It should be language agnostic, allow each developer or organization to define its own enforcement, and run wherever the system can perform deterministic checks.

### Checks can run throughout a coding and agent workflow
[06:03](https://www.youtube.com/watch?v=MpZzWMdmQCE&t=363s)
Sheikh describes a contract that says, given a task, the system should fulfill it, while developers and other teams define what fulfillment means. Checks can happen inside the conversation, when a conversation ends, before a commit, as part of a multi-agent workflow, or during asynchronous agent operations. He also mentions a nondeterministic check that calls an LLM as a judge. The common requirement is the ability to run a check against the work and feed the result into the workflow.

### The engineering focus is moving toward verification
[06:56](https://www.youtube.com/watch?v=MpZzWMdmQCE&t=416s)
Sheikh connects his proposal to several systems and ideas from other companies. Anthropic's executed advisor pattern uses one agent to do the coding and another advisor to create a feedback loop. OpenAI's harness engineering approach adds tools and context and then considers how to verify the work. Cudo provides comprehensive code reviews after an agent has finished. WorkOS uses the phrase "enforce don't instruct." Sheikh's conclusion is that teams must slow down because verification takes time, even when agents can produce code quickly.

## Notable quotes
- "I am the enforcement layer." (01:22)
- "Giving it instructions is not the same thing as giving it verification." (04:02)
- "It has to be something that can be shared by everybody else, and everybody can bring their own version of enforcement to it." (05:46)
- "So, TLDR is work on the harness and not on the code." (08:27)

## Tools & references mentioned
- Claude Code
- Vector V1
- Vector Harness
- Claude hooks
- Anthropic
- Claude Swing
- Project Methuselah
- MCP servers
- Haiku
- Opus
- OpenAI
- Cudo
- WorkOS
- Facebook
- Meta
- Cyberpunk

## Who should watch
- You are building coding agents that often report success before tests or product requirements have been checked.
- Your team is deciding whether to invest in a verification harness, agent instructions, larger models, or more sub-agents.
- You want to run cheaper models for constrained coding tasks and can afford to build the checks that constrain their output.