# Stop Babysitting Your Agents

Brandon Waselnuk, Unblocked | AI Engineer Europe 2026 | 18:54

Source: https://www.youtube.com/watch?v=BiG2ssibKGc
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/stop-babysitting-your-agents
Published: 2026-05-26
Tags: code-generation, context-engineering, mcp, rag

## TL;DR
- Agents fail because they lack an understanding of the organization, even when they can access its tools and data.
- Naive RAG, more MCPs, and larger context windows do not make an agent reason across the information it needs.
- A context engine researches across code, conversations, ownership, and organizational structure, then gives the agent a small research packet before it acts.

## Summary
Brandon Waselnuk argues that software agents start like highly capable engineers who know nothing about a company. Engineers currently provide the missing context by correcting plans, pointing to files, and explaining internal patterns. Unblocked's approach is a context engine that gathers and reasons across code, SaaS systems, Slack, Microsoft Teams, PR history, and organizational relationships before the agent writes code. Waselnuk explains why naive RAG can stop after its first plausible result, why MCP connections provide access without understanding, and why a large context window cannot replace structured reasoning. The engine should resolve conflicts, respect permissions, personalize retrieval, and compress its findings into a token-efficient research packet. A comparison in the talk shows an MCP-only agent producing code that compiled but would have broken the system, while the context-engine run received one senior-engineer nitpick. He also demos an open-source social graph component and a Zendesk integration workflow.

## Key ideas
### Agents begin with almost no organizational context
[00:15](https://www.youtube.com/watch?v=BiG2ssibKGc&t=15s)
Waselnuk describes an agent launched from a CLI as a brilliant software engineer that knows nothing about the task, the organization, or the systems around it. Engineers currently fill that gap by moving through the same learning process they experienced when joining a company. They learn from rejected pull requests, conversations with teammates, and repeated work. Over time, they learn which questions to ask, which information is accurate, and which details are irrelevant. In current agent workflows, the human is still performing this context-building work before the agent can act effectively.

### Static instruction files cannot contain all the context agents need
[02:44](https://www.youtube.com/watch?v=BiG2ssibKGc&t=164s)
Teams are moving toward a curated context layer with files such as CLAUDE.md and AGENTS.md, along with other documents containing company rules and patterns. Waselnuk says these stores are useful but static. Someone must maintain them, and they do not contain the raw runtime information engineers need for a particular task. A context engine should combine this standing information with live data when a query arrives. For a request such as implementing an integration, it can find the organization's factory patterns, fallback infrastructure, and related material across its knowledge systems.

### The main gap is understanding rather than access
[04:44](https://www.youtube.com/watch?v=BiG2ssibKGc&t=284s)
Waselnuk separates access from understanding. MCPs give an agent pipes into SaaS tools and other data sources, but the agent may still not know which service matters or what it does not know. In the example from the talk, an agent with the required MCP access wrote code that passed checks and compiled. A senior engineer judged the implementation completely wrong and said it would have broken the entire system if shipped. With the context engine, the same task produced a pull request that needed one small nitpick before approval.

### Naive RAG can stop at the first plausible answer
[05:58](https://www.youtube.com/watch?v=BiG2ssibKGc&t=358s)
Waselnuk calls the first myth the idea that naive RAG over documentation is enough. He borrows the term "satisfaction of search" from radiology, where finding one abnormality can cause a reader to stop looking for others. An agent can behave the same way. Asked to create a Zendesk integration, it may find one piece of data through an MCP, assume that it is the organization's pattern, and stop investigating. If the search is not exhaustive, the agent can miss the correct implementation or the root cause. The engineer then has to point it toward the right files and enter another correction loop.

### More MCPs and a larger context window do not create reasoning
[07:03](https://www.youtube.com/watch?v=BiG2ssibKGc&t=423s)
Connecting more MCPs gives an agent more possible sources, but it does not make the agent reason across them. Waselnuk also rejects the assumption that a million-token context window solves the problem. The data can sit in the window without useful entities or relationships, and the model still cannot reason effectively over an arbitrarily large collection. An agent could spend a long session discovering factory patterns and other codebase details, but that consumes many tokens and the work disappears when the terminal session closes.

### Social and context graphs help agents resolve relevance and conflicts
[08:08](https://www.youtube.com/watch?v=BiG2ssibKGc&t=488s)
A context engine can use a social graph to understand who is asking, which codebases they work in, who reviews their pull requests, and which people or systems are relevant to the question. It also needs to handle disagreements between sources. Waselnuk gives the example of code in main that conflicts with a Slack message where the CTO says the implementation was wrong. The engine should provide both pieces of information and account for the authority and relationship attached to the message. He says this kind of truth handling is difficult and is not fully solved.

### Context engines need permissions and token-efficient output
[09:36](https://www.youtube.com/watch?v=BiG2ssibKGc&t=576s)
The engine must preserve the permissions of its source systems. Waselnuk says a user should receive answers from their own private conversations, while another user should not see those private chats. He also describes an output that chooses the right context and model at the right time, then returns only the details the agent needs. The system ingests many sources, reasons across them, and compresses the result into a smaller response. This lets background agents ask questions without waking a human to approve every tool or data access decision.

### Fresh answers are safer than blindly cached answers
[13:11](https://www.youtube.com/watch?v=BiG2ssibKGc&t=791s)
Unblocked initially tried caching good answers to reduce latency. Waselnuk says this failed because a correct answer can become false as the system changes. A response that was accurate when someone documented it may be wrong when the same question is asked the next day. Some questions are stable, but caching answers broadly created problems. He recommends being cautious about this optimization, even when the cached response was previously correct.

### The engine supports planning, execution, code review, and internal support
[13:44](https://www.youtube.com/watch?v=BiG2ssibKGc&t=824s)
Waselnuk describes context engines answering questions in an organization's ask-engineering channel, where support or sales staff can ask what is running in production. Other uses include ticket enrichment, triage, incident management, coding agents, custom skills, and workflows based on standard operating procedures. In his Zendesk demo, the engine first creates a research packet, then lets exploration agents search the right places. The resulting plan identifies the provider registration, factory pattern, library modules, and client work. He recommends using the engine for planning, during execution, and again during code review.

## Notable quotes
- "The gap is not intelligence at this point. It is context." (04:44)
- "The problem is people think that access is the answer, but it is not understanding." (04:44)
- "An agent should write code that feels like it was written by someone who's been on your team for years." (14:53)
- "If you cache a correct answer, and then tomorrow someone asks the same question, and you answer it, you probably lied to them now because things probably changed." (13:25)

## Tools & references mentioned
- Unblocked
- MCP
- Claude
- CLAUDE.md
- AGENTS.md
- Basim Eldis
- Andrej Karpathy
- Anthropic
- Claude agents
- Bedrock
- Zendesk
- Slack
- Microsoft Teams

## Who should watch
- You are adding coding agents to a large organization and keep correcting implementations that compile but ignore internal patterns.
- Your team has connected agents to many systems through MCPs, but they still stop at the first plausible result.
- You are designing background agents that need company context, permission-aware retrieval, planning, or code review without waiting for a human.

## Related talks

- [Why More Context Makes Your Agent Dumber and What to Do About It](https://aietalks.com/talks/why-more-context-makes-your-agent-dumber-and-what-to-do-about-it) (Nupur Sharma, Qodo, 26:27)
- [Building Agents Is Trivial Now, Context Is the Next Frontier](https://aietalks.com/talks/building-agents-is-trivial-now-context-is-the-next-frontier) (Jeff Ng, Unblocked, 13:22)
- [Demand-Driven Context: A Methodology for Coherent Knowledge Bases Through Agent Failure](https://aietalks.com/talks/demand-driven-context-a-methodology-for-coherent-knowledge-bases-through-agent) (Raj Navakoti, IKEA, 1:08:15)
- [Agentic Engineering: Working With AI, Not Just Using It](https://aietalks.com/talks/agentic-engineering-working-with-ai-not-just-using-it) (Brendan O'Leary, Kilo Code, 27:03)
- [Building Agent Interfaces: Lessons from Chrome DevTools (MCP) for Agents](https://aietalks.com/talks/building-agent-interfaces-lessons-from-chrome-devtools-mcp-for-agents) (Michael Hablich, Google, 22:38)