# AI Agents Don't Read Your Policy Docs. They Hit Your APIs

 | AI Engineer World's Fair 2026 | 17:50

Source: https://www.youtube.com/watch?v=7v2myBde05o
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/ai-agents-dont-read-your-policy-docs-they-hit-your-apis
Published: 2026-10-10
Tags: caching, guardrails, mcp, security

## TL;DR
- Governance written in prompts or policy documents cannot stop an agent from calling an available tool.
- A gateway can block destructive actions, personal-data requests, repeated calls, and oversized prompts before they reach backend systems or an LLM.
- Gateway controls also reduce cost and vendor dependence through rate limits, semantic caching, token limits, model routing, and smaller MCP tool sets.

## Summary
Sam argues that agent governance belongs in the infrastructure around an agent rather than in its prompt, skills, or markdown policy files. He demonstrates this with five live hotel-booking scenarios. A gateway blocks a request to delete all bookings and rejects a request for guests' personal data. It rate-limits a buggy agent that repeats the same request eight times, serves repeated questions from a semantic cache, and rejects a prompt that is too large before sending it to an LLM. Sam also shows how gateway telemetry can expose activity across agent-to-agent flows, how model routing can reduce dependence on one hyperscaler, and how narrowing an MCP server's tool list reduces context and cost. He applies the same idea to shadow AI on employee laptops, where edge management can provide visibility and force traffic through governance controls.

## Key ideas
### Policy documents cannot stop an agent from using an available tool
[00:29](https://www.youtube.com/watch?v=7v2myBde05o&t=29s)
Sam contrasts the number of organizations with agents taking action and the smaller number that have governance around them. He says agents are designed to be helpful, so an agent may follow a request to delete a database or return customer data even when a policy document says not to. In his hotel example, the ungoverned agent sees a tool named "delete all bookings" and calls it. The policy existed, but the agent did not consult it at runtime. Sam's point is that a rule in a prompt or markdown file cannot enforce permissions on its own.

### Gateway identity controls block destructive actions at execution time
[01:32](https://www.youtube.com/watch?v=7v2myBde05o&t=92s)
The governed hotel-booking agent discovers the same tools as the ungoverned one, but the gateway controls its runtime identity. Token exchange and JWT claims are scoped to the user and task. When the agent tries to call the destructive tool, the gateway returns a 403 forbidden response because the identity lacks permission. The agent can still cancel the individual booking, then tells the user it could not delete all bookings. Sam also shows request flow and database results to demonstrate that the control applies to the tool call rather than merely changing the agent's wording.

### The same gateway can stop personal-data exposure before it leaves the system
[04:46](https://www.youtube.com/watch?v=7v2myBde05o&t=286s)
In the second scenario, a user asks for every guest's email and card details under the pretext of reconciliation. The ungoverned flow returns the requested information because the model finds a suitable tool and tries to help. The governed flow blocks the request at the gateway because the agent lacks permission. Sam says this prevents the PII leak and can direct the user toward authentication if access is legitimate. He connects this control to observability across user-to-agent and agent-to-agent calls, where permissions inside individual agents become difficult to manage as the number of agents grows.

### Rate limits stop runaway calls before they consume model tokens
[06:56](https://www.youtube.com/watch?v=7v2myBde05o&t=416s)
Sam runs a buggy agent that sends the same request eight times. Without a gateway limit, every request reaches the model and receives a response. In his example, the repeated requests consume about 1,000 tokens, which he describes as roughly one cent for that small case. The gateway allows the first three requests, then applies a limit of three per minute. The later calls are blocked before they reach the LLM. Sam presents this as a way to control both spend and the amount of traffic allowed for particular users or audiences.

### Semantic caching avoids repeated LLM calls for similar questions
[08:21](https://www.youtube.com/watch?v=7v2myBde05o&t=501s)
The semantic-cache example repeats a request to list hotel bookings five times. Without caching, every request goes to the downstream LLM. With caching, similar prompts can be answered from a built-in Redis store. Sam says the cache can provide zero downstream model cost and latency for a cache hit, while the example uses about 200 tokens for the input and output instead of another full model call. Similarity is configurable on a scale from zero to one, so the threshold can vary by agent, audience, or user. The gateway logs whether a response came from the cache.

### Token limits reject oversized context before the model processes it
[11:01](https://www.youtube.com/watch?v=7v2myBde05o&t=661s)
Sam shows a user adding a long chat transcript to a simple hotel-booking request. The ungoverned example sends about 2,000 input tokens and produces 312 output tokens. The gateway blocks the same request before any downstream LLM call, using a policy that asks the user to split the context into smaller steps. Sam says this check must happen before the prompt is sent, because checking after the model has processed it would not avoid the token cost. He also describes content controls that reject requests unrelated to the agent's use case, such as asking hotel-booking software for the weather.

### Gateway controls can reduce dependence on one model provider
[15:01](https://www.youtube.com/watch?v=7v2myBde05o&t=901s)
Sam describes model routing through the gateway based on latency, cost, or task complexity. In his example, the gateway routes to Groq and Anthropic, while the same setup could route to Gemini or OpenAI within a few clicks. He says the gateway handles transformations between different OpenAI specifications, so an application does not need to be recoded for each provider. This separates the front-end agent from the backend model choice and lets teams change the downstream model without changing every agent.

### Smaller MCP tool lists reduce context and limit the blast radius
[15:41](https://www.youtube.com/watch?v=7v2myBde05o&t=941s)
Sam says the CRM MCP server used by his organization exposes 64 tools. Passing all of those tools into a model creates a large context and adds token cost on every interaction. He recommends creating a narrower MCP configuration that exposes only the tools an agent needs. This reduces the amount of information placed in the model's context and limits what the agent can call. In his framing, the narrower tool set also reduces the potential damage if an agent behaves incorrectly.

### Edge controls are needed for shadow AI on employee laptops
[16:16](https://www.youtube.com/watch?v=7v2myBde05o&t=976s)
Sam describes users running AI locally on laptops and putting documents, emails, or other material into tools outside the organization's gateway. Those requests are invisible to the central traffic path. He says Gravity released an edge-management capability that can discover this shadow AI traffic, secure it, log it, and force it through the gateway. He connects the feature to visibility and compliance needs in regulated industries. The same infrastructure approach therefore applies to agent traffic inside an application and to unsanctioned AI use at the edge.

## Notable quotes
- "The question isn't like how do we kind of secure the agent within its own kind of context or prompt is how do we remove that to be within its identity so it can't do those things in the first place rather than relying on a user prompting it correctly or some kind of markdown files kind of limiting what it can do." (01:11)
- "The agent doesn't care. Like, at runtime, no one reads those documents." (03:56)
- "They're blocked at the gateway. And again, that allows us to save costs, but also start to kind of control which users, which permissions, ultimately how much you want to spend in certain audiences." (07:56)
- "Basically said this is too big, blocked at the gateway level with a built-in policy." (11:51)
- "The only real correct way that we're going to scale but also keep the governance and securability as we move forward is shifting it up to the infrastructure layer." (17:12)

## Tools & references mentioned
- Gravitee
- Gravity Gateway
- Databricks
- Sonnet
- Groq
- Anthropic
- Gemini
- OpenAI
- Redis
- OpenTelemetry
- DataDog
- Splunk
- Prometheus
- MCP
- A2A protocol
- JWT
- GDPR
- HIPAA
- ICO

## Who should watch
- You are putting agent permissions in prompts or markdown files and need controls that apply when tools are called.
- Your agents make repeated model calls, accept large user context, or handle personal data, and you want to stop those requests before they create cost or exposure.
- You are building an agent platform across several model providers or need visibility into AI use on employee devices.

## Related talks

- [Your AI Isn't the Problem. Your Infrastructure Is.](https://aietalks.com/talks/your-ai-isnt-the-problem-your-infrastructure-is) (Jiachen Jiang, Coder, 31:30)
- [Paving the Road to Production: Governed Enterprise AI Agents](https://aietalks.com/talks/paving-the-road-to-production-governed-enterprise-ai-agents) (Harish Gaggar, Credit Karma, 29:27)
- [IT Admin for the AI Workforce](https://aietalks.com/talks/it-admin-for-the-ai-workforce) (Sarthak Aggarwal, Decawork, 16:17)
- [An AI Future Without the Lock-In](https://aietalks.com/talks/an-ai-future-without-the-lock-in) (Remy Guercio, Tailscale, 19:38)
- [Gateways are All You Need](https://aietalks.com/talks/gateways-are-all-you-need) (Karan Sampath, Anthropic, 17:48)