# Skill Issue: How We Used AI to Make Agents Actually Good at Supabase

Pedro Rodrigues, Supabase | AI Engineer Europe 2026 | 1:18:41

Source: https://www.youtube.com/watch?v=GmAQKINjv1E
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/skill-issue-how-we-used-ai-to-make-agents-actually-good-at-supabase
Published: 2026-05-04
Tags: agent-skills, agents, evals, mcp, security

## TL;DR
- Agent Skills give agents progressive access to instructions, reference files, and scripts without loading everything into context at once.
- Skills and MCP solve different problems: MCP provides integrations and tools, while Skills explain workflows and product-specific rules.
- A useful Skill needs an evaluation loop that compares agent behavior with and without the Skill, while checking that the evaluation itself measures the intended behavior.

## Summary
Pedro Rodrigues explains how Supabase uses Agent Skills to guide coding agents through product-specific workflows. He starts with the Skill format, where a small front matter block tells the agent when a Skill applies and linked files provide more detail only when needed. He then compares Skills with MCP, arguing that MCP is better for integrations and remote tools, while Skills provide instructions, workflows, and scripts. The workshop demonstrates this with a Supabase performance review app. An agent creates a database view, but misses PostgreSQL's security-invoker requirement and bypasses row-level security. A Supabase security Skill adds that requirement and changes the agent's migration. Rodrigues then builds an evaluation harness with scenarios, assertions, and with-Skill and without-Skill conditions. He is honest about the difficulty: evals can fail because the grading logic checks the wrong thing, and agent behavior is nondeterministic. Production Skills need versioning, maintenance, and cleanup like other CI artifacts.

## Key ideas
### A Skill is a small entry point into a larger workflow
[02:16](https://www.youtube.com/watch?v=GmAQKINjv1E&t=136s)
Rodrigues describes Skills as folders containing instructions and files for repeated workflows, custom information, or scripts. The required file is skill.md, with front matter containing a name and description. The description tells the agent what the Skill does and when it should load the rest. Reference files can contain ordinary Markdown, while scripts can be written in Bash, Python, or another suitable language. Reference files may link to other files, so a Skill can form a graph rather than a single document.

### Progressive disclosure keeps unused detail out of the agent context
[04:38](https://www.youtube.com/watch?v=GmAQKINjv1E&t=278s)
The main benefit Skills add around tools such as MCP is progressive disclosure. The agent first receives the Skill's front matter, which acts as an envelope. It can then decide whether to load the rest of skill.md and its referenced material. Rodrigues compares this to a book: skill.md is an expanded index, and the reference files are its pages or chapters. This lets a Skill carry detailed product knowledge without placing all of it in the initial context.

### Skills and MCP should be used together for different jobs
[07:11](https://www.youtube.com/watch?v=GmAQKINjv1E&t=431s)
Rodrigues says an integration should use MCP, especially when the agent needs access to a service or does not have Bash. MCP tools can run remotely and include authentication through the protocol. Skills provide the context MCP tool descriptions do not have room to contain, including workflows and product-specific instructions. Scripts run in the local environment and depend on its operating system. In his database example, he recommends an MCP tool to retrieve schema information and a Skill to explain how the agent should retrieve it in chunks.

### The workshop exposes a security bug that a coding agent misses
[21:26](https://www.youtube.com/watch?v=GmAQKINjv1E&t=1286s)
The demo application is a performance review system with employees, managers, and an HR representative. Rodrigues asks an agent to create a department statistics view showing headcount and average salary. The agent creates the view and reports success, but Bob, an engineering manager, can see HR and product information. An employee can also see data that should be restricted. The problem is PostgreSQL view behavior: by default, a view can use the permissions of its creator and bypass row-level security on the underlying tables.

### A product Skill can add database rules the model does not reliably infer
[34:49](https://www.youtube.com/watch?v=GmAQKINjv1E&t=2089s)
The prepared Supabase security Skill tells the agent to use the security-invoker option when creating the view. Rodrigues explains that, since PostgreSQL 15, this option enables the row-level security policies on the view. The Skill also includes checks for row-level security on public or exposed schemas. When the agent loads the Skill, its migration includes the security-invoker flag. The live demo still has other policy problems, so the final application behavior is not fully fixed, but the agent's database change clearly differs after the Skill is loaded.

### Skill loading itself needs to be tested
[43:05](https://www.youtube.com/watch?v=GmAQKINjv1E&t=2585s)
An agent may decide whether to load a Skill from its description, so Rodrigues recommends testing both cases where it should load and cases where it should not. Developers can inspect the Claude Code CLI to see whether the Skill loaded, then revise the description and repeat the tests. He found that putting the verb "use" at the start of the description increased loading in his Claude experiments. A slash command followed by the Skill name gives an explicit way to load it in Claude Code.

### Eval scenarios must describe the behavior that actually matters
[1:02:11](https://www.youtube.com/watch?v=GmAQKINjv1E&t=3731s)
Rodrigues says the hardest part of creating evals is writing representative scenarios and deciding what good behavior means. An eval can include a prompt, expected output, tool-call assertions, and success criteria for an LLM judge. The Agent Skills Open Standard format he uses stores these scenarios in eval.json. The harness resets the local database, runs the agent headlessly, and tests conditions with and without the Skill. This creates a first automated comparison instead of relying only on a manual demonstration.

### Bad grading logic can reverse the apparent result
[1:10:56](https://www.youtube.com/watch?v=GmAQKINjv1E&t=4256s)
The workshop's evaluation initially reports that the run with the Skill failed and the run without it passed. Rodrigues finds that the assertion was inspecting the wrong database metadata instead of checking the view directly. Both outputs were effectively the same, while the manual test had already shown that the Skill caused the security-invoker flag to be added. His example shows that evaluation failures can come from the evaluator rather than the agent. LLM-based grading adds another source of variation because the judge can also behave nondeterministically.

### Production Skills need the same maintenance as documentation and CI artifacts
[1:13:31](https://www.youtube.com/watch?v=GmAQKINjv1E&t=4411s)
For production, Rodrigues recommends keeping only the Skills needed for the particular environment or workflow. He says Skills should be treated as documentation that lives in the repository, with updates included in the normal research and development workflow. Teams can periodically check whether a Skill still completes its intended workflow and whether users still load it. A Skill that is no longer used may not deserve to remain installed, even though progressive disclosure makes unused descriptions relatively cheap.

## Notable quotes
- "The secret sauce has been basically skills." (02:52)
- "If you're building anything that it's an integration you should use MCP." (07:11)
- "The main part of skills is that you can change the behavior or guide the agent on demand based on information that you put." (52:25)
- "The most difficult part to create evals, I would say, is actually coming up with the scenarios." (1:02:20)

## Tools & references mentioned
- Supabase
- PostgreSQL
- MCP
- Claude Code
- Anthropic
- OpenAI
- Braintrust
- Langfuse
- Agent Skills Open Standard
- Vercel skills package
- Firebase
- Cursor
- Google DeepMind

## Who should watch
- You are writing Skills for coding agents and need a way to tell whether they change behavior rather than merely add documentation.
- Your agent works with Supabase or PostgreSQL and can create database objects without reliably applying row-level security rules.
- You are building an eval harness and want to avoid declaring a Skill ineffective because the assertions or judge are checking the wrong thing.

## Editor's note

From the pack [Agent skills](https://aietalks.com/packs/skills):

Gomes checks both the assigned worktree and the primary checkout, while Klingen and Schmid compare behavior before and after a skill changes. Those tests need more than the agent's final answer. Kitaru records the inputs, outputs and tool calls from a run, then replays the same task so a team can see whether a revised skill changed the route as well as the result.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Combine Skills and MCP to Close the Context Gap](https://aietalks.com/talks/combine-skills-and-mcp-to-close-the-context-gap) (Pedro Rodrigues, Supabase, 18:27)
- [Skills are the New SDKs](https://aietalks.com/talks/skills-are-the-new-sdks) (Elvin Aghammadzada, DataRobot, 26:40)
- [Don't Build Agents, Build Skills Instead](https://aietalks.com/talks/dont-build-agents-build-skills-instead) (Barry Zhang & Mahesh Murag, Anthropic, 16:22)
- [Full Walkthrough: Writing & Using Skills](https://aietalks.com/talks/full-walkthrough-writing-using-skills) (Nick Nisi & Zack Proser, WorkOS, 1:21:03)
- [Will Agent Evaluation via MCP Stabilize Agent Networks?](https://aietalks.com/talks/will-agent-evaluation-via-mcp-stabilize-agent-networks) (Ari Heljakka, Root Signals, 14:11)