Larger models can be more vulnerable because they understand attack instructions that smaller models miss, while broader agents have more infrastructure access to abuse.
2
An agent specification needs more than example inputs and outputs. It should define rules, domain knowledge, rights, roles, and robustness limits such as tolerance for typos and rephrasing.
3
Specifications should stay separate from the agent implementation so the same integration, unit, security, and penetration tests can survive model or infrastructure changes.
Summary
Steven Willmott argues that evaluating an AI agent requires more than an eval dataset. A larger model may perform better, but it can also understand jailbreaks that smaller models miss. An agent with a broad remit has more tools and infrastructure access, which increases both its attack surface and the amount that needs testing. Willmott proposes writing an agent specification independently of its implementation. The specification can include expected examples, business rules, domain ontologies, internal terminology, permissions, roles, and robustness requirements. A customer support agent might never offer more than a 10% discount, while an airline agent should stay within the destinations its airline serves. These specifications can drive normal evaluation, security checks, and tests that vary wording or introduce typos. Willmott also wants specifications versioned in a shared repository so teams can reuse them when they change models or agent frameworks.
A larger model can understand jailbreaks that a smaller model misses
Willmott uses a poem-wrapped instruction to show why model size does not directly determine safety. A lower-end model may fail to understand the poem, while a larger model can extract and execute the malicious instruction inside it. Broader agents create another problem because they can perform more tasks and access more infrastructure. An agent that can move millions of dollars is riskier than one that only answers questions. Large models can also cost more and run more slowly for simple tasks. Willmott's practical target is a model capable enough for its job without being able to cause arbitrary harm.
An agent specification must describe the role independently of the model
Willmott defines spec-driven validation by asking what should be true if the task were designed independently of the agent. A dataset of good examples is one part of the specification, but it does not capture every requirement. The specification should describe the task and role in terms that remain useful when the underlying model or framework changes. This gives teams something more concrete than a collection of inputs and outputs, and it makes expected behavior available for both evaluation and security work.
Business rules need explicit tests because violations are easy to miss
A support agent may have rules such as never offering more than a 10% discount or refusing refunds after 30 days. Willmott points out that it is difficult to establish that a rule is never violated. The rule therefore needs to be written into the testing specification rather than left as an informal expectation. The same applies to distinctions that matter in a business domain. Gross profit and gross sales may look substitutable to a general language model, but they have different meanings in business.
Domain ontologies, permissions, and roles define the valid operating space
The testing system needs to know the relevant universe for the agent. An airline chatbot should be tested against the destinations that airline actually serves. Company-specific terminology and policies also need to be included, along with valid substitutions. The specification can then account for differences between logged-in and logged-out users and between users with different rights. These details affect what the agent may say and what it may do, so they belong in the agent's test definition.
Robustness requirements describe how much variation the agent can tolerate
A test set should work under the conditions users create in practice. For an agent, that includes questions with typos, different phrasings, and changes in wording that should preserve the same result. Willmott compares this with testing a vision system in fog, at sunrise, or with camera shake. The specification should state how much variation is acceptable and when the system has failed. This turns robustness into a task-specific requirement instead of a general hope that the model will cope.
The same specification can drive evaluation and security testing
Willmott describes using the task specification for two related purposes. Security testing can focus on the domains the agent is meant to discuss, where it is likely to have relevant permissions and tools. The tasks it performs also identify where it has power inside the surrounding infrastructure. Robustness testing can vary inputs within the permitted range and measure whether the agent continues to answer correctly. He compares these tests to integration tests from ordinary software engineering.
Specifications should survive changes to models and agent frameworks
Willmott recommends keeping the behavior definition independent of tools such as LangSmith or Vertex agents. Teams should be able to retain their integration tests, unit tests, and penetration tests when they move to different infrastructure. He also describes an outer feedback loop that runs the agent, records failures, and tries to close robustness gaps. This is not reinforcement learning on the model. It is an external process for iterating on the agent and its tests.
Versioned specifications could become shared engineering artifacts
Willmott wants the different parts of an agent specification to live in a GitHub repository and be pulled into whichever testing or development tool a team uses. Versioning the rules, domain definitions, permissions, and robustness requirements would let teams track changes over time and reuse the same material across systems. His background in API infrastructure and the OpenAPI specification informs this idea. The aim is a portable description of agent behavior rather than a definition trapped inside one product.
"And so really this is the point here is we need to go beyond the test set to have like task and role specific benchmarks that are for the agent itself."07:27
Who should watch
You are deploying an agent with tools or permissions that could cause financial, operational, or customer-facing harm.
Your current evaluation is mostly a dataset of example prompts and answers, and you need to encode business rules or user permissions.
You expect to change models or agent frameworks and want tests that remain usable after the implementation changes.
