Nathanael Weill tests agents by writing a known answer first, generating an input that should lead to it, and checking the result with code.
2
For supplier-contract review, he builds tests from past decisions, planted rule violations, removed clauses, harmless changes, and contract templates with filled-in blanks.
3
He avoids LLM judges when exact matching or other code checks can grade the output, because judge models added more noise than the agent itself.
Summary
Nathanael Weill describes how Air Canada builds agents for narrow business tasks when no public benchmark exists and domain experts have little time to grade results. His main method reverses the usual evaluation process: start with a known answer, generate or modify the input around it, then check whether the agent finds the expected answer. A telecom RAG project went from one expert review a week to dozens of automated evaluations a day. He applies the same idea to a supplier-contract agent that finds clauses, gathers supporting documents, writes amendments, and classifies them. The test set includes planted rule violations, removed clauses, changes that should not affect the result, and template-generated contracts. Weill prefers code-based checks over LLM judges when possible. He also uses expert feedback to create better tests rather than repeatedly fixing the agent by hand.
Narrow business agents often have no usable benchmark
At Air Canada, Weill's team builds AI solutions for operations, corporate, commercial, and loyalty work. The tasks are specific enough that public benchmarks do not apply. The engineers can build agents, but they are not domain experts. Those experts sit in other business units and have their own work, so access is limited. This creates two problems at once: the team lacks a ready-made test set, and the people who can judge answers cannot spend much time doing so. Weill presents the rest of the talk as a response to that practical situation, rather than as a general benchmark for agents.
Weill describes an earlier telecom Q&A assistant built over roughly 10,000 documents, including vendor material. The hard part was finding the right document for a question. His team built a complex RAG pipeline and kept trying published techniques, but added complexity often broke something that had worked before. The team could change an agent in minutes, then had to wait for a weekly meeting where experts compared several chatbots on the same questions. The experts sometimes disagreed, and some had to look up the answer before grading. The result was a budget of about one trial a week, which was too slow for useful iteration.
Writing the answer first turns evaluation into a check
Weill's solution was to reverse the evaluation process. Instead of asking an agent a question and asking an expert whether the answer is good, the team starts with a page whose contents are known. An LLM writes a question about that page. The agent then searches for the answer, and code compares what it found with the source page used to create the question. This removes the need for an expert opinion on every run. Weill says the approach took the telecom project from one evaluation a week to dozens a day. The team also removed unnecessary complexity and focused on the embedding model, chunking, and BM25 alpha.
The Air Canada project analyzes supplier contracts. Experts inspect clauses, decide whether they comply with rules, and suggest amendments. Missing a clause can delay a supplier, while overlooking a problem can create later costs. The agent has four simplified steps: identify the relevant clause, find documents that support an amendment, write the amendment, and classify whether the amendment should be kept. Weill could reuse the answer-first idea for finding supporting documents, because that part resembled RAG. The other parts had no established evaluation method, so the team developed several ways to generate contracts with known expected outcomes.
Past decisions help only when the original documents still exist
The first proposed test method uses past decisions. The team would compare an original signed contract with a later changed version, then check whether the agent produces the same flags as the real review. That would provide a direct answer and a realistic input. In practice, the team did not retain enough original versions. It often had only the latest contract, while negotiations and multiple versions had occurred earlier. Without pairs of original and changed contracts, this method could not supply the needed test set. The example shows why an attractive evaluation plan can fail because the required historical data was never kept.
Planted rule violations create tests with known answers
The team also starts from a contract rule, such as a cap above one million dollars. It creates a clause that breaks the rule, generates the surrounding contract, and checks whether the agent finds the planted violation. The method is simple and can produce many tests, but it depends on knowing the rules. Weill's team gathered rules from experts and existing documentation, then added rules of its own. This approach produced 275 tests whose expected answers were known. The team had effectively planted the problem and asked the agent to find it. Coverage still depended on which rules the team knew.
Removed clauses test whether the agent notices what is missing
Another method begins with a signed contract and removes a clause. The agent should report that something is missing. Weill calls this difficult because suppliers can use different templates, and deciding what belongs in a particular contract requires domain judgment. The team mixes these altered contracts with existing contracts that should produce no flags. This gives the evaluation both positive and negative cases. Weill says the approach was the most powerful one they had at that point, because it generated many contracts and exposed that the agent was missing most of the removed clauses.
Harmless changes and templates expand code-checkable coverage
The team changes details that should not affect the review, such as a supplier's name or address, and checks that the number of flags stays the same. It also reorders clauses and reruns the same contract to test consistency. A fifth method uses contract templates with blanks. Instead of filling each blank manually, the team creates arrays of possible values and combines them into many contracts. Some combinations break rules and others do not, so the expected flags are known. Because Air Canada's experts already use templates, the team can reuse them. A script returns an evaluation in about five minutes after an expert feedback cycle that took four weeks.
Code checks are usually less noisy than LLM judges
Weill prefers exact matches and other code-based checks whenever the expected result can be defined. He says LLM judges produced more noise than the agent under test in this project. Getting a judge to work required finding the right prompt and harness, along with possible fact checking and other extra steps. He does mention cases where model-based checks may be useful, including agent trajectories with known endpoint order. He also mentions text-to-SQL as another setting where the system can generate an input and check whether the agent retrieves the known answer. Sometimes his team changed the agent design so it could be evaluated at all.
