# Backtest Your Agent Against Yesterday's Web

Ryan Ahern, Exa | AI Engineer NYC 2026 | 26:42

Source: https://www.youtube.com/watch?v=L4dUFimVDKU
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/backtest-your-agent-against-yesterdays-web
Published: 2026-10-19
Tags: agents, benchmarks, evals, search

## TL;DR
- AI agents need search because they can explore multiple pages and refine queries beyond the way people usually search.
- Exa Snapshot lets a system search a historical version of the web, which prevents future information from leaking into a backtest.
- Backtests need holdout data because systems can overfit a small set of companies and produce rules that will fail on new cases.

## Summary
Ryan Ahern explains why web search is becoming part of agent systems and why historical search matters when those systems make predictions. Exa Snapshot stores versions of pages and supports searches constrained to information available before a chosen date. That lets a team test whether an agent could have predicted an event without seeing its outcome. Ahern uses company acquisitions as the hands-on example. Participants create yes-or-no features, such as executive turnover or reports that a company is exploring a sale, then test them one day, seven days, and 30 days before an acquisition. The exercise also exposes overfitting. Some winning rules simply memorised quirks of the small test set, including named companies. Ahern says a holdout set would have exposed this problem. The talk is strongest when it treats historical web access as an evaluation requirement rather than a way to make a prediction look better after the fact.

## Key ideas
### Agents search more deeply because they can keep querying and reading
[01:13](https://www.youtube.com/watch?v=L4dUFimVDKU&t=73s)
Ahern says AI systems search more than people because they can handle repeated searches, inspect several pages, and keep working until they find an answer. A person may enter a few keywords and stop if the result is not on the first page. An agent can search again with a better query and combine what it finds. Exa supplies this search to applications such as Cursor and Gamma. In Cursor, the system can find documentation that the coding agent needs. In Gamma, it can retrieve information such as a company's historical financial returns before generating a slide.

### Search quality, token cost, and response time all affect an agent
[03:56](https://www.youtube.com/watch?v=L4dUFimVDKU&t=236s)
Exa evaluates search with a benchmark that it can update as search changes. Ahern says a fixed public benchmark can become a target for providers and labs, so Exa wants benchmarks that match a customer's use case and stay current. Cost also includes the tokens placed into the model's context, so Exa works to return only the useful content. Latency ranges from sub-300 millisecond searches to longer jobs that can run for 20 minutes and find thousands of entities. The right tradeoff depends on where search sits in the application.

### Historical page versions make past web states available
[06:02](https://www.youtube.com/watch?v=L4dUFimVDKU&t=362s)
Exa stores versions when it discovers pages and when it finds newer versions of existing pages. Ahern says its index covers about 100 billion documents and tracks trillions of URLs. With content versioning, a user gives a URL and a date, and Exa returns the page from that date or the nearest earlier version it has. Version search goes further. It runs a search as of a past date, rather than searching for one known URL. That makes it possible to recreate the information an agent could have found at the time.

### Backtests must block outcome information from entering the past
[09:49](https://www.youtube.com/watch?v=L4dUFimVDKU&t=589s)
A live search always tries to return current information. That creates look-ahead bias when the system is supposed to predict an earlier event. For an acquisition, a current search may simply return that the company was acquired, producing an apparently perfect prediction. Ahern says Snapshot can constrain the search to a date before the announcement. The system then sees material that existed at that point without seeing the confirmation. The same setup can test due diligence, adverse-media checks, business classification, and other decisions that depend on changing web information.

### Historical search can supply features to an existing model
[11:58](https://www.youtube.com/watch?v=L4dUFimVDKU&t=718s)
A traditional model may already exist, but its historical features may depend on web pages that cannot be retrieved as they appeared in the past. Ahern also warns that article publication dates can change when publishers update content or refresh pages for search-engine optimisation. A team can use Snapshot to retrieve relevant URLs at a past date, turn the material into arrays or other structured features, and pass those features into its existing model. In this pattern, Exa does not make the prediction. It supplies time-appropriate data for the backtest.

### Agentic features can be tested one at a time
[13:25](https://www.youtube.com/watch?v=L4dUFimVDKU&t=805s)
Ahern describes a model that adds an agentic web-search feature to a traditional system. The agent could search for adverse media, analyse earnings calls, or inspect CEO interviews, then turn those findings into a structured signal. Each feature can be backtested against a known outcome and tuned separately. Teams can compare different search instructions and see whether one produces more accurate predictions. A more fully agentic system could decide at runtime what to search, combine web results with internal and structured external data, and still be restricted to information available at that time.

### Versioned search can evaluate coding agents without future leakage
[15:41](https://www.youtube.com/watch?v=L4dUFimVDKU&t=941s)
Ahern gives a coding example. To test whether a coding model can solve a pull request, a team can freeze the external repository immediately before the pull request was solved. The system then runs against the earlier state. If it solves the task, the result gives more evidence that the model or another changed component helped. Without the freeze, information added after the solution could leak into the evaluation and make the improvement appear larger than it is.

### Small backtest sets make overfitting easy
[22:01](https://www.youtube.com/watch?v=L4dUFimVDKU&t=1321s)
In the acquisition exercise, participants wrote yes-or-no features and tested them against companies that were acquired and companies that were not. One winning rule effectively excluded several named companies. Another included a particular company name along with other conditions. Ahern says these rules fit the visible set and probably would not work in the future. He regrets removing a holdout set from the exercise because participants could optimise directly against the known examples. He recommends training and iterating on one set, then checking performance on companies kept outside that process.

## Notable quotes
- "AI is actually searching a lot more than humans do nowadays." (01:33)
- "You can now go back in time." (06:20)
- "If you're creating some predictive system, and all it does is goes like, "Okay, I look at the news and it says that this thing was acquired, so tick, it was acquired. 100% successful true positive rate."" (10:23)
- "I do not think that this particular algorithm would be successful in the future, but who knows?" (24:11)
- "You can overfit it, but the idea is like because you have this capability, you can check for that." (24:59)

## Tools & references mentioned
- Exa
- Exa Snapshot
- Cursor
- Gamma
- ChatGPT
- Claude
- OpenAI
- Axios
- The Verge
- Ed Sheeran
- Delivery Hero
- Qiagen
- Vitris
- Bank of Cryptos
- American Electric Power

## Who should watch
- You are building an agent that relies on web search and need to measure whether search actually improves its decisions.
- Your evaluation uses changing web pages, news, documentation, or repositories and could accidentally expose information published after the prediction date.
- You are tuning an agent or model on a small collection of examples and need a practical warning about holdout sets and overfitting.

## Related talks

- [Version-Controlled Search: The Web as It Was](https://aietalks.com/talks/version-controlled-search-the-web-as-it-was) (Ryan Ahern, Exa, 19:13)
- [The Search Engine for the Agentic Web](https://aietalks.com/talks/the-search-engine-for-the-agentic-web) (Will Bryk, Exa, 17:49)
- [How to Test AI Agents When No Benchmark Exists](https://aietalks.com/talks/how-to-test-ai-agents-when-no-benchmark-exists) (Nathanael Weill, Air Canada, 19:50)
- [Your Agent's Biggest Lie: "I Searched the Web"](https://aietalks.com/talks/your-agents-biggest-lie-i-searched-the-web) (Rafael Levi, Bright Data, 15:49)
- [User Signal Dies at the Retrieval Boundary](https://aietalks.com/talks/user-signal-dies-at-the-retrieval-boundary) (Sonam Pankaj, StarlightSearch Inc, 15:37)
