# How to Defend Your Sites from AI Bots

David Mytton, Arcjet | AI Engineer World's Fair 2025 | 20:12

Source: https://www.youtube.com/watch?v=Gi4V8viBGYQ
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-to-defend-your-sites-from-ai-bots
Published: 2025-07-30
Tags: privacy, security

## TL;DR
- Automated clients can create infrastructure, bandwidth, and availability problems, while AI crawlers add new kinds of traffic that site owners may or may not want.
- User agents and robots.txt are useful starting points, but they are voluntary or easy to spoof, so important decisions need IP verification and additional signals.
- Fingerprinting, rate limits, proof of work, and request signatures can raise the cost of unwanted automation, but each approach has limits around accessibility, incentives, and adoption.

## Summary
David Mytton explains why site owners need to distinguish between search crawlers, AI training crawlers, real-time assistants, and agents acting for users. Automated traffic can increase database and serverless costs, consume bandwidth, and make services unavailable. Robots.txt and user-agent strings help with cooperative crawlers, while reverse DNS checks can verify services such as Google, Bing, Apple, and OpenAI. For less cooperative clients, Mytton recommends combining IP reputation, network and location data, proof-of-work challenges, HTTP request signatures, TLS and HTTP fingerprints, and rate limits tied to sessions or fingerprints rather than IP addresses alone. He is clear about the tradeoffs. CAPTCHAs are increasingly easy for AI systems to solve, residential proxies weaken IP data, proof of work may not stop profitable attacks, and harder challenges can create accessibility problems. The talk gives a layered approach that developers can implement themselves, starting with simple signals and adding controls where the site or resource warrants them.

## Key ideas
### Automated clients can turn ordinary requests into infrastructure and availability costs
[00:35](https://www.youtube.com/watch?v=Gi4V8viBGYQ&t=35s)
Mytton says almost 50% of web traffic is automated, with gaming approaching 60%. Requests become expensive when a site reads from a database or generates dynamic content, especially on serverless platforms that charge per request. Automated clients can also download large assets, consuming bandwidth and resources that legitimate users need. In the worst case, the traffic looks like a denial-of-service attack because the service becomes unavailable. The problem is not limited to small sites, since even large websites do not have infinite capacity.

### AI traffic includes crawlers with different effects on a site owner
[02:36](https://www.youtube.com/watch?v=Gi4V8viBGYQ&t=156s)
Mytton separates AI crawlers into several categories. OpenAI's search crawler can index a site and bring citations and traffic, much like Googlebot. ChatGPT user requests may fetch a URL because a real user asked for a summary or implementation help. GPTBot collects content for training and may provide no direct traffic or citation. Newer computer-use agents act through browser-like clients and can perform actions for users. An agent checking an inbox may be acceptable, while one buying 500 concert tickets for resale is likely unwanted. The site owner has to decide which behavior to allow.

### Robots.txt and user-agent strings help with cooperative clients but cannot prove identity
[07:27](https://www.youtube.com/watch?v=Gi4V8viBGYQ&t=447s)
Robots.txt lets an owner describe which crawlers may access parts of a site, so it is a useful place to define policy. It remains voluntary, and malicious crawlers may ignore it or use disallowed paths as a map of interesting content. Every HTTP request also includes a user-agent string, and open-source libraries can classify thousands of known agents. The problem is that any client can set this string. Bad bots can claim to be Google or Chrome, so the value of the signal depends on whether it can be checked.

### Reverse DNS and IP reputation add evidence about who is connecting
[09:24](https://www.youtube.com/watch?v=Gi4V8viBGYQ&t=564s)
When a request claims to come from Apple, Bing, Google, or OpenAI, the owner can verify its source IP through reverse DNS with that service. IP data can also show network ranges, operators, data centers, countries, VPNs, proxies, and residential or mobile connections. Mytton cites AWS as the source of 12% of bot traffic on the Cloudflare network last year. That may be acceptable for a crawler, but it is suspicious for a signup form intended for humans. He cautions that geolocation is inaccurate and residential proxies can make automated traffic look like it comes from a home connection.

### CAPTCHAs fail when solving them is cheap for AI systems
[12:56](https://www.youtube.com/watch?v=Gi4V8viBGYQ&t=776s)
Mytton says image and audio CAPTCHAs are increasingly easy for AI systems to solve. An audio challenge can be downloaded and transcribed within seconds, making this defense cheap to bypass. Proof of work changes the economics by requiring a client to spend CPU time before receiving a resource. A single request may take only a second or two on a laptop or phone, while a crawler visiting millions of sites pays that cost repeatedly. It still may not stop a profitable ticket attack if the attacker can spend a few dollars and resell a ticket for much more.

### Challenges should be applied selectively because difficulty can hurt accessibility
[14:46](https://www.youtube.com/watch?v=Gi4V8viBGYQ&t=886s)
A site can combine signals and give a harder challenge to a client with an unverified IP address or other suspicious characteristics. Mytton warns that this creates accessibility problems, since people already encounter CAPTCHAs they cannot solve and must refresh repeatedly. He mentions Anubis, GoAway, and Nepenthes as open-source proxies that can run in front of an application or Kubernetes cluster and apply proof-of-work challenges to clients considered suspicious.

### Request signatures and privacy tokens offer ways to verify clients
[15:43](https://www.youtube.com/watch?v=Gi4V8viBGYQ&t=943s)
Cloudflare has proposed HTTP message signatures for automated clients. Each request would carry a cryptographic signature that the site could verify quickly. Mytton says the proposal was still being developed and questioned whether it would improve on IP verification. Apple Private Access Tokens, previously called Privacy Pass, let sites verify that a request came from a browser associated with an iCloud subscriber. The mechanism is built into Apple devices and can reduce CAPTCHA use, but adoption outside Apple's ecosystem has been limited.

### Fingerprints and session-based limits handle rotating IP addresses
[16:43](https://www.youtube.com/watch?v=Gi4V8viBGYQ&t=1003s)
Rotating IPs make IP-only rules weak, especially when crawlers use large IPv6 pools. Mytton recommends generating a fingerprint from stable client characteristics. The open-source JA4 hash uses the TLS fingerprint and SSL configuration, while proprietary HTTP fingerprints use headers and other request characteristics. A site can block a fingerprint across many IP addresses and then apply quotas to it. Rate limits should use a logged-in session ID or a fingerprint where possible, since malicious clients can change IP addresses themselves.

### A layered defense starts simply and becomes stricter for valuable targets
[18:26](https://www.youtube.com/watch?v=Gi4V8viBGYQ&t=1106s)
Mytton describes eight defenses as a progression. Robots.txt is a starting point for cooperative crawlers, not a final barrier. User-agent analysis and verification are enough for many sites. Sites with popular resources, restricted inventory, or items people may try to buy in quantity need more controls. Those controls can include IP reputation, proof of work, experimental HTTP signatures, fingerprints, and rate limits. The components can be implemented directly in application code, and Mytton points to Arcjet's detailed blog post for examples.

## Notable quotes
- "Almost 50% of web traffic today is automated clients." (00:35)
- "It's just a string in a HTTP header and you can set it to whatever you want." (08:44)
- "CAPTCHAs are becoming increasingly easy for AI to solve." (12:56)
- "Robots.txt is where you start. It's not where you finish though because it's not going to prevent all the bots." (18:50)
- "You can't just rate limit on an IP address because people have different IPs." (17:50)

## Tools & references mentioned
- Arcjet
- Googlebot
- OpenAI
- GPTBot
- ChatGPT
- Operator
- robots.txt
- MaxMind
- IPinfo
- Cloudflare
- Anubis
- GoAway
- Nepenthes
- JA4
- Apple Private Access Tokens
- Privacy Pass

## Who should watch
- You run a dynamic or serverless site and automated requests are increasing database, compute, or bandwidth costs.
- Your service has scarce resources, such as limited-inventory products, and you need to control agents acting through browser-like clients.
- You are replacing CAPTCHAs and want a layered design based on verification, fingerprints, reputation, and rate limits.

## Related talks

- [Your Agent's Biggest Lie: "I Searched the Web"](https://aietalks.com/talks/your-agents-biggest-lie-i-searched-the-web) (Rafael Levi, Bright Data, 15:49)
- [The Missing Layer in Agentic AI](https://aietalks.com/talks/the-missing-layer-in-agentic-ai) (Giedrius Šteimantas, Oxylabs, 15:04)
- [The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans](https://aietalks.com/talks/the-dark-arts-of-web-automation-teaching-agents-to-use-websites-like-humans) (Corey Gallon, Rexmore, 21:38)
- [AI Didn't Kill the Web, It Moved in!](https://aietalks.com/talks/ai-didnt-kill-the-web-it-moved-in) (Yohan Lasorsa, Microsoft & Olivier Leplus, AWS, 52:55)
- [Securing Code-Executing AI Agents](https://aietalks.com/talks/securing-code-executing-ai-agents) (Fouad Matin, OpenAI, 14:00)