6.7%
Blocks an AI crawler
1,086 / 16,217
46%
Blocks GPTBot
7,466 / 16,217
5.4%
Blocks CCBot
874 / 16,217
45%
Blocks Google-Extended
7,326 / 16,217
AI crawlers blocked in robots.txt
| Crawler | Operator | Global | Netherlands | United Kingdom | Italy | Spain | Japan | Latin America |
|---|---|---|---|---|---|---|---|---|
| GPTBot | OpenAI (training) | 11% | 46% | 53% | 9.7% | 10% | 4.4% | 8.4% |
| ChatGPT-User | OpenAI (user fetch) | 4.9% | 43% | 49% | 3.9% | 5.3% | 2.1% | 2.4% |
| OAI-SearchBot | OpenAI (search) | 3.6% | 42% | 49% | 3.4% | 4.5% | 2% | 1.8% |
| ClaudeBot | Anthropic | 11% | 46% | 53% | 9.4% | 9.6% | 4% | 8.2% |
| anthropic-ai | Anthropic (legacy) | 9.6% | 45% | 52% | 8.2% | 8% | 3% | 7.4% |
| Google-Extended | Google (Gemini) | 9.7% | 45% | 52% | 8.6% | 7.5% | 3.3% | 7.5% |
| CCBot | Common Crawl | 11% | 5.4% | 5.8% | 9.6% | 8.5% | 3.7% | 8.3% |
| PerplexityBot | Perplexity | 4.7% | 42% | 49% | 4% | 4.9% | 2.1% | 2.2% |
| Bytespider | ByteDance | 11% | 5.1% | 5.5% | 8.4% | 8.4% | 4% | 8.7% |
| Applebot-Extended | Apple | 9.6% | 4.1% | 4.9% | 7.3% | 6.4% | 3.4% | 7.4% |
| Amazonbot | Amazon | 9.9% | 46% | 52% | 9% | 9.3% | 3.6% | 8.1% |
| meta-externalagent | Meta | 9.8% | 45% | 52% | 8.9% | 9.1% | 3.9% | 7.7% |
| cohere-ai | Cohere | 9.3% | 45% | 52% | 8% | 7.6% | 3% | 7.4% |
| Diffbot | Diffbot | 9.2% | 45% | 52% | 8% | 7.7% | 2.8% | 7.5% |
Share of sites whose robots.txt blocks the bot from the homepage by name.
By market
| Metric | Global | Netherlands | United Kingdom | Italy | Spain | Japan | Latin America |
|---|---|---|---|---|---|---|---|
| Blocks an AI crawler | 12% | 6.7% | 6.5% | 11% | 11% | 4.5% | 8.9% |
| Blocks GPTBot | 11% | 46% | 53% | 9.7% | 10% | 4.4% | 8.4% |
| Blocks ClaudeBot | 11% | 46% | 53% | 9.4% | 9.6% | 4% | 8.2% |
| Blocks CCBot | 11% | 5.4% | 5.8% | 9.6% | 8.5% | 3.7% | 8.3% |
| Blocks Google-Extended | 9.7% | 45% | 52% | 8.6% | 7.5% | 3.3% | 7.5% |
| Blocks PerplexityBot | 4.7% | 42% | 49% | 4% | 4.9% | 2.1% | 2.2% |
| Content-Signal in robots.txt | 6.8% | 3.1% | 3.7% | 5.5% | 4.2% | 1.2% | 6.7% |
| llms.txt | 15% | 11% | 13% | 12% | 13% | 5.9% | 18% |
Search bots blocked
- 1Baiduspider58%7,385
- 2SeznamBot54%6,771
- 3Yeti53%6,754
- 4Naverbot53%6,743
- 5Sogou53%6,738
- 6DuckDuckBot53%6,712
- 7PetalBot5.8%738
- 8YandexBot3.2%399
- 9Applebot1.3%169
- 10Bingbot0.8%104
Share of 12,639 sites in the sample.
SEO bots blocked
- 1BLEXBot55%6,969
- 2Barkrowler54%6,847
- 3Screaming Frog SEO Spider54%6,762
- 4SiteAuditBot53%6,759
- 5MJ12bot4.6%587
- 6SemrushBot4.1%512
- 7AhrefsBot3.9%488
- 8DotBot3.4%426
- 9DataForSeoBot2%254
- 10rogerbot1.4%181
Share of 12,639 sites in the sample.
Our crawler: 8,376 sites · Netherlands · updated 13 Nov 2026 · Methodology
Lots of domains, little time?
We review your whole domain portfolio and fix DNS and email security.