Crawler Index
A large sample of top crawlers that are blocked by websites.
Update on 4-September-24
23.8%
of websites do not block any bots.
23.8%
of websites do not block any bots.
| % of websites blocking all (*) and explicitly blocking user agent | % of websites explicitly blocking user agent | Company | Purpose | User Agent |
|---|---|---|---|---|
| 5.9% | Open AI | GPT | GPTBot | |
| 5.28% | 0.23% | Common Crawl | Training Data | CCBot |
| 4.73% | 0.42% | Bard/Gemini/PaLM/Bison | Google-Extended | |
| 4.61% | 0.16% | OpenAI | Chat GPT | chatgpt-user |
| 4.54% | 0.39% | Amazon | Alexa | amazonbot |
| 4.26% | 0.74% | Meta AI | LIaMA | FacebookBot |
| 4.21% | 1.25% | Brandwatch | Magpie Crawler | magpie-crawler |
| 4.21% | 1.61% | ByteDance | ByteDance LLM N/A | Bytespider |
| 4.18% | 3.29% | Anthropic | Claude | Anthropic-AI |
| 4.13% | 3.42% | Anthropic | Claude | claudebot |
| 4.08% | 3.57% | Anthropic | Claude | claude-web |
| 4.04% | 5.42% | Perplexity | Chatbot | perplexitybot |
| 3.98% | 4.191% | Cohere | Cohere Command | Cohere-AI |
| 3.97% | 5.39% | Apple | Apple's foundational models | Applebot-Extended |
| 3.75% | 1.47% | Apple | Siri | Applebot |
Bright Data scrapes the world’s most sought-after public web data on billions of top websites. Through our compliance product, Bright Shield, we collect allow and disallow commands for user agents in robot.txt from the websites we scrape. Our current sample size of websites is 1,227,821 and we have collected about 1,700 unique user agents.
Our research team has identified the percentage of time each user agent is explicitly blocked within our sample and each user agent that is explicitly blocked combined with the block all (*) command to determine the most blocked user agents. We also track the overall percentage of websites that allow all crawlers. Each user agent is identified to the best of our ability by company, use, and a link that includes additional information such as how to block it.
Comments on user agents? Email comments to [email protected]