Crawler Index

A large sample of crawlers that are blocked by websites.

72.3%

of websites have at least a partial disallow command.

72.3%

of websites have at least a partial disallow command.

% of websites explicitly blocking user agent % of websites blocking explicitly and with * command Company Purpose User Agent
2.0%59.3%Open AIGPTGPTBot
1.3%59.2%Common CrawlTraining DataCCBot
0.8%59.2%GoogleBard/Gemini/PaLM/BisonGoogle-Extended
0.7%59.2%OpenAIChat GPTchatgpt-user
0.8%62.7%AmazonAlexaamazonbot
0.5%60.0%Meta AILIaMAFacebookBot
0.5%62.8%BrandwatchMagpie Crawlermagpie-crawler
1.0%62.7%ByteDanceByteDance LLM N/ABytespider
0.5%59.2%AnthropicClaudeAnthropic-AI
0.6%62.6%AnthropicClaudeclaudebot
0.4%60.1%AnthropicClaudeclaude-web
0.4%60.1%PerplexityChatbotperplexitybot
0.3%60.1%CohereCohere CommandCohere-AI
0.3%60.2%AppleApple's foundational modelsApplebot-Extended
0.3%62.6%AppleSiriApplebot
0.3%62.6%Diffbottraining datadiffbot
0.4%62.6%MetaAll Meta AImeta-externalagent
0.2%62.6%OpenAISearchGPToai-searchbot
0.2%62.6%TimpiWilson AItimpibot
0.1%62.6%webz.iowebzio-extendedwebzio-extended
0.1%62.6%GoogleBard/Gemini/PaLM/Bisongoogleother
0.01%62.8%Perplexityperplexity-aiperplexity-ai
0.03%62.6%MetaAll Meta AImeta-externalfetcher
% of websites explicitly blocking user agent % of websites blocking explicitly and with * command Company Purpose User Agent
23.9%85.8%Open AIChat GPTgptbot
17.4%85.1%Common CrawlTraining Dataccbot
13.9%85.6%GoogleBard/Gemini/PaLM/Bisongoogle-extended
12.1%84.8%Open AIChat GPTchatgpt-user
11.8%84.6%AnthropicClaudeanthropic-ai
12.5%84.4%AnthropicClaudeclaudebot
9.3%84.2%AnthropicClaudeclaude-web
8.3%84.1%MetaLIaMAfacebookbot
9.8%84.6%ByteDanceByteDance LLM N/Abytespider
10.0%84.4%PerplexityChatbotperplexitybot
8.6%84.7%CohereCohere Commandcohere-ai
8.2%84.9%Apple.comApple's foundational modelsapplebot-extended
4.4%85.6%Brand WatchMagpie Crawlermagpie-crawler
5.6%84.6%AmazonAlexaamazonbot
3.1%84.8%AppleSiriapplebot
0.9%85.3%GoogleBard/Gemini/PaLM/Bisongoogleother
1.2%85.1%Webzwebzio-extendedwebzio-extended
2.2%85.2%TimpiWilson AItimpibot
1.4%85.7%Perplexityperplexity-aiperplexity-ai
2.9%85.6%MetaAll Meta AImeta-externalfetcher
4.4%84.4%Open AISearch GPTOAI-searchbot
6.2%84.8%MetaAll Meta AImeta-externalagent
Methodology

Bright Data scrapes the world’s most sought-after public web data on billions of top websites. Through our compliance product, Bright Shield, we collect allow and disallow commands for user agents in robot.txt from the websites we scrape. Our current sample size of websites is 2,971,145 and we have collected about 1,700 unique user agents.

Our research team has identified the percentage of time each user agent of interest is explicitly blocked within our sample and each user agent that is blocked with the (*) command. We also track the overall percentage of websites that disallow all crawlers. Each user agent is identified to the best of our ability by company, use, and a link that includes additional information such as how to block it.

Comments on user agents? Email comments to media@brightdata.com