Crawler Index

A large sample of top crawlers that are blocked by websites.

Update on 12-June-24

18.8%

of websites do not block any bots.

18.8%

of websites do not block any bots.

% of websites blocking * % change week over week Company Purpose User Agent
4.84%-0.3%Open AIGPTGPTBot
1.59%0.015%Common CrawlTraining DataCCBot
1.29%-0.2%GoogleBard/Gemini/PaLM/BisonGoogle-Extended
0.95% 0.048%AmazonAlexaamazonbot
0.76%0.092%Meta AILIaMAFacebookBot
0.6%0.006%TurnitinPlagiarism & AI detectorturnitinbot
0.52%0.165%OpenAIChat GPTchatgpt-user
0.29%0.126%BrandwatchMagpie Crawlermagpie-crawler
0.1%1.02%AnthropicClaudeAnthropic-AI
0.03%3.544%CohereCohere CommandCohere-AI
% of websites blocking * % change week over week Company Purpose User Agent
9.37%-0.06%Yandexyandex
8.95%-0.066%Nutchnutch
6.05%-0.054%Majesticmj12bot
4.18%-0.06%Petalpetalbot
1.18%0.007%Baidubaiduspider
0.85%-0.042%Bing bingbot
0.21%-0.014%Yahooslurp
0.06%0.705%Googlegooglebot
0.05%0.91%Ask.comTeoma
0.02%0.144%You.comyoubot
0.01%-0.068%Duck Duck Goduckduckbot
0%-0.068%Internet Archive Wayback Machinearchive.org
0%-0.068%Nutchnutchorg
Methodology

Bright Data scrapes the world’s most sought-after public web data on billions of top websites. Through our compliance product, Bright Shield, we collect allow and disallow commands for user agents in robot.txt from the websites we scrape. Our current sample size of websites is 902,348 and we have collected about 1,700 unique user agents.

Our research team has identified the percentage of time each user agent is blocked and allowed within our sample to determine the most blocked and most allowed user agents. This is how user agents are identified for each chart. Those numbers are tracked week-over-week and the percentage of change is recorded. We also track the overall percentage of websites that allow all crawlers. Each user agent is identified to the best of our ability by company, use, and a link that includes additional information such as how to block it.

Comments on user agents? Email comments to [email protected]