In this guide, you will learn:
- What Cloudflare is
- Why its WAF solution poses a challenge for your scraping scripts
- How to bypass Cloudflare WAF using all-in-one solutions
- How to tackle each of the main anti-bot measures it relies on
Let’s dive in!
What Is Cloudflare?
Cloudflare is a web infrastructure and security company that operates one of the largest networks on the Web. It offers a comprehensive suite of services designed to make websites faster and more secure.
At its core, Cloudflare functions primarily as a CDN (Content Delivery Network), caching site content on a global network to improve load times and reduce latency. Additionally, it provides features like DDoS (Distributed Denial-of-Service) protection, a WAF (Web Application Firewall), bot management, DNS services, and more.
By integrating with Cloudflare’s network, sites can quickly gain enhanced security and optimized performance. This has made Cloudflare the go-to solution for millions of websites worldwide.
Cloudflare WAF in a Nutshell
A WAF, short for Web Application Firewall, is a security system that filters and monitors HTTP traffic between a web application and the Internet. It helps protect websites from attacks like DDoS, cross-site scripting (XSS), SQL injection, and other malicious activity.
In particular, Cloudflare WAF is one of the most widely used WAF solutions in the world. Its popularity is due to the widespread adoption of Cloudflare as a CDN. For websites already on Cloudflare, enabling the WAF with default configurations requires just a few clicks.
The key anti-bot technologies and techniques implemented by Cloudflare WAF include:
- Rate limiting: Limit the number of requests a single IP can make in a given timeframe to stop DDoS attacks and prevent brute-force attempts.
- JavaScript challenges: Verify if the visitor can execute JavaScript, which is a typical behavior for real users.
- Turnstile CAPTCHA: Present CAPTCHA tests to suspected bots.
- IP reputation: Maintain a reputation database to block suspicious IP addresses immediately.
- Behavior analysis: Monitor visitor behavior to detect automated patterns or abnormal activity.
When a site is protected by Cloudflare WAF, it typically employs one or more anti-bot solutions to block automated requests. The combination of those defenses is what makes scraping a Cloudflare-protected site particularly challenging.
First Solutions to Avoid Cloudflare Blocks When Scraping a Site
Discover the best solutions and ideas for a first approach to web scraping on Cloudflare-protected sites.
Bypass Cloudflare Entirely
Do not forget that Cloudflare acts as a CDN, which means it caches and distributes site content across multiple geographically dispersed servers. So, sites distributed via Cloudflare are typically only accessible through servers in the CDN network.
Now, imagine if you managed to discover the IP address of the site server behind the CDN. The consequence would be that you could interact with the site while bypassing Cloudflare entirely. After all, Cloudflare can only evaluate requests that pass through its network.
That is possible by looking at DNS history lookup tools like SecurityTrails to identify any historical DNS records that reveal the original server’s IP address. Once you obtain the IP, you can attempt to send requests directly to the server, eluding Cloudflare.
The problem is that the server may have additional configurations in place to accept requests only from Cloudflare’s IP range. That would make it nearly impossible to connect to the site directly without being blocked. Additionally, successfully finding the original server IP is quite difficult and unlikely.
Free Cloudflare Solvers
Online, you can find several free and open-source libraries designed to bypass Cloudflare. Some of the most popular ones include:
- cloudscraper: A Python module that handles Cloudflare’s anti-bot challenges.
- Cfscrape: A lightweight PHP module to bypass Cloudflare’s anti-bot pages.
- Humanoid: A Node.js package to bypass Cloudflare’s anti-bot JavaScript challenges.
While these solutions may temporarily work, remember that anti-scraping is a cat-and-mouse game. What works today might not work tomorrow, as Cloudflare continuously updates its protection mechanisms.
Not surprisingly, most of these projects have not received updates in years. The reason is that developers gave up due to the ongoing struggle to keep up with Cloudflare’s updates.
Premium Cloudflare Solvers
In most cases, the best solution for scraping a Cloudflare-protected site is to use a premium product. The cost ensures regular updates from experts in the scraping field, maintaining high reliability against Cloudflare’s defenses.
On top of that, top-notch providers like Bright Data also offer 24/7 technical support to help resolve any issues. If you are looking for a professional Cloudflare scraping solution, try our Scraping Browser.
As a cloud-based, scalable, GUI browser, it integrates with Playwright, Puppeteer, Selenium, and any other headless browser libraries. To guarantee high effectiveness against Cloudflare, it includes features like IP rotation, CAPTCHA-solving capabilities, User-Agent rotation, and more.
Scraping a Cloudflare-Protected Site: DIY Approach to Bypassing Anti-Bots
Cracking Cloudflare is difficult, especially if you do not want to use a premium all-in-one solution. If that is the path you want to follow, you must take into account all of Cloudflare’s defenses against bots and find ways to overcome them.
In this section, you will see some of the most useful high-level techniques for eluding Cloudflare and scraping sites protected by its WAF. For detailed instructions, check out our guide on how to bypass Cloudflare.
Let’s begin!
JavaScript Rendering
One of the most common techniques Cloudflare uses to detect bots is JavaScript challenges. These are JavaScript scripts embedded in web pages that are executed during rendering time by the browser. They perform specific checks to determine the likelihood of the visitor being a bot:
If Cloudflare suspects you are a bot based on the result of those challenges, it will show you a CAPTCHA. Otherwise, you will be allowed to access the page content.
Thus, to target a page protected by Cloudflare, you need to use a browser automation tool like Playwright, Selenium, or Puppeteer. Those tools enable you to instruct a browser to interact with web pages like regular users. Learn more in our guide on web scraping with Playwright.
The issue is that headless browsers have default configurations that can expose them to anti-bot detection systems. To avoid this, you should use libraries like Playwright Stealth or Puppeteer Stealth through Puppeteer Extra, which help mask headless browser activity.
CAPTCHA Solving
If Cloudflare thinks that you might be a bot, it will try to stop you with a Turnstile CAPTCHA:
Depending on the configuration, the CAPTCHA might be a simple click-based test as above or a more complex puzzle as below:
Automating CAPTCHA resolution is complex, as CAPTCHAs are tests specifically designed to differentiate between bots and humans. If your headless browser encounters such a challenge, you can try the techniques outlined in our guide on bypassing CAPTCHAs with Python.
For a more reliable solution that works regardless of the technology you are using in your scraping script, consider Bright Data’s Cloudflare Turnstile Solver. This quickly and automatically resolves Cloudflare Turnstile CAPTCHAs for you.
Rate Limiter Bypass
If you make too many requests from the same IP in a short time, Cloudflare is likely to temporarily or even permanently ban your IP. This is problematic as it halts your scraping operation and damages your IP’s reputation.
The technique described above, used to stop DDoS attacks and unwanted automated requests, is called rate limiting. Since your IP is tied to the network you are connected to, you cannot easily change it. The only effective way to implement IP rotation and avoid bans is to use a proxy service.
With solutions like residential proxies, you can make your script’s requests appear as if they are coming from real-world devices in a specific location. Find out more about our residential proxy offerings.
Browser Spoofing
Browsers, even in headless mode, consume a lot of resources. So, building a scraping operation around a Cloudflare-protected website using a browser automation tool can result in a resource-intensive process. That may potentially require multiple servers and a complex architecture.
To avoid that hassle—and in cases where Cloudflare’s WAF has been configured not to be overly aggressive—you can try a different approach. The idea is to make automated requests from HTTP clients that mimic real browsers, which is known as browser spoofing.
The goal is to make your HTTP requests look as close as possible to those from a regular browser. You can achieve the result by setting specific HTTP headers like User-Agent. For more information, follow our guide on the best User-Agent for web scraping.
In more complex scenarios, that trick alone may not be enough. Cloudflare can still detect your requests as coming from an HTTP client rather than a browser due to the TLS fingerprint:
If you are not familiar with that concept, TLS fingerprinting involves identifying a client based on the way it establishes secure connections over TLS. To replicate a browser’s TLS fingerprint, you can use an HTTP client like curl-impersonate, as explained in our dedicated tutorial.
Conclusion
In this article, you saw several tips and tricks to scrape Cloudflare-protected sites. Cloudflare is the most popular CDN service on the market, and it also offers advanced anti-bot solutions. As learned here, bypassing Cloudflare’s anti-scraping measures is challenging but not impossible.
Regardless of the approach you choose, remember that everything becomes easier with professional, fast, and reliable scraping solutions, such as:
- Web Unlocker: Autonomously bypass rate limiting, fingerprinting, and other anti-bot restrictions, enabling seamless public web data collection.
- Scraping Browser: A fully hosted browser that allows you to scrape dynamic web data while automating the process of unblocking websites.
With Bright Data’s extensive suite of scraping tools, extracting data from Cloudflare-protected sites has never been easier!
Sign up now to find out which of Bright Data’s solutions best suits your needs. Start with a free trial today!
No credit card required
