Never Get Blocked Again

From eCommerce to public data monitoring, whatever your industry, you may encounter rate limitations on IPs, detection of User-Agents or IP geolocation-based blockages. Whatever the reason is, Web Unlocker offers a better, fully automated unblocking solution with a 100% success rate guaranteed
Aviv Besinksky
Aviv Besinsky | Product Manager
10-Nov-2021

In this article we will discuss:

Most common website blocks

Here are a few ‘popular’ ways in which target sites block companies from monitoring, and collecting open-source data from their sites.

Site block #1: Rate limitations on IPs 

When you send multiple requests from the same IP address to a target site, this can be categorized as suspicious activity. Whatever limit has been set by that site (could be 15 requests or 1,500), once it is passed your data collection operations will be interrupted by CAPTCHAs or error messages requiring manual/human intervention. 

Site block #2: Detection of User-Agents 

Websites use different headers, for example a site may utilize an ‘HTTP’ header in order to discover third-parties and block or limit access. 

Site block #3: IP geolocation detection 

Some sites will actually block you based on your current geolocation. This is typically due to their desire to customize content based on location, a government restriction on certain information or a content licencing restriction based on deals signed with local tv channels, and the like.

Use cases that can benefit most from web unblocking

There is a wide variety of use cases that can meaningfully benefit from automated, web unblocking technology. I will review two specific examples to give you some context so that you can also extrapolate how this can be beneficial to your business:

  1. Ecommerce and Retail: When it comes to companies that are involved in digital commerce (front or back-end) it is of the utmost importance to be able to react in real-time to changes happening in the competitive space which your business is operating. It is also very important to be proactive. But the ability for you and/or your algorithms to make live changes in order to drive sales is oftentimes reliant on being able to obtain hard-to-reach datasets that competing entities work hard to make difficult to access, and accurately collect.

For example, you may want to implement a real-time dynamic pricing strategy based on how your competitors are pricing items but can’t obtain this open-source data or maybe you are being fed different pricing than consumers. 

  1. Public data monitoring: This scenario can replicate itself in a similar way when trying to obtain online financial/people data, or when trying to access government/public databases. The information on these types of target sites is open-source, to be sure but still many companies work overtime to create complex/dynamic site architectures to keep competing entities/and or malicious actors from snooping around or meddling. 

Fast forward to ‘Web Unlocker’, the ultimate unblocking solution 

Bright Data’s Web Unlocker tool automates the entire unblocking process for you with a near-100% success while paying only for successful requests. Web Unlocker’s main capabilities include:

  • Network and IP: The ability to route requests automatically through the correct network and IP
  • Headers: Automatically set User-Agent, and other headers based on target site requirements
  • Protocols: Seamlessly upgrade HTTP protocols and rotate TLS/SSL fingerprinting
  • Cookies: Automate your IP priming, and cookie management
  • Detection, and Matching: Enjoy intelligent detection of blocked requests based on response codes, content, timing, and other unique capabilities
  • Retries: Automatically retries failed requests
Web Unlocker Capabilities and unblocking the sites you need. Network and IP - route request automatically the the correct network IP, Headers - Automatically set User-Agents and other headers based on target site requirements. Protocols - Seamlessly upgrades HTTP protocols and rotates TLS/SSL fingerprint. Cookies - Automatic IP priming and Cookie Management. Detection and Matching - Intelligent detection of blocked requests based on response codes, content, timing and more. Retries - Automatically retries failed requests.

How does it work? 

The process of working with Web Unlocker is simple, and fast. The heavy lifting is done by Web Unlocker, and doesn’t require any effort on your part. This is the three-step process:

  1. You decide which target sites you need to unlock.
  2. Then send a request. Here is a quick example: ‘curl -k -proxy lum-customer-<id>-zone:<password>@zproxy.lum-superproxy.io:22225 https://example.com
  3. Get responses in HTML format for easy integration with any system
Ho Do It Work? You Decide which target sites you need to unlock, send a request using the CURL -k command, get responses in HTML format for easy integration with any system

Web Unlocker Vs. Self-serve Proxies 

Web Unlocker and self-serve proxies have in common the fact that they both go through Super Proxies, and can be managed using the Proxy Manager. They also both have the potential for global GEO coverage. But that’s where the similarities end. In addition to all this, Web Unlocker brings the following to the table:

  • CAPTCHA-solving 
  • Handling target site markup changes
  • Asynchronous requests for chosen domains
Web Unlocker VS Self-serve proxies surpasses in automated unblocking, captcha solving, handling target sites markup changes and asynchronous requests for choses domains

The bottom line 

Getting blocked is never conducive to healthy business operations; it sets you back, slows things down, and makes your entire business strategy clunky. There is a better way!

Aviv Besinksky
Aviv Besinsky | Product Manager

Aviv is a lead product manager at Bright Data. He has been a driving force in taking data collection technology to the next level - developing technological solutions in the realms of data unblocking, static proxy networks, and more. Sharing his data crawling know-how is one of his many passions.