How Web Unlocker is enabling better fingerprinting, auto-unlocking, and CAPTCHA-solving

From customized Transport Layer Security (TLS) handshakes at the Network level, and User-agent generation at the Protocol level to complete cookie management, and browser fingerprint emulation at the browser-level, ‘Web Unlocker’ takes ‘Unblocking’ to the next level
Web Unlocker Used To Be Called Unblocker copy
Aviv Besinksky
Aviv Besinsky | Product Manager

Web Unlocker was formerly called Web Unblocker – for businesses out there wondering what improvements were made to the product in the process, this article will focus on Web Unlocker’s: 

Better auto-unlocking capabilities 

Web Unlocker’s environment emulation capabilities include: 

  • When examining Web Unlocker from a Network perspective it is able to successfully handle everything from IP type selection to IP rotations (when necessary).
  • On the Protocol-level Web Unlocker is able to carry out HTTP header management more effectively both during the decoding (i.e request received) and encoding (i.e. response sent) processes. Additionally, Web Unlocker has better ‘User-agents generation’ capabilities, meaning that it can deal with unique browser-fingerprint properties while enabling you to restrict the generated User-agents to match target site requirements. Lastly, Web Unlocker can now support HTTP2, meaning it can more effectively handle HTTP header field compression as well as server push. 
  • As far as Hardware (HW), and Operating Systems (OS) are concerned Web Unlocker is able to emulate all devices attached to any given system as well as their correlating drivers (i.e. full device enumeration imitation), mouse movements, screen resolution and various device properties.

Increased CAPTCHA-recognition, and solving 

Web Unlocker has better request management capabilities. It can now adapt itself in real-time to new blockades that have been set up by target sites. It uses Machine Learning (ML), improved CAPTCHA-resolving, and retry logic so that the most efficient (both in terms of time and resources) data-retrieval path can be attained. 

The algorithm that powers Web Unlocker is trained to use ideal fingerprint configurations and attempt retrieving desired responses/information from target sites. When the website responds, for example by:

  • Categorizing activity as ‘suspicious’ based on the number of requests from a specific IP (i.e. ‘rate limitations’) 
  • Detecting ‘unacceptable’ User-Agents (such as ‘HTTP’)
  • Or blocking requests based on geolocation 

Web Unlocker is able to analyze this information in real-time, and recalibrate settings on a per-domain basis, using specific/customized settings that will provide users with the highest success rates. 

Improved fingerprinting 

‘Device fingerprints’  are unique user identifiers for devices browsing the web. A fingerprint contains information regarding the configuration of a user’s browser as well as software/hardware environment. Having the fingerprinting aspect down pat is probably the most important aspect of unblocking as it solves the root of the issue and not just a derivative ‘symptom’ such as a ‘Recaptcha’. 

In this instance, you can ‘easily’, albeit manually change your IP, and start a new session. Or alternatively, trigger the ‘HTML body element’ rule. 

Web Unlocker completely automates all of this including fingerprinting adaptation to target sites. At the browser level it performs browser fingerprint emulation including:

  • Plugins, and fonts
  • Canvas/webGL fingerprints
  • Headers
  • Zombie, and sync cookies
  • WebRTC
  • Web Audio API fingerprints
  • cookie management

Web Unlocker also handles:

  • Disabling target-device fingerprinting
  • Real-time debugging

The bottom line 

‘Unblocker’ has been renamed and is now called ‘Web Unlocker’, utilizing the same technology, and algorithmic approach. Yes. Web Unlocker is the same product as ‘Web Unblocker’ but it has been upgraded as far as the resolution, and span of its environment emulation goes on all levels – Network, Protocol, Hardware/Operating systems as well as at the browser level. Its retry logic, CAPTCHA-resolving, and fingerprinting capabilities have also been taken one step further, putting it in a league of its own. 

Aviv Besinksky
Aviv Besinsky | Product Manager

Aviv is a lead product manager at Bright Data. He has been a driving force in taking data collection technology to the next level - developing technological solutions in the realms of data unblocking, static proxy networks, and more. Sharing his data crawling know-how is one of his many passions.


You might also be interested in

If your company has even ONE developer dedicated to web data collection, you are wasting precious resources

The state of the economy in general, and of tech in particular, is leading many CEOs to put budget cut pressure on Information Technology execs. This article aims to help IT leaders improve their bottom lines by offering a more strategic approach to operational web data collection outsourcing

Shooting ourselves in the foot? Why we willingly killed 10% of our network

Bright Data believes in transparent and ethical practices, especially when it comes to dealing with users who make up its Residential peer network. To ensure compliance, we use advanced monitoring protocols and partner with top anti-virus companies. Sometimes, we make decisions which might seem a little crazy, like hurting our own network. That is what this post is about.
Web Data powering e-commerce

Mystery shoppers are so 2000 and late. Web data is the future of e-commerce.

We sat down with Charmagne Cruz from Shopee, the leading e-commerce platform in Southeast Asia, to discuss how the online conglomerate uses public web data to drive forward the company’s success as well as carve out a large section of the Asian e-commerce market.
Qualitative data collection methods

Qualitative data collection methods

Quantitative pertains to numbers such as competitor product fluctuations, while qualitative pertains to the ‘narrative’ such as audience social sentiment regarding a particular brand. This article explains all the key differences between the two, as well as offering tools to quickly and easily obtain target data points