How Web Unlocker is enabling better fingerprinting, auto-unlocking, and CAPTCHA-solving

From customized Transport Layer Security (TLS) handshakes at the Network level, and User-agent generation at the Protocol level to complete cookie management, and browser fingerprint emulation at the browser-level, ‘Web Unlocker’ takes ‘Unblocking’ to the next level
Aviv Besinksky
Aviv Besinsky | Product Manager
01-Dec-2021

Web Unlocker was formerly called Web Unblocker – for businesses out there wondering what improvements were made to the product in the process, this article will focus on Web Unlocker’s: 

Better auto-unlocking capabilities 

Web Unlocker’s environment emulation capabilities include: 

  • When examining Web Unlocker from a Network perspective it is able to successfully handle everything from IP type selection to IP rotations (when necessary).
  • On the Protocol-level Web Unlocker is able to carry out HTTP header management more effectively both during the decoding (i.e request received) and encoding (i.e. response sent) processes. Additionally, Web Unlocker has better ‘User-agents generation’ capabilities, meaning that it can deal with unique browser-fingerprint properties while enabling you to restrict the generated User-agents to match target site requirements. Lastly, Web Unlocker can now support HTTP2, meaning it can more effectively handle HTTP header field compression as well as server push. 
  • As far as Hardware (HW), and Operating Systems (OS) are concerned Web Unlocker is able to emulate all devices attached to any given system as well as their correlating drivers (i.e. full device enumeration imitation), mouse movements, screen resolution and various device properties.

Increased CAPTCHA-recognition, and solving 

Web Unlocker has better request management capabilities. It can now adapt itself in real-time to new blockades that have been set up by target sites. It uses Machine Learning (ML), improved CAPTCHA-resolving, and retry logic so that the most efficient (both in terms of time and resources) data-retrieval path can be attained. 

The algorithm that powers Web Unlocker is trained to use ideal fingerprint configurations and attempt retrieving desired responses/information from target sites. When the website responds, for example by:

  • Categorizing activity as ‘suspicious’ based on the number of requests from a specific IP (i.e. ‘rate limitations’) 
  • Detecting ‘unacceptable’ User-Agents (such as ‘HTTP’)
  • Or blocking requests based on geolocation 

Web Unlocker is able to analyze this information in real-time, and recalibrate settings on a per-domain basis, using specific/customized settings that will provide users with the highest success rates. 

Improved fingerprinting 

‘Device fingerprints’  are unique user identifiers for devices browsing the web. A fingerprint contains information regarding the configuration of a user’s browser as well as software/hardware environment. Having the fingerprinting aspect down pat is probably the most important aspect of unblocking as it solves the root of the issue and not just a derivative ‘symptom’ such as a ‘Recaptcha’. 

In this instance, you can ‘easily’, albeit manually change your IP, and start a new session. Or alternatively, trigger the ‘HTML body element’ rule. 

Web Unlocker completely automates all of this including fingerprinting adaptation to target sites. At the browser level it performs browser fingerprint emulation including:

  • Plugins, and fonts
  • Canvas/webGL fingerprints
  • Headers
  • Zombie, and sync cookies
  • WebRTC
  • Web Audio API fingerprints
  • cookie management

Web Unlocker also handles:

  • Disabling target-device fingerprinting
  • Real-time debugging

The bottom line 

‘Unblocker’ has been renamed and is now called ‘Web Unlocker’, utilizing the same technology, and algorithmic approach. Yes. Web Unlocker is the same product as ‘Web Unblocker’ but it has been upgraded as far as the resolution, and span of its environment emulation goes on all levels – Network, Protocol, Hardware/Operating systems as well as at the browser level. Its retry logic, CAPTCHA-resolving, and fingerprinting capabilities have also been taken one step further, putting it in a league of its own. 

Aviv Besinksky
Aviv Besinsky | Product Manager

Aviv is a lead product manager at Bright Data. He has been a driving force in taking data collection technology to the next level - developing technological solutions in the realms of data unblocking, static proxy networks, and more. Sharing his data crawling know-how is one of his many passions.