In this article we will discuss:
What are Residential Proxies?
When collecting data over the internet, you send requests and receive responses using your computer’s IP. But when you are located in Australia, for example, and trying to access datasets in the US, responses may provide you with a biased response. Here is where proxy networks come in, allowing you to send requests through a third-party IP.
In this instance, we are talking about Residential Proxies which means that traffic is routed through IPs belonging to real individuals. This is accomplished by having a large network of peers who voluntarily opt-in (and can opt-out at any time) who provide access to their resources in exchange for a perk, such as an ad-free app experience.
Some advantages of using a Residential Proxy solution include:
- Being able to send large quantities of simultaneous requests without being detected and blocked by sites as a malicious threat
- Getting more accurate datasets as websites tend to not distort results such as pricing, and bundle offers for web users that appear to be local individuals
- Gaining access to open source information from tough target sites that might otherwise block your request for data (despite the fact that the information you are seeking is technically open-source)
Some of the disadvantages include:
- This tends to be a more expensive option though sometimes it is necessary and that much more effective
- Residential Proxies may be less effective if they are being utilized out of a shared pool of IPs instead of a dedicated one as other users may have ‘tainted’ them for specific sites that you are currently attempting to target (more on this in the next section)
Why do some businesses prefer dedicated Residential Proxies?
Dedicated Residential Proxies are groups of IPs that you route traffic through and are exclusively yours either for a specific period of time or forever (depending on if you leased or bought them). The opposite of this option is a shared Residential Proxy network, which is what companies typically deliberate between. These kinds of proxies are often referred to as private proxies.
If your data collection tasks are relatively simple, and you are targeting easy sites then your best bet would be to use a shared network. This is also the more cost-effective option of the two as tends to be the case in the sharing economy. Think of a hostel – when you pay for a single bed in a shared room of six, you will pay less than for a private individual room.
If you are targeting tougher sites, you have a larger budget, and your data collection needs are very specific then your best bet would be using a dedicated subnet of Residential IPs. These proxies tend to be more effective as there is no other simultaneous (and in some cases previous) activity associated with them.
Here is an example to illustrate this last very crucial point. When you purchase a home that has previously been lived in you bring in an engineer, and assessor to see the condition of the house. There might be termites in the attic, the electricity may not be able to handle more modern appliances, and the foundations may be rotting, you just don’t know. When buying new construction, you usually can forgo these steps as the property is brand new.
The same is true for shared vs. dedicated proxies – when you have exclusive access, you know that target sites will not have any previous bias towards your requests and you are therefore more likely to get quicker, more accurate responses from sites. If you are using a shared network, and one of the companies sharing the network with you just requested all the pricing on flights to Beijing from the same Online Travel Agency (OTA) that you are targeting, then your requests will likely get blocked or you will be served distorted results. This is due to the fact that when too many requests are received from the same subnet, this triggers a red flag on the other end.
The bottom line
Depending on your use case, the types of datasets you are trying to collect and the difficulty of accessing your desired target sites, will ultimately decide whether you choose a shared or dedicated network. You can also always start by using a shared Residential Proxy network and scale and escalate as needed using dedicated IPs.
