The data collection industry is booming, and over the course of the past few years, dozens of proxy providers have sprung up to accommodate a genuine business need. This was made possible by the fact that the market is still young and the entry barrier low, especially considering the prevalence of white-labeling programs (so-called reselling). Competition is good for customers, but a lack of standards and quality control can easily lead to disappointment and frustration.
At Proxyway, we spend a great deal of time testing proxy services. I’d like to offer you some tips for choosing a reliable provider to meet your individual business needs. I don’t expect to cover every possible use case – only to give you guidelines toward making a well-grounded decision.
In this article we will discuss:
- Identifying the right proxy network for the job
- Figuring out the features you need
- Proxy network size matters
- Not all providers perform the same
- Consider the customer service
- Ethics is more than a word
Identifying the right proxy network for the job
One of the first steps in choosing a proxy provider should be determining if it can give you the type of proxy network you need.
There are several types of proxy networks to choose from, each with its own strengths and weaknesses. Datacenter proxies are fast and cheap but relatively simple to detect. Residential proxies cost more but may have decreased performance in certain instances while simultaneously enabling access to more websites without issues. ISP proxies combine the strengths of the first two but are pricey. And Mobile proxies cost the most of all; however, they have the best IP reputation and easily handle mobile-first websites.
The right tool depends on your use case. It’s a good idea to start with datacenter proxies first, and if they fail, move on to a more sophisticated proxy network. For reference, major search engines, popular retail, and travel aggregation websites will work much better with residential than datacenter IPs. Bright Data offers a handy feature called Proxy Waterfall which can help you determine the most efficient proxy network type.
If you want to simplify your web scraping infrastructure, consider proxy-IP-based tools like Web Unlocker. They might cost more than plain proxy IPs, but you won’t need to worry about blocks and can expect more predictable results.
Figuring out the features you need
Next, you should look for a provider that offers the necessary set of features. This ranges from functionality like IP rotation and filtering to proxy management tools.
For example, you might want IP addresses from a particular country. A provider can offer you a proxy pool that includes IPs from that country but give you no way to target them exclusively. This will quickly cause issues. Alternatively, you might only want to get IPs from a certain internet service provider’s ASN. This functionality is pretty rare, and not everyone supports it.
IP rotation works in the same way. For web scraping, you’ll most likely want proxies that rotate with every connection request. Conversely, some use cases will require keeping the same address for multiple requests – not an issue with datacenter proxies, but a relevant factor for Residential and Mobile IPs. Consider if you can get this functionality out of the box, and if not, whether you’ll be able to set it up yourself.
Proxy management tools are another concern. Things like proxy browser extensions aren’t a must: you can choose from multiple competent third-party options. However, a public API for programmatic proxy management is something that only the provider can give you. For some, it can make or break the deal.
Proxy network size matters
You might have noticed that providers advertise different amounts of proxy IPs under their belt. The numbers range from thousands in the case of datacenter proxies to millions – or even tens of millions – with Residential IPs. Is the proxy network’s size mostly marketing talk, or does it actually impact operational usage? The short answer is that network size absolutely matters!
Take datacenter proxies, for example. One of their major pitfalls is the issue of subnets ( blocks of 256 associated IP addresses). The math is simple here: the larger the proxy network, the more subnets it includes, affording users increased load balancing, and distribution flexibility.
The matter is even more important with residential addresses. Peer-to-peer residential proxies utilize the connections of real people. These IPs are available for proxy-based routing only when they meet certain criteria, and so they come and go throughout the day. The advertised numbers you see are merely monthly estimates; in reality, proxy pools will have much fewer available IPs on a day-to-day basis.
Earlier this year, we performed research pertaining to major Residential proxy providers. We discovered that some of them had as many as 10 times as many unique IPs given the same number of connection requests. This is an important figure, as a large network will not only have cleaner proxies but also ensure constant IP availability in any geolocation.
Not all providers perform the same
It’s relatively simple to check if a provider offers certain features. However, it’s much harder to gauge a proxy network’s performance and infrastructure.
One method is to test the service before committing. The proxy network market largely prefers giving money-back guarantees over free trials. This helps serve and deter potential abuse attempts. However, some premium providers – Bright Data included – do have a free trial for business clients.
Another method is to read reviews, especially ones that include performance tests. This can help you narrow down potential candidates, but you should also be wary of the pitfalls. For one, it’s very hard to evaluate a proxy network for all possible use cases: a provider can tailor its IPs for particular targets, and a lot depends on the web scraping setup. Testing a finite list of datacenter proxies is easier than benchmarking a large pool of residential addresses: they either work with the target website or not.
There are other non-obvious factors to consider. For example, backconnect proxies (anything that involves the words ‘residential’, ‘mobile’, or ‘rotating’) connect via load balancing servers. Their setup and location impact the connection quality. You should also ask for uptime metrics to see if the infrastructure is stable.
Our latest tests of residential proxy networks focus on evaluating their infrastructure: whether you can consistently connect to the load balancing server, proxy node, and finally the target website. Bright Data was among the few providers that succeeded over 99% of the time. Several other alternatives failed between one and three out of 10 requests – and that’s before encountering domain protection mechanisms.
Consider the customer service
Proxy networks are a high-maintenance resource: the IPs and servers require constant supervision, and sooner or later things will break. When that happens, good customer service can be worth its weight in gold (even at gold’s current price!).
But what constitutes good customer service? If your business relies on proxy IPs to deliver a constant stream of data, 24/7 working hours are a must. That’s not only because you might encounter issues at night – sometimes, it’s as simple as being in a different time zone (say, Europe as opposed to the US).
Ideally, you should have direct access to the technical team via live chat, Skype, or another instantaneous communication channel. As we’ve discovered, email or tickets can take hours to respond to, which defeats the purpose of round-the-clock service. Of course, this largely depends on the provider.
Having an account manager is another big plus. During business hours, they will be your main point of contact, with knowledge about your specific needs, issues, and business goals. Account managers effectively turn a simple business arrangement into a lasting and mutually beneficial relationship.
Finally, let’s not discount indirect support that comes in the form of documentation. Extensive help docs can be really empowering and reduce your reliance on the provider’s customer success team. Conversely, incomplete or plainly missing documentation can have a crippling effect. We’ve had cases where we couldn’t even set up the proxy server properly without first contacting a customer support agent.
Ethics is more than a word
Ethical proxy IP acquisition and use has become a hot button topic. Doubly so when it turns to Residential and Mobile proxy networks, as they involve real, sometimes unwitting users.
Ethical proxy handling involves two parts: IP acquisition and use. The first part, acquisition, determines how a provider sources their proxy IPs. The main ways include in-app SDKs, direct traffic-for-money exchange, and sometimes straight-up malware.
It’s important that end users consent to the access of their internet connection, and that they get something in return. Otherwise, your company will effectively be using a botnet service, which will constantly risk shutting down and damaging your reputation in the process.
The second part, proxy network use, defines which purposes they can serve. Proxy servers still carry a stigma from their involvement in ad fraud, credit card theft, email spam, and other illicit activities. This undermines the legitimate uses, such as brand protection or market research, casting a shadow on the whole industry.
If a provider fails or refuses to control black-hat activities, one client can use the same proxy network for ad fraud, while the other performs ad verification, with all the irony that entails. Once again, relying on such a service poses an ethical and reputational hazard for your company.
The bottom line
Finding a reliable proxy network is tricky, with so many options at the tip of your fingers. Consider your individual data needs, make customer service a priority in your decision-making process. And last but not least, choose a data network provider that employs ethical business practices that will maintain derived value for your stakeholders in the long term.
