In this guide on how to rotate proxies in Python, you will learn:
- What proxies are and why use them
- What proxy rotation is and why you might need it
- How to rotate proxies in Python
- Common limitations when you rotate proxies in Python
Let’s dive in!
What Is a Proxy?
A proxy is a server that acts as an intermediary between a user and a network resource over the Internet. So, you can think of a proxy as a middleman that forwards requests and responses between parties.
Why Use Proxies in Python?
When you send a request to a website using a proxy, the request first goes through the proxy server. Then, the proxy forwards your request to the website, receives the response, and sends it back to you. This process masks your IP address to the destination, making it appear as if the request is coming from the proxy server instead of your device.
The typical reason why you want to use a proxy is in the case of web request automation or web scraping. In this scenario, Python is one of the best languages for web scraping, thanks to its extensive libraries and large, active community.
What Is Proxy Rotation and Why Do You Need it?
If you make too many requests from a single IP address, websites may block you through rate limiting or outright IP bans. This is where proxy rotation via proxies comes into help.
Systematically switching between different proxy servers while making web requests is one of the best ways to implement IP address rotation. This procedure helps you bypass common anti-scraping techniques and provides you with the following benefits:
- Avoiding IP blocks: Distribute requests across multiple IPs, making it harder for websites to detect and block your scraping activity.
- Bypassing rate limits: Websites often set request limits per IP address within a specific timeframe. Rotating proxies helps you continue scraping even after reaching these limits on one IP.
- Accessing geo-restricted content: Some websites show different content based on geographic location. Proxy rotation with proxies from different countries allows you to access location-specific content.
How to Rotate Proxies in Python: 3 Approaches
Now that you know what proxies are and why to rotate them, get ready for some step-by-step tutorials in Python. The next paragraphs will show you how to rotate proxies in Python by using different approaches and libraries.
The target site for all scripts will be the /ip
endpoint from the HTTPBin project. This special endpoint returns the caller’s IP address, making it perfect for testing whether the IP seen by the server is rotating.
Time to rotate some proxies in Python!
Requirements
To replicate the tutorials for rotating proxies with Python, you must have Python 3.7 or higher installed on your machine.
Prerequisites
Suppose you call the main folder of your project proxy_rotation/
. At the end of this step, the folder will have the following structure:
Where:
requests.py
, andasync.py
are Python files that store Requests and AIOHTTP proxy rotation logic, respectively.scrapy_rotation/
is a folder that contains a Scrapy project. You will create and instantiate it later.venv/
contains the virtual environment
You can create the venv/
virtual environment directory like so:
To activate it, on Windows, run:
Equivalently, on macOS and Linux, execute:
As a final prerequisite, you need to retrieve a list of proxies. For this article, you can use our free proxies list.
How to Rotate Proxies in Python With Requests
In this tutorial section, you will learn how to rotate proxies in Python with Requests.
Step #1: Install Dependencies
In the virtual environment activated, install Requests with:
Step #2: Define the Rotation Logic
To rotate proxies in Python with Requests, write the following code into the requests_file.py
file:
Where:
- The
get_random_proxy()
function stores the list of proxies you have retrieved and returns a random one with the methodrandom.choice()
. - The
for
loop iterates over the randomized list of proxies and makes the actual request with the methodrequests.get()
. For more information, read our guide on using a proxy with Python Requests.
Step #3: Launch the Script
To launch the script, run:
Below is the expected response:
Wonderful! The exit IPs of your script have been rotated as desired.
How to Rotate Proxies in Python With AIOHTTP
The main limitation of the randomized approach using the Requests library is that it uses one proxy at a time. That means you need to wait for each request to be concluded before the next proxy is used.
To avoid that limitation, you can use AIOHTTP. This library enables you to make asynchronous requests, thus using multiple proxies simultaneously in a non-blocking way. In other words, it allows you to rotate the proxies from your list by making asynchronous, parallel requests to the target server. See AIOHTTP in action in our guide on asynchronous web scraping.
The following section shows how to rotate proxies in Python with AIOHTTP.
Step #1: Install Dependencies
In the virtual environment activated, install AIOHTTP with:
Step #2: Define the Rotation Logic
To rotate proxies in Python with AIOHTTP, write the following code into the async.py
file:
This code does the following:
- The
fetch_ip()
function manages the requests by taking the session, a proxy, and the attempt number. In particular, it sends aGET
request to the target website and prints the response. - The
main()
function:- Creates a session with the method
aiohttp.ClientSession()
for managing HTTP connections. - Sets up a list of tasks to send multiple requests concurrently and the number of attempts.
- Rotates the proxies using the modulus operator (
i % len(proxies_list)
) to ensure proxies are reused in a round-robin fashion if the number of requests exceeds the number of proxies. Discover more in our tutorial on how to set a proxy in AIOHTTP. - Uses the method
asyncio.gathe``r()
to run all the tasks concurrently.
- Creates a session with the method
Step #3: Launch the Script
To launch the script, run:
This is the expected response:
Amazing! The IPs are being rotated as expected.
How to Rotate Proxies With Python Scrapy
In a previous article, we discussed the possibility of rotating proxies in Python with Scrapy by using scrapy-rotating-proxies
.
In this guided section, you will learn how to do that!
Step #1: Install Dependencies
In the virtual environment activated, install the necessary libraries:
Step #2: Create a New Scrapy Project
Inside the main folder of your repository (proxy_rotation/
), instantiate a new Scrapy project with this command:
This will create a new subfolder called scrapy_rotation/
that has the following structure:
From the main folder (proxy_rotation/
), move to the scrapy_rotation/
one:
You can now create a new spider that points to the target website by running:
This script also creates the rotation.py
file inside the spiders/
folder.
Step #3: Define the Rotation Logic
The proxy rotation logic can be managed by modifying the settings.py
file with the following settings:
What manages proxy rotation here is the rotating_proxies.middlewares.RotatingProxyMiddleware: 610
option in DOWNLOADER_MIDDLEWARES
. In particular, this option selects a proxy from the ROTATING_PROXY_LIST
and assigns it to each request.
Also, the rotating_proxies.middlewares.BanDetectionMiddleware: 620
option allows the scraper to detect if an IP has been banned or blocked by the target website. If a request fails due to that reason, the middleware will retry the request with a new proxy. So, this option works closely with the RotatingProxyMiddleware
to ensure that banned proxies are automatically avoided.
Now, in the rotation.py
file inside the spiders/
folder you can write the following:
This class instantiates the whole spider and prints the response at each request.
Step #4: Launch the Script
To launch the script, you have to use the name of the IpSpider()
class—which is ip_spider
:
The data returned by Scrapy on the CLI is particularly complete. So, if everything went fine, among the other information, you will find something like this:
Limitations of the Above Approaches to Proxy Rotation in Python
The proxy rotation methods mentioned above are useful, but they come with some limitations:
- They require you to manually retrieve and manage a list of proxies.
- They involve boilerplate code.
- They may still result in IP bans if you don’t use high-quality proxy servers.
If you are looking for a more efficient and effective way to handle proxy rotation in Python, Bright Data offers some of the best rotating proxies on the market. With just a single proxy URL, you can integrate them into your HTTP client or scraping library. That eliminates the need for boilerplate code and manual rotation management.
Other key benefits of this approach are:
- Automatic IP rotation with configurable sticky IPs
- Access to 72+ million residential IPs
- Geolocation control over the proxy server locations
- Support for HTTP, HTTPS, and SOCKS protocols
Simplify your proxy management—discover our auto-rotating proxies!
Conclusion
In this article, you learned how to rotate proxies in Python using three different libraries: Requests, AIOHTTP, and Scrapy. As demonstrated in the guided sections above, the process is not complex and requires only a few lines of code.
However, that approach comes with a few drawbacks:
- The code is boilerplate-heavy, making your script less maintainable.
- You need to manage and provide access to a large list of proxy servers.
Fortunately, you can skip all that hassle with Bright Data’s auto-rotating proxies—a more efficient solution to rotate proxies in Python.
Bright Data controls the best proxy servers in the world, serving Fortune 500 companies and more than 20,000 customers. Its offer includes a wide range of proxy types:
- Datacenter proxies – Over 770,000 datacenter IPs.
- Residential proxies – Over 72M residential IPs in more than 195 countries.
- ISP proxies – Over 700,000 ISP IPs.
- Mobile proxies – Over 7M mobile IPs.
Create a free Bright Data account today to test our proxies and scraping solutions!
No credit card required