At the end of this AIOHTTP proxy integration guide, you will know:

What AIOHTTP is.
Why you should use a proxy.
How to set an HTTP, HTTPS, or SOCKS proxy in AIOHTTP.
ow to deal with proxy authentication, SSL certificate errors, IP rotation, and more.

What Is AIOHTTP?

AIOHTTP is a popular, open-source, Python async HTTP client/server framework. Its main features are:

Support for both client and server side of HTTP protocol.
Support for both client and server web sockets.
Provide a web server with middleware and pluggable routing.

What matters in this tutorial is its ability to act as an HTTP client to perform HTTP requests on top of asyncio. AIOHTTP is particularly well-suited for building scripts that need to handle multiple concurrent requests efficiently.

As of this writing, AIOHTTP has over 14k stars on GitHub.

Why Do You Need an AIOHTTP Proxy?

A good reason for using a proxy in AIOHTTP is to protect your online identity. By routing your request to a proxy server, you can hide your IP and location, making the destination site believe that your request comes from the proxy server and not from you. This means that it will be more difficult to identify and track you, adding confidentiality to your online operations while avoiding IP bans and geo-restrictions.

Note that you can achieve the same result with requests, as covered in our requests proxy integration tutorial.

Setting a Proxy in AIOHTTP Proxy

In this step-by-step section, you will learn how to set an HTTP, HTTPS, or SOCKS proxy in AIOHTTP.

AIOHTTP Prerequisites

Here, we will assume that you already have Python installed on your machine and a Python project with a virtual environment in place. Install AIOHTTP and its recommended dependencies with the following command:

pip install aiohttp[speedups]

This will add aiohttp to your project’s dependencies.

Open your Python script file and add the aiohttp import:

import aiohttp

To make aiohttp work with the async/await syntax, you will have to use asyncio. So import that library as well and set up a basic async main() function:


import aiohttp
import aiohttp

import asyncio

async def main():
  # aiohttp logic...

asyncio.run(main())

Perfect! You are ready to write the AIOHTTP proxy integration logic!

Proxy Prerequisites

Before seeing how to integrate a proxy in AIOHTTP, you need to understand what a proxy URL looks like.

This is the format of the URL required to connect to a proxy server in AIOHTTP:

<PROTOCOL>://[<USERNAME>:<PASSWORD>@]<HOST>[:<PORT>]

In detail, that consists of:

<PROTOCOL>: http for HTTP proxies, https for HTTPS proxies, and socks, socks4, or socks5 for proxies in the different SOCKS protocols.
<HOST>: The IP of the proxy server or its entire domain (e.g. proxydomain.com).
<PORT>: The port the proxy server listens to.
<USERNAME>: The optional username to authenticate the proxy connection.
<PASSWORD>: The optional password to authenticate the proxy connection.

Note that<PROTOCOL> and <HOST> are the only required parts of the URL. The <PORT> section is optional, but you typically have to specify it. Lastly, the credentials section <USERNAME>:<PASSWORD> is only necessary when dealing with authenticated proxies.

If you do not have the URL of a proxy server, you can get one online for free. At the same time, keep in mind that free proxies are unreliable, data-greedy, inconsistent, error-prone, and shady. You can use them only for learning purposes. In production, you will need a reliable premium proxy from a top-notch provider like Bright Data.

Later in this article, you will see how to get a Bright Data proxy and use it in AIOHTTP.

HTTP Proxy Integration

As mentioned in the documentation, AIOHTTP supports HTTP proxy integration. Assume this is the URL of your HTTP proxy:

http://190.6.23.219:999

You can specify it in your AIOHTTP request by using the proxy option as below:


async with aiohttp.ClientSession() as session:
    async with session.get(
        'https://example.com',
        proxy='http://190.6.23.219:999'
    ) as response:
        print(response.status)

AIOHTTP will now route the GET request through the proxy URL passed as a proxy argument to the get() function.

Keep in mind that you can use an HTTP proxy for both HTTP and HTTPS traffic.

HTTPS Proxy Integration

When it comes to HTTPS proxies, things become a bit more complex. As stated in the AIOHTTP documentation:

“In Python 3.10, support for TLS in TLS is disabled for the transports that asyncio uses. If the further release of Python (say v3.11) toggles one attribute, it’ll just work™.

aiohttp v3.8 and higher is ready for this to happen and has code in place supports TLS-in-TLS, hence sending HTTPS requests over HTTPS proxy tunnels.”

In other words, if you want to use HTTPS proxies in AIOHTTP, you need to upgrade Python to version 3.11 and aiohttp to v3.8 or higher. The approach to HTTPS AIOHTTP proxy integration remains the same as seen above. The only aspect that changes is that the proxy protocol will be https and no longer http.

SOCKS Proxy Integration

AIOHTTP does not support SOCKS proxies out of the box. If you have to use a SOCKS proxy in AIOHTTP, you have to install aiohtpp-socks:

pip install aiohttp_socks

This library uses python-socks to extend AIOHTTP with SOCKS4(a) and SOCKS5(h) support.

Consider this as the URL of your SOCKS proxy:

socks5://98.175.31.195:4145

Use it in AIOHTTP as follows:


# initialize a SOCKS proxy connector
connector = ProxyConnector.from_url('socks5://98.175.31.195:4145')

# initialize an AIOHTTP client with the SOCKS proxy connector
async with aiohttp.ClientSession(connector=connector) as session:
    async with session.get('https://example.com') as response:
        print(response.status)

AIOHTTP will now route the GET request through the specified SOCKS proxy.

ProxyConnector comes from aiohttp-socks, so do not forget to import with:

from aiohttp_socks import ProxyConnector

Behind the scenes, aiohttp-socks defines an AIOHTTP Connector to tweak the requests and make them go through the given SOCKS proxy.

Complete Example

You just learned how to use a proxy with AIOHTTP. Let’s verify that the above approaches work in a complete example!

For the sake of simplicity, we will work with an HTTP proxy, but you can easily adapt the example below to HTTPS and SOCKS.

To target URL of the GET request will be the /ip endpoint from the HTTPBin project. This API returns the IP of the incoming request. Thus, if the proxy integration is successful, the endpoint should respond with the IP of the proxy server and not yours.

Again, assume this is the URL of your HTTP proxy server:

http://190.6.23.219:999

Integrate it into AIOHTTP and print the data returned by the https://httpbin.io/ip endpoint with:


import aiohttp
import asyncio

async def main():
    async with aiohttp.ClientSession() as session:
        # perform a GET request through an HTTP proxy
        async with session.get(
          'https://httpbin.io/ip',
          proxy='http://190.6.23.219:999'
        ) as response:
            # extract the response data and print it
            json = await response.json()
            print(json)

asyncio.run(main())

Execute this script, and you should get:


{
  'origin': '190.6.23.219'
}

Et voilà! That is exactly the IP of the proxy server, which means that the AIOHTTP proxy integration works like a charm.

Note: Proxy servers are short-lived. By the time you read this tutorial, the selected proxy will no longer work and the above script will fail. To make it work, replace the HTTP proxy with a fresh and working one.

AIOHTPP Proxy: Advanced Use Cases

You know the basics of AIOTTP proxy integration, but what about more complex techniques? Find out more in this section!

Global Proxies Via Environment Variables

AIOHTTP supports two ways to determine which proxy to use:

It uses the proxy URL passed to the proxy argument.
If trust_env is set to True, it reads the HTTP_PROXY and HTTPS_PROXY environment variables.

You can therefore configure a proxy globally in AIOHTTP by setting the envs below:

HTTP_PROXY: The URL of the proxy server to use for HTTP requests.
HTTPS_PROXY: The URL of the proxy server to use for HTTPS requests.

For example, set them on Linux or macOS with the following commands:


export HTTP_PROXY="<PROTOCOL>://[<USERNAME>:<PASSWORD>@]<HOST>[:<PORT>]"
export HTTPS_PROXY="<PROTOCOL>://[<USERNAME>:<PASSWORD>@]<HOST>[:<PORT>]"

Next, pass trust_env=True to aiohttp.ClientSession():

async with aiohttp.ClientSession(trust_env=True) as session:

You can now call session.get() with no proxy argument


async with aiohttp.ClientSession(trust_env=True) as session:
    async with session.get('https://example.com') as response:
        print(response.status)

Great! AIOHTTP will use the proxies specified in the environment variables for each HTTP and HTTPS request.

Follow the link to learn more, as the same mechanism also works with proxies in cURL.

Proxy Authentication Approaches

Trying to connect to an authenticated proxy without the right credentials will result in a 407 Proxy Authentication Required error.

Suppose the string below is the URL to connect to an authenticated HTTP proxy:

http://admin:[email protected]:8928

Here, the bare proxy URL is:

http://20.198.112.312:892

And the authentication credentials are:

Username: admin
Password: pK5ip98NWp56l9sjy

AIOHTTP supports two ways to deal with authenticated proxies:

Specify the credentials directly in the proxy URL:


async with session.get(
  'https://example.com/',
  proxy='http://admin:[email protected]:8928'
) as response:

2. Pass the username and password credentials to an aiohttp.BasicAuth() object and then use it in the proxy_auth option:


proxy_auth = aiohttp.BasicAuth('admin', 'pK5ip98NWp56l9sjy')
async with session.get(
    'https://example.com/',
    proxy="http://20.198.112.312:892",
    proxy_auth=proxy_auth
) as response:

Awesome! AIOHTTP proxy authentication is no longer a secret.

Avoid SSL Certificate Errors

When setting a proxy in AIOHTTP, your requests may fail because of the following error:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain

To avoid these SSL certificate issues, turn off SSL verification with ssl=False:


async with aiohttp.ClientSession() as session:
    async with session.get(
      'https://example.com',
      proxy='http://190.6.23.219:999',
      ssl=False # turn of SSL certificate verification
    ) as response:
        print(response.status)

The AIOHTTP SSL certificate verification errors will immediately disappear.

Rotating Proxies in AIOHTTP

If you rely on the same proxy server over and over again, there is a chance that the target site will block its IP address. To avoid that, you must ensure that your request uses different proxy servers.

Here is how you can do it:

Populate a Python list with many proxy URLs.
Randomly pick a proxy URL before each request.
Set the chosen proxy URL in AIOHTTP.

The main problem with this approach is that it involves boilerplate code. Also, it requires access to a pool of reliable proxy servers, which is not free. Luckily, Bright Data has a more effective solution! Its rotating proxies automatically change the exit IP addresses for you, are available in 195 countries, boast exceptional network uptime, and guarantee a success rate of 99.9%.

In the next chapter, you will learn how to get started with Bright Data’s rotating proxies in AIOHTTP.

Integrating AIOHTTP with a Bright Data Proxy

Bright Data controls the best proxy servers in the world, serving Fortune 500 companies and over 20,000 customers. Its worldwide proxy network involves:

Datacenter proxies – Over 770,000 datacenter IPs.
Residential proxies – Over 72M residential IPs in more than 195 countries.
ISP proxies – Over 700,000 ISP IPs.
Mobile proxies – Over 7M mobile IPs.

This is one of the largest and most reliable proxy networks in the world!

Follow the steps below and learn how to use Bright Data’s proxies in AIOHTTP.

If you already have an account, log in to Bright Data. Otherwise, create an account for free. You will gain access to the following user dashboard:

Click the “View proxy products” button as below:

You will be redirected to the following “Proxies & Scraping Infrastructure” page:

Scroll down, find the “Residential Proxies” card, and click on the “Get started” button:

You will reach the residential proxy configuration dashboard. Follow the guided wizard and set up the proxy service based on your needs. If you have any doubts about how to configure the proxy, feel free to contact the 24/7 support.

Go to the “Access parameters” tab and retrieve your proxy’s host, port, username, and password as follows:

Note that the “Host” field already includes the port.

That is all you need to build the proxy URL and use it in AIOHTTP. Put all the information together, and build a URL with the following syntax:

<Username>:<Password>@<Host>

For example, in this case it would be:

brd-customer-hl_4hgu8dwd-zone-residential:[email protected]:XXXXX

Toggle “Active proxy,” follow the last instructions, and you are ready to go!

Your AIOHTTP proxy snippet for Bright Data integration will look like as follows:


import aiohttp
import asyncio

async def main():
    async with aiohttp.ClientSession() as session:
        # perform a GET request through an HTTP proxy
        async with session.get(
          'https://lumtest.com/myip.json',
          proxy='http://brd-customer-hl_4hgu8dwd-zone-residential:[email protected]:XXXXX'
        ) as response:
            # extract the response data and print it
            json = await response.json()
            print(json)

asyncio.run(main())

Conclusion

In this AIOHTTP proxy integration tutorial, you saw why you should use proxies and how to do it with aiohttp. You now know how to set up an HTTP, HTTPS, or SOCKS proxy in AIOHTTP, a popular Python HTTP client library. As proved here, that only takes a few lines of code!

Thanks to this guide, you also understood why you should never use free proxy services. Instead, you should adopt reliable proxy servers from a reputable proxy service provider. The best provider in the market? Bright Data! The proxy integration procedure in AIOHTTP remains the same, but the benefits of premium proxies are endless.

Start free trial

How to Set Proxy in AIOHTTP