Requests vs. HTTPX vs. AIOHTTP: Detailed Comparison

Python is host to a large variety of HTTP clients. For those of you who are unfamiliar with HTTP (Hypertext Transfer Protocol), it’s the underlying framework of the entire web.

Today, we’re going to compare three of Python’s most popular HTTP clients: Requests, HTTPX, and AIOHTTP. If you’d like to learn about some of the others available, take a look here.

Brief Overview

Requests is the standard HTTP client for Python. It utilizes blocking and synchronous operations for ease of use. HTTPX is a newer asynchronous client designed for both speed and ease of use. AIOHTTP has been around for over a decade. It is one of the first and best supported async HTTP clients Python has to offer.

Python Requests

Python Requests is very intuitive and easy to use. If bleeding edge performance isn’t required, it’s the go-to for making HTTP requests in Python. It’s widely used, easy to understand, and the best documented Python HTTP client in the world.

You can install it with the command below.

Installation

pip install requests

Requests supports standard HTTP protocol and even some session management. With sessions, you can create a persistent connection to a server. This allows you to retrieve your data much faster and more efficiently than when making single requests. If you’re just looking to make basic (GET, POST, PUT, and DELETE) requests and high performance isn’t a concern, the Requests library can meet all your HTTP needs.

All over the web, you can find guides on proxy integration, user-agents and all sorts of other things.

You can view some basic usage examples below.

import requests

# make a simple request
response = requests.get("https://jsonplaceholder.typicode.com/posts")
print(response.status_code)

# use a session for multiple requests to the same server
with requests.Session() as client:
    for get_request in range(1000):
        response = client.get("https://jsonplaceholder.typicode.com/posts")
        print(response.status_code)

Requests falls short when it comes to asynchronous operations. With async support, you can make a batch of requests all at once. Then, you can await all of them. With synchronous requests, you can only make one request at a time and you have to wait for a response to come back from the server before making another request. If your program needs to make a lot of HTTP requests, using Requests will bring some inherent limitations to your code.

HTTPX

HTTPX is the newest and most modern of these three libraries. It gives us full support for async operations out of the box. That being said, it still has a user friendly and intuitive syntax. Use HTTPX if Requests just isn’t cutting it and you need to improve performance without too much of a learning curve.

With asyncio (asynchronous input and output), you can write code that fully takes advantage asynchronous responses with the await keyword. This allows us to continue our operations without blocking everything else in the code while we wait for things to happen. With these async operations, you can make large batches of requests. Instead of making one request at a time, you can make 5, 50, or even 100!

Installation

pip install httpx

Here are some examples for getting started with HTTPX.

import httpx
import asyncio

# synchronous response
response = httpx.get("https://jsonplaceholder.typicode.com/posts")
print(response.status_code)

# basic async session usage
async def main():
    async with httpx.AsyncClient() as client:
        for get_request in range(1000):
            response = await client.get("https://jsonplaceholder.typicode.com/posts")
            print(response.status_code)
asyncio.run(main())

HTTPX is an excellent choice when writing new code, but it does come with its own set of limitations. Due to its syntax, it might be difficult to go through and replace your existing Requests codebase with HTTPX. Writing async code requires a bit of boilerplate and unless you’re making thousands of requests, it’s often not worth the extra time spent on development.

In comparison to AIOHTTP (as you’ll soon learn), HTTPX is not quite as fast. If you’re looking to build a webserver, or a complex network, HTTPX is not a good choice due to its immature ecosystem. HTTPX is best for new, client-side applications with modern features such as HTTP/2.

AIOHTTP

When it comes to asynchronous programming, AIOHTTP has been widely used in Python for a long time. Whether you’re running a server, a client-side application, or a distributed network, AIOHTTP can fill all of these needs. However, of the three (Requests, HTTPX, and AIOHTTP), AIOHTTP has the steepest learning curve.

We’re focusing on simple client-side usage, so we’re not going to delve too far down the AIOHTTP rabbit hole. Like HTTPX, we can make batches of requests with AIOHTTP. However, unlike HTTPX, AIOHTTP offers no support for synchronous requests. Just don’t make too many at once… you don’t want to get banned by the server.

Installation

pip install aiohttp

Take a look at the basic usage below.

import aiohttp
import asyncio

# make a single request
async def main():
    async with aiohttp.ClientSession() as client:
        response = await client.get("https://jsonplaceholder.typicode.com/posts")
        print(response)

asyncio.run(main())


# basic async session usage
async def main():
    with aiohttp.ClientSession() as client:
        for response in range(1000):
            response = await client.get("https://jsonplaceholder.typicode.com/posts")
            print(response)

asyncio.run(main())

AIOHTTP is strictly asynchronous. As you can see in the code above, our single request example requires far more code than either of our first two examples. No matter how many requests we need to make, we need to set up an async session, therefore combining proxies with AIOHTTP is recommended. For a single request, this is definitely overkill.

As many strengths as it has, AIOHTTP is not going to completely replace Python Requests for you unless you’re dealing with tons of inbound and outbound requests at once. Then, it will substantially increase your performance. Use AIOHTTP when you’re building complex applications and requiring lightning fast communication. This library is best for servers, distributed networks, and highly complex web scraping applications.

Performance Comparison

Now, we’re going to build small program using each library. The requirements are simple: open a client session and perform 1000 API requests.

First, we need to create our session with the server. Next, we perform 1000 requests. We’ll have two arrays: one for good responses and one for bad responses. Once the run is complete, we print the full counts of good requests and bad requests. If we received and bad requests, we print their status codes to the console.

With our async examples (HTTPX, and AIOHTTP), we’ll use a chunkify() function. This is used to split an array into chunks. We then perform the requests in batches. For example, if we want to perform our requests in batches of 50, we’ll use chunkify() to create the batch and we’ll use process_chunk() to perform all 50 requests at once.

Take a look at these functions below.

def chunkify(iterable, size):
    iterator = iter(iterable)
    while chunk := list(islice(iterator, size)):
        yield chunk


async def process_chunk(client, urls, retries=3):
    tasks = [fetch(client, url, retries) for url in urls]
    return await asyncio.gather(*tasks)

Requests

Here is our code using Requests. It’s very simple compared to the two async examples we use later. We open a session and use a for loop to iterate through the requests.

import requests
import json
from datetime import datetime

start_time = datetime.now()
good_responses = []
bad_responses = []

with requests.Session() as client:

    for get_request in range(1000):
        response = client.get("https://jsonplaceholder.typicode.com/posts")
        status_code = response.status_code
        if status_code  == 200:
            good_responses.append(status_code)
        else:
            bad_responses.append(status_code)

end_time = datetime.now()

print("----------------Requests------------------")
print(f"Time elapsed: {end_time - start_time}")
print(f"Good Responses: {len(good_responses)}")
print(f"Bad Responses: {len(bad_responses)}")

for status_code in set(bad_responses):
    print(status_code)

Requests finished the job for us in just over 51 seconds. This comes out to about 20 requests per second. Without using a session, you can expect 2 seconds per request. This is pretty performant.

HTTPX

Here is the code for HTTPX. We utilize chunkify() and process_chunk() as we mentioned earlier.

import httpx
import asyncio
from datetime import datetime
from itertools import islice

def chunkify(iterable, size):
    iterator = iter(iterable)
    while chunk := list(islice(iterator, size)):
        yield chunk

async def fetch(client, url, retries=3):
    """Fetch a URL with retries."""
    for attempt in range(retries):
        try:
            response = await client.get(url)
            return response.status_code
        except httpx.RequestError as e:
            if attempt < retries - 1:
                await asyncio.sleep(1)
            else:
                return f"Error: {e}"

async def process_chunk(client, urls, retries=3):
    tasks = [fetch(client, url, retries) for url in urls]
    return await asyncio.gather(*tasks)

async def main():
    url = "https://jsonplaceholder.typicode.com/posts"
    total_requests = 1000
    chunk_size = 50

    good_responses = []
    bad_responses = []

    async with httpx.AsyncClient(timeout=10) as client:
        start_time = datetime.now()

        urls = [url] * total_requests
        for chunk in chunkify(urls, chunk_size):
            results = await process_chunk(client, chunk)
            for status in results:
                if isinstance(status, int) and status == 200:
                    good_responses.append(status)
                else:
                    bad_responses.append(status)

        end_time = datetime.now()

        print("----------------HTTPX------------------")
        print(f"Time elapsed: {end_time - start_time}")
        print(f"Good Responses: {len(good_responses)}")
        print(f"Bad Responses: {len(bad_responses)}")

        if bad_responses:
            print("Bad Status Codes or Errors:")
            for error in set(bad_responses):
                print(error)

asyncio.run(main())

Here is our output when using HTTPX. Compared to Requests, this is mindblowing. The total time was just over 7 seconds. This comes out to 139.47 requests per second. HTTPX has given us roughly 7 times the performance of Requests.

AIOHTTP

Now, we’ll perform the same exercise using AIOHTTP. We follow the same basic structure we used with the HTTPX example. The only major difference here is where the AIOHTTP client replaces the HTTPX client.

import aiohttp
import asyncio
from datetime import datetime
from itertools import islice

def chunkify(iterable, size):
    iterator = iter(iterable)
    while chunk := list(islice(iterator, size)):
        yield chunk

async def fetch(session, url, retries=3):
    for attempt in range(retries):
        try:
            async with session.get(url) as response:
                return response.status
        except aiohttp.ClientError as e:
            if attempt < retries - 1:
                await asyncio.sleep(1)
            else:
                return f"Error: {e}"

async def process_chunk(session, urls):
    tasks = [fetch(session, url) for url in urls]
    return await asyncio.gather(*tasks)

async def main():
    url = "https://jsonplaceholder.typicode.com/posts"
    total_requests = 1000
    chunk_size = 50

    good_responses = []
    bad_responses = []

    async with aiohttp.ClientSession() as session:
        start_time = datetime.now()

        urls = [url] * total_requests
        for chunk in chunkify(urls, chunk_size):
            results = await process_chunk(session, chunk)
            for status in results:
                if isinstance(status, int) and status == 200:
                    good_responses.append(status)
                else:
                    bad_responses.append(status)

        end_time = datetime.now()

        print("----------------AIOHTTP------------------")
        print(f"Time elapsed: {end_time - start_time}")
        print(f"Good Responses: {len(good_responses)}")
        print(f"Bad Responses: {len(bad_responses)}")

        if bad_responses:
            print("Bad Status Codes or Errors:")
            for error in set(bad_responses):
                print(error)

asyncio.run(main())

AIOHTTP came in lightning fast with just over 4 seconds. This HTTP client yielded over 241 requests per second! AIOHTTP is roughly 10 times faster than Requests and almost 50% faster than HTTPX. In Python, AIOHTTP is in a league of its own when it comes to performance.

How Bright Data’s Products Can Help

Bright Data offers a range of solutions that can enhance your HTTP client-based workflows, especially for data-heavy operations such as web scraping, API requests, and high-performance integrations. Here’s how each product fits in:

• Residential Proxies
• Web Scraper APIs
• Ready-Made Datasets
• Web Unlocker
• SERP API

By integrating Bright Data’s tools with Python HTTP clients, you can build robust, high-performance systems that simplify data collection while overcoming the typical challenges of web scraping and data acquisition.

Conclusion

In the world of HTTP clients, Requests is the standard simply because of its ease of use. Compared to Requests, HTTPX is more like moving to a modern car from a horse-drawn carriage. It gives us a balance between high performance and ease of use. AIOHTTP is like a rocket ship. You don’t want to use it unless absolutely necessary, but it is still by far, the fastest HTTP client out there.

Sign up for Bright Data today to unlock powerful data solutions and gain the competitive edge your business needs. All products come with a free trial!