---
title: "Asynchronous Web Scraping With AIOHTTP in Python"
slug: web-scraping-with-aiohttp
date: 2025-01-22T12:19:20+00:00
modified: 2026-08-24T12:56:57+00:00
permalink: https://brightdata.com/blog/web-data/web-scraping-with-aiohttp
type: blog
---

[ Blog ](https://brightdata.com/blog "Blog") / [Web Data](https://brightdata.com/blog/web-data)







 [Web Data](https://brightdata.com/blog/web-data)

# Asynchronous Web Scraping With AIOHTTP in Python

Discover AIOHTTP for web scraping! Learn setup, features, and advanced techniques, plus a comparison with Requests for efficient data extraction.

 14 min read





 [ ](https://brightdata.com/blog/authors/antonello-zanini)

 [Antonello Zanini

Technical Writer

 ](https://brightdata.com/blog/authors/antonello-zanini)





 ![web scraping with aiohttp and python blog image](https://media.brightdata.com/2025/01/web-scraping-with-aiohttp-and-python.svg)





In this guide, you will explore:

- What AIOHTTP is and the key features it provides
- A step-by-step section on using AIOHTTP for web scraping
- Advanced techniques for web scraping with AIOHTTP
- An AIOHTTP vs Requests comparison for handling automated requests

Let’s dive in!

## <a></a>What Is AIOHTTP?

[AIOHTTP](https://docs.aiohttp.org/en/stable/) is an asynchronous client/server HTTP framework built on top of Python’s [`asyncio`](https://docs.python.org/3/library/asyncio.html). Unlike traditional HTTP clients, AIOHTTP uses client sessions to maintain connections across multiple requests. That makes it an efficient choice for high-concurrency session-based tasks.

**⚙️ Features**

- Supports both the client and server sides of the HTTP protocol.
- Provides native support for WebSockets (both client and server).
- Offers middleware and pluggable routing for web servers.
- Efficiently handles streaming large data.
- Includes client session persistence, enabling connection reuse and reducing overhead for multiple requests.

## <a></a>Scraping with AIOHTTP: Step-By-Step Tutorial

In the context of web scraping, AIOHTTP is just an HTTP client to fetch the raw HTML content of a page. To parse and extract data from that HTML, you then need an HTML parser like [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/).

Follow this section to learn how to use AIOHTTP for web scraping with BeautifulSoup!

**Warning**: Although AIOHTTP is used primarily in the initial stages of the process, we will guide you through the entire scraping workflow. If you are interested in more advanced AIOHTTP web scraping techniques, feel free to skip ahead to the next chapter after Step 3.

### <a></a>Step #1: Setup Your Scraping Project

Ensure that Python 3+ is installed on your machine. If not, [download it from the official site](https://www.python.org/downloads/) and follow the installation instructions.

Next, create a directory for your AIOHTTP scraping project using this command:

```none
mkdir aiohttp-scraper

```

Navigate into that directory and set up a [virtual environment](https://docs.python.org/3/library/venv.html):

```none
cd aiohttp-scraper
python -m venv env

```

Open the project folder in your preferred Python IDE. [Visual Studio Code with the Python extension](https://code.visualstudio.com/docs/languages/python) or [PyCharm Community Edition](https://www.jetbrains.com/pycharm/download/#section=windows) are both valid choices.

Now, create a `scraper.py` file inside the project folder. It will be empty at first, but you will soon add the scraping logic to it.

In your IDE’s terminal, activate the virtual environment. On Linux or macOS, use:

```none
./env/bin/activate

```

Equivalently, on Windows, run:

```none
env/Scripts/activate

```

Great! You are all set up and ready to go.

### <a></a>Step #2: Set Up the Scraping Libraries

With the virtual environment activated, install AIOHTTP and BeautifulSoup using the command below:

```none
pip install aiohttp beautifulsoup4

```

This will add both the [`aiohttp`](https://docs.aiohttp.org/en/stable/) and [`beautifulsoup4`](https://pypi.org/project/beautifulsoup4/) to your project’s dependencies.

Import them into your `scraper.py` script:

```none
import asyncio
import aiohttp
from bs4 import BeautifulSoup

```

Note that `aiohttp` requires the `asyncio` to work.

Now, add the following `async` function workflow to your `scrper.py` file:

```none
async def scrape_quotes():
    # Scraping logic...

# Run the asynchronous function
asyncio.run(scrape_quotes())

```

`scrape_quotes()` defines an asynchronous function where your scraping logic will run concurrently without blocking. Finally, `asyncio.run(scrape_quotes())` starts and runs the asynchronous function.

Awesome! You can proceed to the next step in your scraping workflow.

### <a></a>Step #3: Get the HTML of the Target Page

In this example, you will see how to scrape data from the “[Quotes to Scrape](https://quotes.toscrape.com/)” site:

With libraries like Requests or AIOHTTP, you would simply make a GET request and directly receive the HTML content of the page. However, AIOHTTP follows a [different request lifecycle](https://docs.aiohttp.org/en/stable/http_request_lifecycle.html).

The AIOHTTP primary component is the [`ClientSession`](https://docs.aiohttp.org/en/stable/client_reference.html), which manages a pool of connections and supports [`Keep-Alive`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Keep-Alive) by default. Instead of opening a new connection for every request, it reuses connections, improving performance.

When making a request, the process typically involves three steps:

1. Opening a session through `ClientSession()`.
2. Sending the GET request asynchronously with [`session.get()`](https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientSession.get).
3. Accessing the response data with methods like `await response.text()`.

This design allows the event loop to use different [`with` contexts](https://docs.python.org/3/reference/datamodel.html#context-managers) between operations without blocking, making it ideal for high-concurrency tasks.

Given that, you can use AIOHTTP to retrieve the HTML of the homepage with this logic:

```none
async with aiohttp.ClientSession() as session:
    async with session.get("http://quotes.toscrape.com") as response:
        # Access the HTML of the target page
        html = await response.text()

```

Behind the scenes, AIOHTTP sends the request to the server and waits for the response, which contains the HTML of the page. Once the response is received, `await response.text()` extracts the HTML content as a string.

Print the `html` variable and you will see:

```none

<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Quotes to Scrape</title>
    <link rel="stylesheet" href="/static/bootstrap.min.css">
    <link rel="stylesheet" href="/static/main.css">
</head>
<body>
    <!-- omitted for brevity... -->
</body>
</html>

```

Way to go! You successfully retrieved the HTML content of the target page. Time to parse this content and extract the data you need.

### <a></a>Step #4: Parse the HTML

Pass the HTML content into the BeautifulSoup constructor to parse it:

```none
# Parse the HTML content using BeautifulSoup
soup = BeautifulSoup(html, "html.parser")

```

[`html.parser`](https://docs.python.org/3/library/html.parser.html) is the default Python HTML parser used to process the content.

The `soup` object contains the parsed HTML and provides methods to extract the data you need.

AIOHTTP has handled retrieving the HTML, and now you are transitioning into the typical data parsing phase with BeautifulSoup. For more details, read our tutorial on [BeautifulSoup web scraping](/blog/how-tos/beautiful-soup-web-scraping).

### <a></a>Step #5: Write the Data Extraction Logic

You can scrape the quotes data from the page using the following code:

```none
# Where to store the scraped data
quotes = []

# Extract all quotes from the page
quote_elements = soup.find_all("div", class_="quote")

# Loop through quotes and extract text, author, and tags
for quote_element in quote_elements:
    text = quote_element.find("span", class_="text").get_text().get_text().replace("“", "").replace("”", "")
    author = quote_element.find("small", class_="author")
    tags = [tag.get_text() for tag in quote_element.find_all("a", class_="tag")]

    # Store the scraped data
    quotes.append({
        "text": text,
        "author": author,
        "tags": tags
    })

```

This snippet initializes a list named `quotes` to hold the scraped data. It then identifies all quote HTML elements and loops through them to extract the quote text, author, and tags. Each extracted quote is stored as a dictionary in the `quotes` list, organizing the data for later use or export.

Super! Scraping logic is now implemented.

### <a></a>Step #6: Export the Scraped Data

Use these lines of code to export the scraped data into a CSV file:

```none
# Open the file for export
with open("quotes.csv", mode="w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=["text", "author", "tags"])

    # Write the header row
    writer.writeheader()

    # Write the scraped quotes data
    writer.writerows(quotes)

```

The above snippet opens a file named `quotes.csv` in write mode. Then it, sets up column headers (`text`, `author`, `tags`), writes the headers, and then writes each dictionary from the `quotes` list to the CSV file.

[`csv.DictWriter`](https://docs.python.org/3/library/csv.html#csv.DictWriter) simplifies data formatting, making it easier to store structured data. To make it work, remember to import `csv` from the Python Standard Library:

```none
import csv

```

### <a></a>Step #7: Put It All Together

This is what your final AIOHTTP web scraping script should look like:

```none
import asyncio
import aiohttp
from bs4 import BeautifulSoup
import csv

# Define an asynchronous function to make the HTTP GET request
async def scrape_quotes():
    async with aiohttp.ClientSession() as session:
        async with session.get("http://quotes.toscrape.com") as response:
            # Access the HTML of the target page
            html = await response.text()

            # Parse the HTML content using BeautifulSoup
            soup = BeautifulSoup(html, "html.parser")

            # List to store the scraped data
            quotes = []

            # Extract all quotes from the page
            quote_elements = soup.find_all("div", class_="quote")

            # Loop through quotes and extract text, author, and tags
            for quote_element in quote_elements:
                text = quote_element.find("span", class_="text").get_text().replace("“", "").replace("”", "")
                author = quote_element.find("small", class_="author").get_text()
                tags = [tag.get_text() for tag in quote_element.find_all("a", class_="tag")]

                # Store the scraped data
                quotes.append({
                    "text": text,
                    "author": author,
                    "tags": tags
                })

            # Open the file name for export
            with open("quotes.csv", mode="w", newline="", encoding="utf-8") as file:
                writer = csv.DictWriter(file, fieldnames=["text", "author", "tags"])

                # Write the header row
                writer.writeheader()

                # Write the scraped quotes data
                writer.writerows(quotes)

# Run the asynchronous function
asyncio.run(scrape_quotes())

```

You can run it with:

```none
python scraper.py

```

Or, on Linux/macOS:

```none
python3 scraper.py

```

A `quotes.csv` file will appear in the root folder of your project. Open it and you will see:

Et voilà! You just learned how to perform web scraping with AIOHTTP and BeautifulSoup.

## <a></a>AIOHTTP for Web Scraping: Advanced Features and Techniques

Now that you understand how to use AIOHTTP for basic web scraping, it is time to see more advanced scenarios.

In the following examples, the target site will be the [HTTPBin.io `/anything` endpoint](https://httpbin.io/anything). That is a handy API that returns the IP address, headers, and other data sent by the requester.

Get ready to master AIOHTTP for web scraping!

### <a></a>Set Custom Headers

You can [specify custom headers](https://docs.aiohttp.org/en/stable/client_advanced.html#custom-request-headers) in an AIOHTTP request with to the `headers` argument:

```none
import aiohttp
import asyncio

async def fetch_with_custom_headers():
    # Custom headers for the request
    headers = {
        "Accept": "application/json",
        "Accept-Language": "en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7,es-US;q=0.6,es;q=0.5,it-IT;q=0.4,it;q=0.3"
    }

    async with aiohttp.ClientSession() as session:
        # Make a GET request with custom headers
        async with session.get("https://httpbin.io/anything", headers=headers) as response:
            data = await response.json()
            # Handle the response...
            print(data)

# Run the event loop
asyncio.run(fetch_with_custom_headers())

```

This way, AIOHTTP will make a GET HTTP request with the [`Accept`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept) and [`Accept-Language`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language) headers set.

### <a></a>Set a Custom User Agent

[`User-Agent`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent) is one of the most critical [HTTP headers for web scraping](/blog/web-data/http-headers-for-web-scraping). By default, AIOHTTP uses this `User-Agent`:

```none
Python/<PYTHON_VERSION> aiohttp/<AIOHTTP_VERSION>

```

The default value above can easily expose your requests as coming from an automated script. That will increase the risk of being blocked by the target site.

To reduce the chances of getting detected, you can set a custom real-world `User-Agent` as before:

```none
import aiohttp
import asyncio

async def fetch_with_custom_user_agent():
    # Define a Chrome-like custom User-Agent
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
    }

    async with aiohttp.ClientSession(headers=headers) as session:
        # Make a GET request with the custom User-Agent
        async with session.get("https://httpbin.io/anything") as response:
            data = await response.text()
            # Handle the response...
            print(data)

# Run the event loop
asyncio.run(fetch_with_custom_user_agent())

```

Discover the [best user agents for web scraping](/blog/how-tos/user-agents-for-web-scraping-101)!

### <a></a>Set Cookies

Just like HTTP headers, you can [set custom cookies](https://docs.aiohttp.org/en/v3.7.3/client_advanced.html#custom-cookies) using the `cookies` in `ClientSession()`:

```none
import aiohttp
import asyncio

async def fetch_with_custom_cookies():
    # Define cookies as a dictionary
    cookies = {
        "session_id": "9412d7hdsa16hbda4347dagb",
        "user_preferences": "dark_mode=false"
    }

    async with aiohttp.ClientSession(cookies=cookies) as session:
        # Make a GET request with custom cookies
        async with session.get("https://httpbin.io/anything") as response:
            data = await response.text()
            # Handle the response...
            print(data)

# Run the event loop
asyncio.run(fetch_with_custom_cookies())

```

Cookies help you include session data required in your web scraping requests.

Note that cookies set in `ClientSession` are shared across all requests made with that session. To access session cookies, refer to [`ClientSession.cookie_jar`](https://docs.aiohttp.org/en/v3.7.3/client_reference.html#aiohttp.ClientSession.cookie_jar).

### <a></a>Proxy Integration

In AIOHTTP, you can route your requests through a proxy server to reduce the risk of IP bans. Do that by using the [`proxy` argument](https://docs.aiohttp.org/en/v3.7.3/client_advanced.html#proxy-support) in the HTTP method function on `session`:

```none
import aiohttp
import asyncio

async def fetch_through_proxy():
    # Replace with the URL of your proxy server
    proxy_url = "<YOUR_PROXY_URL>"

    async with aiohttp.ClientSession() as session:
        # Make a GET request through the proxy server
        async with session.get("https://httpbin.io/anything", proxy=proxy_url) as response:
            data = await response.text()
            # Handle the response...
            print(data)

# Run the event loop
asyncio.run(fetch_through_proxy())

```

Find out how to perform proxy authentication and rotation in our guide on [how to use a proxy in AIOHTTP](/blog/how-tos/proxy-in-aiohttp).

### <a></a>Error Handling

By default, AIOHTTP raises errors only for connection or network issues. To raise exceptions for HTTP responses when receiving [`4xx`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses) and [`5xx`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses) status codes, you can use any of the following approaches:

1. **Set `raise_for_status=True` when creating the `ClientSession`**: Automatically raise exceptions for all requests made through the session if the response status is `4xx` or `5xx`.
2. **Pass `raise_for_status=True` directly to request methods**: Enable error raising for individual request methods (like `session.get()` or `session.post()`) without affecting others.
3. **Call `response.raise_for_status()` manually**: Give full control over when to raise exceptions, allowing you to decide on a per-request basis.

Option #1 example:

```none
import aiohttp
import asyncio

async def fetch_with_session_error_handling():
    async with aiohttp.ClientSession(raise_for_status=True) as session:
        try:
            async with session.get("https://httpbin.io/anything") as response:
                # No need to call response.raise_for_status(), as it is automatic
                data = await response.text()
                print(data)
        except aiohttp.ClientResponseError as e:
            print(f"HTTP error occurred: {e.status} - {e.message}")
        except aiohttp.ClientError as e:
            print(f"Request error occurred: {e}")

# Run the event loop
asyncio.run(fetch_with_session_error_handling())

```

When `raise_for_status=True` is set at the session level, all requests made through that session will raise an `aiohttp.ClientResponseError` for `4xx` or `5xx` responses.

Option #2 example:

```none
import aiohttp
import asyncio

async def fetch_with_raise_for_status():
    async with aiohttp.ClientSession() as session:
        try:
            async with session.get("https://httpbin.io/anything", raise_for_status=True) as response:
                # No need to manually call response.raise_for_status(), it is automatic
                data = await response.text()
                print(data)
        except aiohttp.ClientResponseError as e:
            print(f"HTTP error occurred: {e.status} - {e.message}")
        except aiohttp.ClientError as e:
            print(f"Request error occurred: {e}")

# Run the event loop
asyncio.run(fetch_with_raise_for_status())

```

In this case, the `raise_for_status=True` argument is passed directly to the `session.get()` call. This ensures that an exception is raised automatically for any `4xx` or `5xx` status codes.

Option #3 example:

```none
import aiohttp
import asyncio

async def fetch_with_manual_error_handling():
    async with aiohttp.ClientSession() as session:
        try:
            async with session.get("https://httpbin.io/anything") as response:
                response.raise_for_status()  # Manually raises error for 4xx/5xx
                data = await response.text()
                print(data)
        except aiohttp.ClientResponseError as e:
            print(f"HTTP error occurred: {e.status} - {e.message}")
        except aiohttp.ClientError as e:
            print(f"Request error occurred: {e}")

# Run the event loop
asyncio.run(fetch_with_manual_error_handling())

```

If you prefer more control over individual requests, you can call `response.raise_for_status()` manually after making a request. This approach allows you to decide exactly when to handle errors.

### <a></a>Retry Failed Requests

AIOHTTP does not provide built-in support for retrying requests automatically. To implement that, you must use custom logic or a third-party library like [`aiohttp-retry`](https://github.com/inyutin/aiohttp_retry). This enables you to configure retry logic for failed requests, helping to handle transient network issues, timeouts, or rate limits.

Install [`aiohttp-retry`](https://pypi.org/project/aiohttp-retry/) with:

```none
pip install aiohttp-retry

```

Then, you can use it as follows:

```none
import asyncio
from aiohttp_retry import RetryClient, ExponentialRetry

async def main():
    retry_options = ExponentialRetry(attempts=1)
    retry_client = RetryClient(raise_for_status=False, retry_options=retry_options)
    async with retry_client.get("https://httpbin.io/anything") as response:
        print(response.status)

    await retry_client.close()

```

This configures retry behavior, with an exponential backoff strategy. Learn more in the [official docs](https://github.com/inyutin/aiohttp_retry?tab=readme-ov-file#documentation).

## <a></a>AIOHTTP vs Requests for Web Scraping

Below is a summary table to compare AIOHTTP and [Requests for web scraping](/blog/web-data/python-requests-guide):

**Feature****AIOHTTP****Requests****GitHub stars**15.3k52.4k**Client support**✔️✔️**Sync support**❌✔️**Async support**✔️❌**Server support**✔️❌**Connection pooling**✔️✔️**HTTP/2 support**❌❌**User-agent customization**✔️✔️**Proxy support**✔️✔️**Cookie handling**✔️✔️**Retry mechanism**Available only via a third-party libraryAvailable via `HTTPAdapter`s**Performance**HighMedium**Community support and popularity**MediumLargeFor a complete comparison, check out our blog post on [Requests vs HTTPX vs AIOHTTP](/blog/web-data/requests-vs-httpx-vs-aiohttp).

Learn how [scrape websites with HTTPX](/blog/web-data/web-scraping-with-httpx).

## <a></a>Conclusion

In this article, you learned how to use the `aiohttp` library for web scraping. You explored what it is, the features it offers, and the benefits it provides. AIOHTTP stands out as a fast and reliable choice for making HTTP requests when gathering online data.

However, automated HTTP requests expose your public IP address. That can reveal your identity and location, putting your privacy at risk. To safeguard your security and privacy, one of the most effective strategies is to use a proxy server to [hide your IP address](/blog/how-tos/five-ways-to-hide-your-ip-address).

Bright Data controls the best proxy servers in the world, serving Fortune 500 companies and 50,000+ customers. Its offer includes a wide range of proxy types:

- [Datacenter proxies](/proxy-types/datacenter-proxies) – 1,300,000+ datacenter IPs.
- [Residential proxies](/proxy-types/residential-proxies) – 400M+ residential IPs in more than 195 countries.
- [ISP proxies](/proxy-types/isp-proxies) – 1,300,000+ ISP IPs.

Create a free Bright Data account today to test our proxies and scraping solutions!



Contact usStart free trial

No credit card required











 [ ](https://www.linkedin.com/in/antonello-zanini/)

Antonello Zanini

 Technical Writer



  5.5 years experience



Antonello Zanini is a technical writer, editor, and software engineer with 5M+ views. Expert in technical content strategy, web development, and project management.



Expertise

  Web Development   Web Scraping   AI Integration



 [ View all articles ](https://brightdata.com/blog/authors/antonello-zanini)











 Table of Contents







Dedicated Scraper APIs &amp; No-Code Scrapers

Over 1000 scrapers for all popular domains. Simplify your web scraping.

[See pricing](/pricing/web-scraper "See pricing")

Just want data? Skip scraping.

Hundreds of ready-to-use datasets from all popular domains.

[See pricing](/pricing/datasets "See pricing")







 [ ](https://news.ycombinator.com/submitlink?t=Asynchronous+Web+Scraping+With+AIOHTTP+in+Python&u=https://brightdata.com/blog/web-data/web-scraping-with-aiohttp) [ ](https://www.linkedin.com/shareArticle?mini=true&title=Asynchronous+Web+Scraping+With+AIOHTTP+in+Python&url=https://brightdata.com/blog/web-data/web-scraping-with-aiohttp) [ ](http://www.reddit.com/submit?title=Asynchronous+Web+Scraping+With+AIOHTTP+in+Python&url=https://brightdata.com/blog/web-data/web-scraping-with-aiohttp)







##  You might also be interested in

 [ ](https://brightdata.com/blog/ai/openhuman-with-bright-data "Production-Ready Web Access in OpenHuman Through the Bright Data CLI")

 [AI





Antonello Zanini

Technical Writer





### Production-Ready Web Access in OpenHuman Through the Bright Data CLI

Integrate Bright Data CLI with OpenHuman to enable production-ready web access and data collection for AI agents.



 09-Sep-2026

 12 min read

 ](https://brightdata.com/blog/ai/openhuman-with-bright-data)

 [ ](https://brightdata.com/blog/ai/minimax-m3-with-bright-data "Giving self-hosted MiniMax M3 agents live web access with Bright Data")

 [AI





Satyam Tripathi

Technical Writer





### Giving self-hosted MiniMax M3 agents live web access with Bright Data

Self-hosted MiniMax M3 agents get live web access using Bright Data’s search and scraping tools. Bypass blocks and CAPTCHAs.



 09-Sep-2026

 54 min read

 ](https://brightdata.com/blog/ai/minimax-m3-with-bright-data)

 [ ](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax "Multimodal Web Scraping with MiniMax")

 [Web Data





Antonello Zanini

Technical Writer





### Multimodal Web Scraping with MiniMax

Pair Bright Data Web Unlocker with MiniMax M3 vision to extract structured data from images and web page screenshots.



 09-Sep-2026

 4 min read

 ](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax)