---
title: "How to Make Web Scraping Faster: A Complete Guide"
slug: speed-up-web-scraping
date: 2024-11-17T11:31:40+00:00
modified: 2025-09-16T16:39:02+00:00
permalink: https://brightdata.com/blog/web-data/speed-up-web-scraping
type: blog
---
[ Blog ](https://brightdata.com/blog "Blog") / [Web Data](https://brightdata.com/blog/web-data)
[Web Data](https://brightdata.com/blog/web-data)
# How to Make Web Scraping Faster: A Complete Guide
Learn how to optimize your web scraping process with advanced techniques for faster data retrieval.
23 min read
[ ](https://brightdata.com/blog/authors/antonello-zanini)
[Antonello Zanini
Technical Writer
](https://brightdata.com/blog/authors/antonello-zanini)
In this article, you will see:
- The primary causes of a slow web scraping process
- Various techniques to speed up web scraping
- How to optimize a sample Python scraping script for faster data retrieval
Let’s dive in!
## Reasons Why Your Scraping Process Is Slow
Explore the key reasons why your web scraping process may be slow.
### Reason #1: Slow Server Responses
One of the most notable factors affecting your web scraping speed is the server response time. When you send a request to a website, the server will process and respond to your request. If the server is slow, your requests will take longer to complete. Reasons for a server to be slow are heavy traffic, limited resources, or network slowdowns.
Unfortunately, there is little you can do to speed up a target server. That is beyond your control, unless the slowdown is due to an overwhelming number of requests from your end. If this is the case, spread your requests over a longer duration by adding random delays between them.
### Reason #2: Slow CPU Processing
CPU processing speed plays a crucial role in how quickly your scraping scripts can operate. When you run your scripts sequentially, your CPU is tasked with processing each operation one at a time, which can be time-consuming. This is particularly noticeable when your scripts involve complex computations or data transformations.
Additionally, HTML parsing takes some time and can significantly slow down your scraping process. Learn more in our article on [HTML web scraping](/blog/how-tos/html-web-scraping).
### Reason #3: Limited I/O Operations
Input/Output (I/O) operations can easily become the bottleneck of your scraping operation. That is especially true when your target site consists of several pages. If your script is designed to wait for responses from external resources before proceeding, that can lead to considerable delays.
Sending a request, waiting for the server to respond, processing it, and then moving on to the next request is not an efficient way to perform web scraping.
### Other reasons
Other reasons that make your web scraping script slow are:
- **Inefficient code**: Poorly optimized scraping logic can make the entire scraping process slow. Avoid inefficient data structures, unnecessary loops, or excessive logging.
- **Rate limiting**: If the target site restricts the number of requests a user can make in a specified time frame, your automated scraper will be slow as a result. The solution? [Proxy services!](/proxy-types)
- **CAPTCHAs and other anti-scraping solutions**: CAPTCHAs and anti-bot measures can interrupt your scraping process by requiring user interaction. Discover other [anti-scraping techniques](/blog/web-data/anti-scraping-techniques).
## Techniques to Speed Up Web Scraping
In this section, you will discover the most popular methods to speed up web scraping. We will start with a basic Python scraping script and demonstrate the impact of various optimizations on it.
**Note**: The techniques explored here work with any programming language or technology. Python is used just for simplicity and because it is one of the [best programming languages for web scraping](/blog/web-data/best-languages-web-scraping).
This is the initial Python scraping script:
```none
import requests
from bs4 import BeautifulSoup
import csv
import time
def scrape_quotes_to_scrape():
# array of the with the URLs of the page to scrape
urls = [
"http://quotes.toscrape.com/",
"https://quotes.toscrape.com/page/2/",
"https://quotes.toscrape.com/page/3/",
"https://quotes.toscrape.com/page/4/",
"https://quotes.toscrape.com/page/5/",
"https://quotes.toscrape.com/page/6/",
"https://quotes.toscrape.com/page/7/",
"https://quotes.toscrape.com/page/8/",
"https://quotes.toscrape.com/page/9/",
"https://quotes.toscrape.com/page/10/"
]
# where to store the scraped data
quotes = []
# scrape the pages sequentially
for url in urls:
print(f"Scraping page: '{url}'")
# send a GET request to get the page HTML
response = requests.get(url)
# parse the page HTML using BeautifulSoup
soup = BeautifulSoup(response.content, "html.parser")
# select all quote elements on the page
quote_html_elements = soup.select(".quote")
# iterate over the quote elements and scrape their content
for quote_html_element in quote_html_elements:
# extract the text of the quote
text = quote_html_element.select_one(".text").get_text()
# extract the author of the quote
author = quote_html_element.select_one(".author").get_text()
# extract tags associated with the quote
tags = [tag.get_text() for tag in quote_html_element.select(".tag")]
# populate a new quote object and add it to the list
quote = {
"text": text,
"author": author,
"tags": ", ".join(tags)
}
quotes.append(quote)
print(f"Page '{url}' scraped successfullyn")
print("Exporting scraped data to CSV")
# export the scraped quotes to a CSV file
with open("quotes.csv", "w", newline="", encoding="utf-8") as csvfile:
fieldnames = ["text", "author", "tags"]
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(quotes)
print("Quotes exported to CSVn")
# measure execution time
start_time = time.time()
scrape_quotes_to_scrape()
end_time = time.time()
execution_time = end_time - start_time
print(f"Execution time: {execution_time:.2f} seconds")
```
The above scraper targets 10 paginated URLs from the [Quotes to Scrape](https://quotes.toscrape.com/) website. For each URL, the script performs the following operations:
1. Sends a GET request using [`requests`](https://requests.readthedocs.io/en/latest/) to fetch the page’s HTML.
2. Parses the HTML content with [`BeautifulSoup`](https://www.crummy.com/software/BeautifulSoup/bs4/doc/).
3. Extracts the quote text, author, and tags for each quote element on the page.
4. Stores the scraped data in a list of dictionaries.
Finally, it exports the scraped data to a CSV file named `quotes.csv`.
To run the script, install the required libraries with:
```none
pip install requests beautifulsoup4
```
The `scrape_quotes_to_scrape()` function call is wrapped with [`time.time()`](https://docs.python.org/3/library/time.html#time.time) calls to measure how long the scraping process takes. On our machine, the initial script takes approximately **4.51 seconds to complete**.
Running the script generates a `quotes.csv` file in your project folder. Additionally, you will see logs similar to the following:
```none
Scraping page: 'http://quotes.toscrape.com/'
Page 'http://quotes.toscrape.com/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/2/'
Page 'https://quotes.toscrape.com/page/2/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/3/'
Page 'https://quotes.toscrape.com/page/3/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/4/'
Page 'https://quotes.toscrape.com/page/4/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/5/'
Page 'https://quotes.toscrape.com/page/5/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/6/'
Page 'https://quotes.toscrape.com/page/6/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/7/'
Page 'https://quotes.toscrape.com/page/7/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/8/'
Page 'https://quotes.toscrape.com/page/8/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/9/'
Page 'https://quotes.toscrape.com/page/9/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/10/'
Page 'https://quotes.toscrape.com/page/10/' scraped successfully
Exporting scraped data to CSV
Quotes exported to CSV
Execution time: 4.63 seconds
```
This output clearly shows that the script sequentially scrapes each paginated webpage from *Quotes to Scrape*. As you are about to see, some optimizations will significantly change the flow and speed of this process.
Now, let’s find out how to make web scraping faster!
### 1. Use a Faster HTML Parsing Library
[Data parsing](/blog/web-data/what-is-data-parsing) consumes time and resources, and different HTML parsers use various approaches to accomplish this task. Some focus on providing a rich feature set with a self-describing API, while others prioritize performance. For more details, check out our guide on the [best HTML parsers](/blog/web-data/best-html-parsers).
In Python, Beautiful Soup is the most popular HTML parser, but it is not necessarily the fastest. [See some benchmarks for more context](https://thehftguy.com/2020/07/28/making-beautifulsoup-parsing-10-times-faster/).
In reality, Beautiful Soup acts just as a wrapper around different underlying parsers. You can specify the parser you want to use when initializing it, via the second argument:
```none
soup = BeautifulSoup(response.content, "html.parser")
```
Generally, Beautiful Soup is used in combination with [`html.parser`](https://docs.python.org/3/library/html.parser.html), the built-in parser from Python’s standard library. However, if you are looking for speed, you should consider [`lxml`](https://lxml.de/). This is one of the fastest HTML parsers available in Python, as it is based on a C implementation.
To install `lxml`, run the following command:
```none
pip install lxml
```
Once installed, you can use it with Beautiful Soup like this:
```none
soup = BeautifulSoup(response.content, "lxml")
```
Now, run your Python scraping script again. This time, you should see the following output:
```none
# omitted for brevity...
Execution time: 4.35 seconds
```
The execution time dropped **from 4.61 seconds to 4.35 seconds**. While this change might seem small, the impact of this optimization greatly depends on the size and complexity of the HTML pages being parsed and how many elements are being selected.
In this example, the target site has pages with a simple, short, and shallow DOM structure. Still, achieving a speed improvement of around 6% with just a small code change is a worthwhile gain!
**👍 Pros**:
- Easy to implement in Beautiful Soup
**👎 Cons**:
- Small advantage
- Only works on pages with complex DOM structures
- Faster HTML parsers may have a more complex API
### 2. Implement Multiprocessing Scraping
[Multiprocessing](https://www.techtarget.com/searchdatacenter/definition/multiprocessing) is an approach to parallel execution where a program spawns multiple processes. Each of these processes operates in parallel and independently on a CPU core to perform tasks simultaneously rather than sequentially.
This method is particularly beneficial for I/O-bound operations like web scraping. The reason is that the primary bottleneck is often the time spent waiting for responses from web servers. By utilizing multiple processes, you can send requests to several pages at the same time, reducing overall scraping time.
To adapt your scraping script for multiprocessing, you need to make some important modifications to the execution logic. Follow the steps below to transform your Python scraper from a sequential to a multiprocessing approach.
To get started with multiprocessing in Python, the first step is to import `Pool` and `cpu_count` from [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) module:
```none
from multiprocessing import Pool, cpu_count
```
`Pool` provides what you need to manage a pool of worker processes. Instead, `cpu_count` helps you determine the number of CPU cores available for parallel processing.
Next, isolate the logic to scrape a single URL within a function:
```none
def scrape_page(url):
print(f"Scraping page: '{url}'")
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")
quote_html_elements = soup.select(".quote")
quotes = []
for quote_html_element in quote_html_elements:
text = quote_html_element.select_one(".text").get_text()
author = quote_html_element.select_one(".author").get_text()
tags = [tag.get_text() for tag in quote_html_element.select(".tag")]
quotes.append({
"text": text,
"author": author,
"tags": ", ".join(tags)
})
print(f"Page '{url}' scraped successfullyn")
return quotes
```
The above function will be called by each worker process and executed on a CPU core at a time.
Then, replace the sequential scraping flow with a multiprocessing logic:
```none
def scrape_quotes():
urls = [
"http://quotes.toscrape.com/",
"https://quotes.toscrape.com/page/2/",
"https://quotes.toscrape.com/page/3/",
"https://quotes.toscrape.com/page/4/",
"https://quotes.toscrape.com/page/5/",
"https://quotes.toscrape.com/page/6/",
"https://quotes.toscrape.com/page/7/",
"https://quotes.toscrape.com/page/8/",
"https://quotes.toscrape.com/page/9/",
"https://quotes.toscrape.com/page/10/"
]
# create a pool of processes
with Pool(processes=cpu_count()) as pool:
results = pool.map(scrape_page, urls)
# flatten the results list
quotes = [quote for sublist in results for quote in sublist]
print("Exporting scraped data to CSV")
with open("quotes_multiprocessing.csv", "w", newline="", encoding="utf-8") as csvfile:
fieldnames = ["text", "author", "tags"]
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(quotes)
print("Quotes exported to CSVn")
```
Finally, run the `scrape_quotes()` function while measuring the execution time:
```none
if __name__ == "__main__":
start_time = time.time()
scrape_quotes()
end_time = time.time()
execution_time = end_time - start_time
print(f"Execution time: {execution_time:.2f} seconds")
```
Note that the `if __name__ == "__main__":` construct is required to prevent certain parts of your code from being executed when the module is imported. Without this check, the multiprocessing module may attempt to spawn new processes that can lead to unexpected behavior, especially on Windows.
Put it all together, and you will get:
```none
from multiprocessing import Pool, cpu_count
import requests
from bs4 import BeautifulSoup
import csv
import time
def scrape_page(url):
print(f"Scraping page: '{url}'")
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")
quote_html_elements = soup.select(".quote")
quotes = []
for quote_html_element in quote_html_elements:
text = quote_html_element.select_one(".text").get_text()
author = quote_html_element.select_one(".author").get_text()
tags = [tag.get_text() for tag in quote_html_element.select(".tag")]
quotes.append({
"text": text,
"author": author,
"tags": ", ".join(tags)
})
print(f"Page '{url}' scraped successfullyn")
return quotes
def scrape_quotes():
urls = [
"http://quotes.toscrape.com/",
"https://quotes.toscrape.com/page/2/",
"https://quotes.toscrape.com/page/3/",
"https://quotes.toscrape.com/page/4/",
"https://quotes.toscrape.com/page/5/",
"https://quotes.toscrape.com/page/6/",
"https://quotes.toscrape.com/page/7/",
"https://quotes.toscrape.com/page/8/",
"https://quotes.toscrape.com/page/9/",
"https://quotes.toscrape.com/page/10/"
]
# create a pool of processes
with Pool(processes=cpu_count()) as pool:
results = pool.map(scrape_page, urls)
# flatten the results list
quotes = [quote for sublist in results for quote in sublist]
print("Exporting scraped data to CSV")
with open("quotes_multiprocessing.csv", "w", newline="", encoding="utf-8") as csvfile:
fieldnames = ["text", "author", "tags"]
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(quotes)
print("Quotes exported to CSVn")
if __name__ == "__main__":
start_time = time.time()
scrape_quotes()
end_time = time.time()
execution_time = end_time - start_time
print(f"Execution time: {execution_time:.2f} seconds")
```
Execute the script again. This time, it will produce some logs as follows:
```none
Scraping page: 'http://quotes.toscrape.com/'
Scraping page: 'https://quotes.toscrape.com/page/2/'
Scraping page: 'https://quotes.toscrape.com/page/3/'
Scraping page: 'https://quotes.toscrape.com/page/4/'
Scraping page: 'https://quotes.toscrape.com/page/5/'
Scraping page: 'https://quotes.toscrape.com/page/6/'
Scraping page: 'https://quotes.toscrape.com/page/7/'
Scraping page: 'https://quotes.toscrape.com/page/8/'
Page 'http://quotes.toscrape.com/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/9/'
Page 'https://quotes.toscrape.com/page/3/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/10/'
Page 'https://quotes.toscrape.com/page/4/' scraped successfully
Page 'https://quotes.toscrape.com/page/5/' scraped successfully
Page 'https://quotes.toscrape.com/page/6/' scraped successfully
Page 'https://quotes.toscrape.com/page/7/' scraped successfully
Page 'https://quotes.toscrape.com/page/2/' scraped successfully
Page 'https://quotes.toscrape.com/page/8/' scraped successfully
Page 'https://quotes.toscrape.com/page/9/' scraped successfully
Page 'https://quotes.toscrape.com/page/10/' scraped successfully
Exporting scraped data to CSV
Quotes exported to CSV
Execution time: 1.87 seconds
```
As you can see, the execution order is no longer sequential. Your script can now scrape several pages simultaneously. Specifically, it can scrape up to the number of cores available on your CPU (8, in our case).
Parallel processing results in an around 145% time improvement, **reducing the execution time from 4.61 seconds to 1.87 seconds**. That is impressive!
**👍 Pros**:
- Great execution time enchanment
- Natively supported by most programming languages
**👎 Cons**:
- Limited by the number of cores available on your machine
- Does not respect the order of the URLs in the list
- Requires a lot of changes in the code
### 3. Implement Multithreading Scraping
[Multithreading](https://www.techtarget.com/whatis/definition/multithreading) is a programming technique to run multiple threads concurrently within a single process. This enables your script to perform multiple tasks simultaneously, with each task handled by a dedicated thread.
While similar to multiprocessing, multithreading does not necessarily require multiple CPU cores. This is because a single CPU core can run numerous threads simultaneously, sharing the same memory space. Dig into this concept in our guide on [concurrency vs parallelism](/blog/web-data/concurrency-vs-parallelism).
Keep in mind that transforming a scraping script from a sequential approach to a multithreaded one requires changes similar to those described in the previous chapter.
In this implementation, we will use `ThreadPoolExecutor` from the Python [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) module. You can import it as below:
```none
from concurrent.futures import ThreadPoolExecutor
```
`ThreadPoolExecutor` provides a high-level interface for managing a pool of threads, running them concurrently for you.
As before, start by isolating the logic for scraping a single URL into a function, just like we did in the previous chapter. The key difference is that you now need to utilize [`ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor) to run the function in multiple threads:
```none
quotes = []
# create a thread pool with up to 10 workers
with ThreadPoolExecutor(max_workers=10) as executor:
# use map to apply the scrape_page function to each URL
results = executor.map(scrape_page, urls)
# combine the results from all threads
for result in results:
quotes.extend(result)
```
By default, if `max_workers` is `None` or not specified, it will default to the number of processors on your machine, multiplied by 5. In this case, we have only 10 pages, so setting it to 10 will be fine. Do not forget that opening too many threads can slow down your system and may lead to performance drawbacks instead of enhancements.
The entire scraping script will contain the following code:
```none
from concurrent.futures import ThreadPoolExecutor
import requests
from bs4 import BeautifulSoup
import csv
import time
def scrape_page(url):
print(f"Scraping page: '{url}'")
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")
quote_html_elements = soup.select(".quote")
quotes = []
for quote_html_element in quote_html_elements:
text = quote_html_element.select_one(".text").get_text()
author = quote_html_element.select_one(".author").get_text()
tags = [tag.get_text() for tag in quote_html_element.select(".tag")]
quotes.append({
"text": text,
"author": author,
"tags": ", ".join(tags)
})
print(f"Page '{url}' scraped successfullyn")
return quotes
def scrape_quotes():
urls = [
"http://quotes.toscrape.com/",
"https://quotes.toscrape.com/page/2/",
"https://quotes.toscrape.com/page/3/",
"https://quotes.toscrape.com/page/4/",
"https://quotes.toscrape.com/page/5/",
"https://quotes.toscrape.com/page/6/",
"https://quotes.toscrape.com/page/7/",
"https://quotes.toscrape.com/page/8/",
"https://quotes.toscrape.com/page/9/",
"https://quotes.toscrape.com/page/10/"
]
# where to store the scraped data
quotes = []
# create a thread pool with up to 10 workers
with ThreadPoolExecutor(max_workers=10) as executor:
# use map to apply the scrape_page function to each URL
results = executor.map(scrape_page, urls)
# combine the results from all threads
for result in results:
quotes.extend(result)
print("Exporting scraped data to CSV")
with open("quotes_multiprocessing.csv", "w", newline="", encoding="utf-8") as csvfile:
fieldnames = ["text", "author", "tags"]
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(quotes)
print("Quotes exported to CSVn")
if __name__ == "__main__":
start_time = time.time()
scrape_quotes()
end_time = time.time()
execution_time = end_time - start_time
print(f"Execution time: {execution_time:.2f} seconds")
```
Launch it, and it will log messages as below:
```none
Scraping page: 'http://quotes.toscrape.com/'
Scraping page: 'https://quotes.toscrape.com/page/2/'
Scraping page: 'https://quotes.toscrape.com/page/3/'
Scraping page: 'https://quotes.toscrape.com/page/4/'
Scraping page: 'https://quotes.toscrape.com/page/5/'
Scraping page: 'https://quotes.toscrape.com/page/6/'
Scraping page: 'https://quotes.toscrape.com/page/7/'
Scraping page: 'https://quotes.toscrape.com/page/8/'
Scraping page: 'https://quotes.toscrape.com/page/9/'
Scraping page: 'https://quotes.toscrape.com/page/10/'
Page 'http://quotes.toscrape.com/' scraped successfully
Page 'https://quotes.toscrape.com/page/6/' scraped successfully
Page 'https://quotes.toscrape.com/page/7/' scraped successfully
Page 'https://quotes.toscrape.com/page/10/' scraped successfully
Page 'https://quotes.toscrape.com/page/8/' scraped successfully
Page 'https://quotes.toscrape.com/page/5/' scraped successfully
Page 'https://quotes.toscrape.com/page/9/' scraped successfully
Page 'https://quotes.toscrape.com/page/4/' scraped successfully
Page 'https://quotes.toscrape.com/page/3/' scraped successfully
Page 'https://quotes.toscrape.com/page/2/' scraped successfully
Exporting scraped data to CSV
Quotes exported to CSV
Execution time: 0.52 seconds
```
Similar to multiprocessing scraping, the execution order of the pages is no longer sequential. This time, the performance improvement is even greater than with multiprocessing. That is because the script can now execute 10 requests simultaneously, exceeding the previous limit of 8 requests (the number of CPU cores).
The time improvement is huge, **from 4.61 seconds to 0.52 seconds**, resulting in a percentage reduction of around 885%!
**👍 Pros**:
- Huge execution time improvement
- Natively supported by most technologies
**👎 Cons**:
- Finding the right number of threads is not easy
- Does not respect the order of the URLs in the list
- Requires a lot of changes in the code
### 4. Using Async/Await Scraping
[Asynchronous programming](https://realpython.com/async-io-python/) is a modern programming paradigm that enables you to write non-blocking code. The idea is to give developers the ability to deal with concurrent operations without explicitly managing multithreading or multiprocessing.
In a traditional synchronous approach, each operation must terminate before the next one begins. This can lead to inefficiencies, especially in I/O-bound tasks like web scraping. With async programming, you can initiate multiple I/O operations simultaneously and then wait for them to complete. That keeps your script responsive and efficient.
In Python, asynchronous scraping is generally implemented using the [`asyncio`](https://docs.python.org/3/library/asyncio.html) module from the standard library. This package provides the infrastructure for writing single-threaded concurrent code using [coroutines](https://docs.python.org/3/library/asyncio-task.html), via the `async` and `await` keywords.
However, standard HTTP libraries like `requests` do not support asynchronous operations. Thus, you need to use an asynchronous HTTP client like [AIOHTTP](https://docs.aiohttp.org/en/stable/), which is specifically designed to work seamlessly with `asyncio`. This combination helps you send multiple HTTP requests concurrently without blocking your script’s execution.
Install AIOHTTP using the following command:
```none
pip install aiohttp
```
Then, import `asyncio` and `aiohttp`:
```none
import asyncio
import aiohttp
```
Just as in previous chapters, encapsulate the logic for scraping a single URL into a function. However, this time, the function will be asynchronous:
```none
async def scrape_url(session, url):
async with session.get(url) as response:
print(f"Scraping page: '{url}'")
html_content = await response.text()
soup = BeautifulSoup(html_content, "html.parser")
# scraping logic...
```
Note the use of the `await` function to retrieve the HTML of the webpage.
To execute the function in parallel, create an AIOHTTP session and gather multiple scraping tasks:
```none
# executing the scraping tasks concurrently
async with aiohttp.ClientSession() as session:
tasks = [scrape_url(session, url) for url in urls]
results = await asyncio.gather(*tasks)
# flatten the results list
quotes = [quote for sublist in results for quote in sublist]
```
Finally, use `asyncio.run()` to execute your asynchronous main scraping function:
```none
if __name__ == "__main__":
start_time = time.time()
asyncio.run(scrape_quotes())
end_time = time.time()
execution_time = end_time - start_time
print(f"Execution time: {execution_time:.2f} seconds")
```
Your async scraping script in Python will contain these lines of code:
```none
import asyncio
import aiohttp
from bs4 import BeautifulSoup
import csv
import time
async def scrape_url(session, url):
async with session.get(url) as response:
print(f"Scraping page: '{url}'")
html_content = await response.text()
soup = BeautifulSoup(html_content, "html.parser")
quote_html_elements = soup.select(".quote")
quotes = []
for quote_html_element in quote_html_elements:
text = quote_html_element.select_one(".text").get_text()
author = quote_html_element.select_one(".author").get_text()
tags = [tag.get_text() for tag in quote_html_element.select(".tag")]
quotes.append({
"text": text,
"author": author,
"tags": ", ".join(tags)
})
print(f"Page '{url}' scraped successfullyn")
return quotes
async def scrape_quotes():
urls = [
"http://quotes.toscrape.com/",
"https://quotes.toscrape.com/page/2/",
"https://quotes.toscrape.com/page/3/",
"https://quotes.toscrape.com/page/4/",
"https://quotes.toscrape.com/page/5/",
"https://quotes.toscrape.com/page/6/",
"https://quotes.toscrape.com/page/7/",
"https://quotes.toscrape.com/page/8/",
"https://quotes.toscrape.com/page/9/",
"https://quotes.toscrape.com/page/10/"
]
# executing the scraping tasks concurrently
async with aiohttp.ClientSession() as session:
tasks = [scrape_url(session, url) for url in urls]
results = await asyncio.gather(*tasks)
# flatten the results list
quotes = [quote for sublist in results for quote in sublist]
print("Exporting scraped data to CSV")
with open("quotes_multiprocessing.csv", "w", newline="", encoding="utf-8") as csvfile:
fieldnames = ["text", "author", "tags"]
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(quotes)
print("Quotes exported to CSVn")
if __name__ == "__main__":
start_time = time.time()
asyncio.run(scrape_quotes())
end_time = time.time()
execution_time = end_time - start_time
print(f"Execution time: {execution_time:.2f} seconds")
```
Launch it, and you will get an output like this:
```none
Scraping page: 'http://quotes.toscrape.com/'
Page 'http://quotes.toscrape.com/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/3/'
Scraping page: 'https://quotes.toscrape.com/page/7/'
Scraping page: 'https://quotes.toscrape.com/page/9/'
Scraping page: 'https://quotes.toscrape.com/page/6/'
Scraping page: 'https://quotes.toscrape.com/page/8/'
Scraping page: 'https://quotes.toscrape.com/page/10/'
Page 'https://quotes.toscrape.com/page/3/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/5/'
Scraping page: 'https://quotes.toscrape.com/page/4/'
Page 'https://quotes.toscrape.com/page/7/' scraped successfully
Page 'https://quotes.toscrape.com/page/9/' scraped successfully
Page 'https://quotes.toscrape.com/page/6/' scraped successfully
Scraping page: 'https://quotes.toscrape.com/page/2/'
Page 'https://quotes.toscrape.com/page/10/' scraped successfully
Page 'https://quotes.toscrape.com/page/5/' scraped successfully
Page 'https://quotes.toscrape.com/page/4/' scraped successfully
Page 'https://quotes.toscrape.com/page/8/' scraped successfully
Page 'https://quotes.toscrape.com/page/2/' scraped successfully
Exporting scraped data to CSV
Quotes exported to CSV
Execution time: 0.51 seconds
```
Note that the execution time is similar to the multithreading approach, but with the added benefit of not having to manually manage threads.
**👍 Pros**:
- Huge execution time gain
- Modern programming is based on async logic
- Does not require manual management of threads or processes
**👎 Cons**:
- Not so easy to master
- Does not respect the order of the URLs in the list
- Requires dedicted async libraries
### 5. Other Scraping Speeding Up Tips and Approaches
Other ways to make web scraping faster are:
- **Request rate optimization**: Fine-tune request intervals to find the optimal balance between speed and avoiding rate-limiting or getting banned.
- **Rotating Proxies**: Use rotating proxies to distribute requests across multiple IP addresses, reducing the chances of being blocked and enabling faster scraping. See the [best rotating proxies](/blog/proxy-101/best-rotating-proxies).
- **Parallel scraping with distributed systems**: Distribute scraping tasks across multiple online machines.
- **Reduce JavaScript rendering**: Try to avoid browser automation tools, preferring tools like HTTP clients as HTML parsers. Remember that browsers eat up a lot of resources and are way slower than most traditional HTML parsers.
## Conclusion
In this guide, we saw how to make web scraping faster. We uncovered the main reasons why a scraping script can be slow and examined various techniques to address these issues using a sample Python script. With just a few adjustments to the scraping logic, we achieved an 8x improvement in execution time.
While manual optimization of your web scraping logic is crucial for speeding up the data retrieval process, using the right tools is equally important. When targeting dynamic sites that require browser automation solutions, things can become more complicated, as browsers tend to be slow and resource-intensive.
To overcome these challenges, try [Scraping Browser](/products/scraping-browser), a fully hosted cloud-based solution designed for scraping. It integrates seamlessly with Puppeteer, Selenium, Playwright, and other popular browser automation tools. Equipped with a [CAPTCHA auto-solver](/products/web-unlocker/captcha-solver) and backed by a [proxy network of 150+ million residential IPs](/proxy-types/residential-proxies), it offers unlimited scalability for your covering any scraping need!
Sign up now and start your free trial.
Contact usStart free trial
No credit card required
[ ](https://www.linkedin.com/in/antonello-zanini/)
Antonello Zanini
Technical Writer
5.5 years experience
Antonello Zanini is a technical writer, editor, and software engineer with 5M+ views. Expert in technical content strategy, web development, and project management.
Expertise
Web Development Web Scraping AI Integration
[ View all articles ](https://brightdata.com/blog/authors/antonello-zanini)
Table of Contents
Dedicated Scraper APIs & No-Code Scrapers
Over 1000 scrapers for all popular domains. Simplify your web scraping.
[See pricing](/pricing/web-scraper "See pricing")
Just want data? Skip scraping.
Hundreds of ready-to-use datasets from all popular domains.
[See pricing](/pricing/datasets "See pricing")
[ ](https://news.ycombinator.com/submitlink?t=How+to+Make+Web+Scraping+Faster%3A+A+Complete+Guide&u=https://brightdata.com/blog/web-data/speed-up-web-scraping) [ ](https://www.linkedin.com/shareArticle?mini=true&title=How+to+Make+Web+Scraping+Faster%3A+A+Complete+Guide&url=https://brightdata.com/blog/web-data/speed-up-web-scraping) [ ](http://www.reddit.com/submit?title=How+to+Make+Web+Scraping+Faster%3A+A+Complete+Guide&url=https://brightdata.com/blog/web-data/speed-up-web-scraping)
## You might also be interested in
[ ](https://brightdata.com/blog/ai/openhuman-with-bright-data "Production-Ready Web Access in OpenHuman Through the Bright Data CLI")
[AI
Antonello Zanini
Technical Writer
### Production-Ready Web Access in OpenHuman Through the Bright Data CLI
Integrate Bright Data CLI with OpenHuman to enable production-ready web access and data collection for AI agents.
09-Sep-2026
12 min read
](https://brightdata.com/blog/ai/openhuman-with-bright-data)
[ ](https://brightdata.com/blog/ai/minimax-m3-with-bright-data "Giving self-hosted MiniMax M3 agents live web access with Bright Data")
[AI
Satyam Tripathi
Technical Writer
### Giving self-hosted MiniMax M3 agents live web access with Bright Data
Self-hosted MiniMax M3 agents get live web access using Bright Data’s search and scraping tools. Bypass blocks and CAPTCHAs.
09-Sep-2026
54 min read
](https://brightdata.com/blog/ai/minimax-m3-with-bright-data)
[ ](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax "Multimodal Web Scraping with MiniMax")
[Web Data
Antonello Zanini
Technical Writer
### Multimodal Web Scraping with MiniMax
Pair Bright Data Web Unlocker with MiniMax M3 vision to extract structured data from images and web page screenshots.
09-Sep-2026
4 min read
](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax)