---
title: "Web Scraping With HTTPX in Python"
slug: web-scraping-with-httpx
date: 2025-01-22T11:38:15+00:00
modified: 2026-08-24T12:56:47+00:00
permalink: https://brightdata.com/blog/web-data/web-scraping-with-httpx
type: blog
---
[ Blog ](https://brightdata.com/blog "Blog") / [Web Data](https://brightdata.com/blog/web-data)
[Web Data](https://brightdata.com/blog/web-data)
# Web Scraping With HTTPX in Python
Explore HTTPX, a powerful Python HTTP client for web scraping. Learn setup, features, advanced techniques, and how it compares with Requests.
13 min read
[ ](https://brightdata.com/blog/authors/antonello-zanini)
[Antonello Zanini
Technical Writer
](https://brightdata.com/blog/authors/antonello-zanini)
In this article, you will learn:
- What HTTP is and the features it offers
- How to use HTTPX for web scraping in a guided section
- Advanced HTTPX features for web scraping
- A comparison of HTTPX vs. Requests for automated requests
Let’s dive in!
## What Is HTTPX?
[HTTPX](https://github.com/projectdiscovery/httpx) is a fully featured HTTP client for Python 3, built on top of the [`retryablehttp`](https://github.com/projectdiscovery/retryablehttp-go) library. It is designed to ensure reliable results even with a high number of threads. HTTPX provides both synchronous and asynchronous APIs, with support for HTTP/1.1 and HTTP/2 protocols.
**⚙️ Features**
- Simple and modular codebase, making it easy to contribute.
- Fast and fully configurable flags for probing multiple elements.
- Supports various HTTP-based probing methods.
- Smart automatic fallback from HTTPS to HTTP by default.
- Accepts hosts, URLs, and CIDR as input.
- Supports proxies, custom HTTP headers, custom timeouts, basic authentication, and more.
**👍 Pros**
- Available from the command line using [`httpx[cli]`](https://pypi.org/project/httpx/).
- Packed with features, including support for HTTP/2 and an asynchronous API.
- This project is actively developed…
**👎 Cons**
- …with frequent updates that may introduce breaking changes with new releases.
- Less popular than the [`requests`](https://requests.readthedocs.io/en/latest/) library.
## Scraping with HTTPX: Step-By-Step Guide
HTTPX is an HTTP client, meaning it helps you retrieve the raw HTML content of a page. To then parse and extract data from the HTML, you will need an HTML parser like [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/).
Actually, HTTPX is not just any HTTP client, but one of the [best Python HTTP clients for web scraping](/blog/web-data/best-python-http-clients).
Follow this tutorial to learn how to use HTTPX for web scraping with BeautifulSoup!
**Warning**: While HTTPX is only used in the early stages of the process, we will walk you through a complete workflow. If you are interested in more advanced HTTPX scraping techniques, you can skip ahead to the next chapter after Step 3.
### Step #1: Project Setup
Make sure you have Python 3+ installed on your machine. Otherwise, [download it from the official site](https://www.python.org/downloads/) and follow the installation instructions.
Now, use the following command to create a directory for your HTTPX scraping project:
```none
mkdir httpx-scraper
```
Navigate into it and initialize a [virtual environment](https://docs.python.org/3/library/venv.html) inside it:
```none
cd httpx-scraper
python -m venv env
```
Open the project folder in your preferred Python IDE. [Visual Studio Code with the Python extension](https://code.visualstudio.com/docs/languages/python) or [PyCharm Community Edition](https://www.jetbrains.com/pycharm/download/#section=windows) will do.
Next, create a `scraper.py` file inside the project folder. Currently, `scraper.py` is an empty Python script but it will soon contain the scraping logic.
In your IDE’s terminal, activate the virtual environment. On Linux or macOS, run:
```none
./env/bin/activate
```
Equivalently, for Windows, fire:
```none
env/Scripts/activate
```
Amazing! You are now fully set up.
### Step #2: Install the Scraping Libraries
In an activated virtual environment, install HTTPX and BeautifulSoup with the following command:
```none
pip install httpx beautifulsoup4
```
This will add both the [`httpx`](https://pypi.org/project/httpx/) and [`beautifulsoup4`](https://pypi.org/project/beautifulsoup4/) to your project’s dependencies.
Import them into your `scraper.py` script:
```none
import httpx
from bs4 import BeautifulSoup
```
Great! You are ready to move on to the next step in your scraping workflow.
### Step #3: Retrieve the HTML of the Target Page
In this example, the target page will be the “[Quotes to Scrape](https://quotes.toscrape.com/)” site:
Use HTTPX to retrieve the HTML of the homepage with the `get()` method:
```none
# Make an HTTP GET request to the target page
response = httpx.get("http://quotes.toscrape.com")
```
Behind the scenes, HTTPX will make an HTTP GET request to the server, which will respond with the HTML of the page. You can access the HTML content using the `response.text` attribute:
```none
html = response.text
print(html)
```
This will print the raw HTML content of the page:
```none
Quotes to Scrape
```
Terrific! Time to parse this content and extract the data you need.
### Step #4: Parse the HTML
Feed the HTML content to the BeautifulSoup constructor to parse it:
```none
# Parse the HTML content using
BeautifulSoup soup = BeautifulSoup(html, "html.parser")
```
[`html.parser`](https://docs.python.org/3/library/html.parser.html) is the standard Python HTML parser that will be used to parse the content.
The `soup` variable now holds the parsed HTML and exposes the methods to extract the data you need.
HTTPX has done its job of retrieving the HTML, and now you are moving into the traditional data parsing phase with BeautifulSoup. For more information, refer to our tutorial on [BeautifulSoup web scraping](/blog/how-tos/beautiful-soup-web-scraping).
### Step #5: Scrape Data From It
You can scrape quotes data from the page with the following lines of code:
```none
# Where to store the scraped data
quotes = []
# Extract all quotes from the page
quote_elements = soup.find_all("div", class_="quote")
# Loop through quotes and extract text, author, and tags
for quote_element in quote_elements:
text = quote_element.find("span", class_="text").get_text().get_text().replace("“", "").replace("”", "")
author = quote_element.find("small", class_="author")
tags = [tag.get_text() for tag in quote_element.find_all("a", class_="tag")]
# Store the scraped data
quotes.append({
"text": text,
"author": author,
"tags": tags
})
```
This snippet defines a list named `quotes` to store the scraped data. It then selects all quote HTML elements and iterates over them to extract the quote text, author, and tags. Each extracted quote is stored as a dictionary within the `quotes` list, organizing the data for further use or export.
Yes! Scraping logic implemented.
### Step #6: Export the Scraped Data
Use the following logic to export the scraped data to a CSV file:
```none
# Specify the file name for export
with open("quotes.csv", mode="w", newline="", encoding="utf-8") as file:
writer = csv.DictWriter(file, fieldnames=["text", "author", "tags"])
# Write the header row
writer.writeheader()
# Write the scraped quotes data
writer.writerows(quotes)
```
This snippet opens a file named `quotes.csv` in write mode, defines column headers (`text`, `author`, `tags`), writes the headers to the file, and then writes each dictionary from the `quotes` list to the CSV file. The `csv.DictWriter` handles the formatting, making it easy to store structured data.
Do not forget to import `csv` from the Python Standard Library:
```none
import csv
```
### Step #7: Put It All Together
Your final HTTPX web scraping script will contain:
```none
import httpx
from bs4 import BeautifulSoup
import csv
# Make an HTTP GET request to the target page
response = httpx.get("http://quotes.toscrape.com")
# Access the HTML of the target page
html = response.text
# Parse the HTML content using BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
# Where to store the scraped data
quotes = []
# Extract all quotes from the page
quote_elements = soup.find_all("div", class_="quote")
# Loop through quotes and extract text, author, and tags
for quote_element in quote_elements:
text = quote_element.find("span", class_="text").get_text().replace("“", "").replace("”", "")
author = quote_element.find("small", class_="author").get_text()
tags = [tag.get_text() for tag in quote_element.find_all("a", class_="tag")]
# Store the scraped data
quotes.append({
"text": text,
"author": author,
"tags": tags
})
# Specify the file name for export
with open("quotes.csv", mode="w", newline="", encoding="utf-8") as file:
writer = csv.DictWriter(file, fieldnames=["text", "author", "tags"])
# Write the header row
writer.writeheader()
# Write the scraped quotes data
writer.writerows(quotes)
```
Execute it with:
```none
python scraper.py
```
Or, on Linux/macOS:
```none
python3 scraper.py
```
A `quotes.csv` file will appear in the root folder of your project. Open it and you will see:
Et voilà! You just learned how to perform web scraping with HTTPX and BeautifulSoup.
## HTTPX Web Scraping Advanced Features and Techniques
Now that you know how to use HTTPX for web scraping in a basic scenario, you are ready to see it in action with more complex use cases.
In the examples below, the target site will be the [HTTPBin.io `/anything` endpoint](https://httpbin.io/anything). This is a special API that returns the IP address, headers, and other information sent by the caller.
Master HTTPX for web scraping!
### Set Custom Headers
HTTPX allows you to [specify custom headers](https://www.python-httpx.org/quickstart/#custom-headers) thanks to the `headers` argument:
```none
import httpx
# Custom headers for the request
headers = {
"accept": "application/json",
"accept-language": "en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7,es-US;q=0.6,es;q=0.5,it-IT;q=0.4,it;q=0.3"
}
# Make a GET request with custom headers
response = httpx.get("https://httpbin.io/anything", headers=headers)
# Handle the response...
```
### Set a Custom User Agent
[`User-Agent`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent) is one of the most important [HTTP headers for web scraping](/blog/web-data/http-headers-for-web-scraping). By default, HTTPX uses the following `User-Agent`:
```none
python-httpx/
```
This value can easily reveal that your requests are automated, which could lead to blocking by the target site.
To avoid that, you can set a custom `User-Agent` to mimic a real browser, like so:
```none
import httpx
# Define a custom User-Agent
headers = {
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
}
# Make a GET request with the custom User-Agent
response = httpx.get("https://httpbin.io/anything", headers=headers)
# Handle the response...
```
Discover the [best user agents for web scraping](/blog/how-tos/user-agents-for-web-scraping-101)!
### Set Cookies
Just like HTTP headers, you can set cookies in HTTPX using the [`cookies` argument](https://www.python-httpx.org/quickstart/#cookies):
```none
import httpx
# Define cookies as a dictionary
cookies = {
"session_id": "3126hdsab161hdabg47adgb",
"user_preferences": "dark_mode=true"
}
# Make a GET request with custom cookies
response = httpx.get("https://httpbin.io/anything", cookies=cookies)
# Handle the response...
```
This gives you the ability to include session data required for your web scraping requests.
### Proxy Integration
You can [route your HTTPX requests through a proxy](https://www.python-httpx.org/advanced/proxies/) to protect your identity and avoid IP bans while performing web scraping. That is possible by using the `proxies` argument:
```none
import httpx
# Replace with the URL of your proxy server
proxy = ""
# Make a GET request through a proxy server
response = httpx.get("https://httpbin.io/anything", proxy=proxy)
# Handle the response...
```
Find out more in our guide on [how to use HTTPX with a proxy](/blog/proxy-101/httpx-with-proxies).
### Error Handling
By default, HTTPX raises errors only for [connection or network issues](https://www.python-httpx.org/exceptions/). To raise exceptions also for HTTP responses with [`4xx`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses) and [`5xx`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses) status codes,use the `raise_for_status()` method as below:
```none
import httpx
try:
response = httpx.get("https://httpbin.io/anything")
# Raise an exception for 4xx and 5xx responses
response.raise_for_status()
# Handle the response...
except httpx.HTTPStatusError as e:
# Handle HTTP status errors
print(f"HTTP error occurred: {e}")
except httpx.RequestError as e:
# Handle connection or network errors
print(f"Request error occurred: {e}")
```
### Session Handling
When using the top-level API in HTTPX, a new connection is established for every single request. In other words, TCP connections are not reused. As the number of requests to a host increases, that approach becomes inefficient.
In contrast, using a `httpx.Client` instance enables [HTTP connection pooling](https://devblogs.microsoft.com/premier-developer/the-art-of-http-connection-pooling-how-to-optimize-your-connections-for-peak-performance/). This means that multiple requests to the same host can reuse an existing TCP connection instead of creating a new one for each request.
The benefits of using a `Client` over the top-level API are:
- Reduced latency across requests (avoiding repeated handshaking)
- Lower CPU usage and fewer round-trips
- Reduced network congestion
Additionally, `Client` instances support session handling with features unavailable in the top-level API, including:
- Cookie persistence across requests.
- Applying configuration across all outgoing requests.
- Sending requests through HTTP proxies.
The recommended way to use a `Client` in HTTPX is with a context manager (`with` statement):
```none
import httpx
with httpx.Client() as client:
# Make an HTTP request using the client
response = client.get("https://httpbin.io/anything")
# Extract the JSON response data and print it
response_data = response.json()
print(response_data)
```
Alternatively, you can manually manage the client and close the connection pool explicitly with `client.close()`:
```none
import httpx
client = httpx.Client()
try:
# Make an HTTP request using the client
response = client.get("https://httpbin.io/anything")
# Extract the JSON response data and print it
response_data = response.json()
print(response_data)
except:
# Handle the error...
pass
finally:
# Close the client connections and release resources
client.close()
```
**Note**: If you are familiar with the `requests` library, `httpx.Client()` serves a similar purpose to [`requests.Session()`](https://requests.readthedocs.io/en/latest/user/advanced/#session-objects).
### Async API
By default, HTTPX exposes a standard synchronous API. At the same time, it also offers an [asynchronous client](https://www.python-httpx.org/async/) for cases where it is needed. If you are working with [`asyncio`](https://docs.python.org/3/library/asyncio.html), using an async client is essential for sending outgoing HTTP requests efficiently.
Asynchronous programming is a concurrency model that is significantly more efficient than multi-threading. It offers notable performance improvements and supports long-lived network connections like WebSockets. That makes it a key factor in [speeding up web scraping](/blog/web-data/speed-up-web-scraping).
To make asynchronous requests in HTTPX, you’ll need an `AsyncClient`. Initialize it and use it to make a GET request as shown below:
```none
import httpx
import asyncio
async def fetch_data():
async with httpx.AsyncClient() as client:
# Make an async HTTP request
response = await client.get("https://httpbin.io/anything")
# Extract the JSON response data and print it
response_data = response.json()
print(response_data)
# Run the async function
asyncio.run(fetch_data())
```
The [`with`](https://docs.python.org/3/reference/compound_stmts.html#with) statement ensures the client is automatically closed when the block ends. Alternatively, if you manage the client manually, you can close it explicitly with `await client.close()`.
Remember, all HTTPX request methods (`get()`, `post()`, etc.) are asynchronous when using an `AsyncClient`. Therefore, you must add `await` before calling them to get a response.
### Retry Failed Requests
Network instability during web scraping can lead to connection failures or timeouts. HTTPX simplifies handling such issues via its [`HTTPTransport`](https://www.python-httpx.org/advanced/transports/) interface. This mechanism retries requests when an `httpx.ConnectError` or `httpx.ConnectTimeout` occurs.
The following example demonstrates how to configure a transport to retry requests up to 3 times:
```none
import httpx
# Configure transport with retry capability on connection errors or timeouts
transport = httpx.HTTPTransport(retries=3)
# Use the transport with an HTTPX client
with httpx.Client(transport=transport) as client:
# Make a GET request
response = client.get("https://httpbin.io/anything")
# Handle the response...
```
Note that only connection-related errors trigger a retry. To handle read/write errors or specific HTTP status codes, you need to implement custom retry logic with libraries like [`tenacity`](https://tenacity.readthedocs.io/en/latest/).
## HTTPX vs Requests for Web Scraping
Here is a summary table to compare HTTPX and [Requests for web scraping](/blog/web-data/python-requests-guide):
**Feature****HTTPX****Requests****GitHub stars**8k52.4k**Async support**✔️❌**Connection pooling**✔️✔️**HTTP/2 support**✔️❌**User-agent customization**✔️✔️**Proxy support**✔️✔️**Cookie handling**✔️✔️**Timeouts**Customizable for connection and readCustomizable for connection and read**Retry mechanism**Available via transportsAvailable via `HTTPAdapter`s**Performance**HighMedium**Community support and popularity**GrowingLarge## Conclusion
In this article, you explored the `httpx` library for web scraping. You gained an understanding of what it is, what it offers, and its advantages. HTTPX is a fast and reliable option for making HTTP requests when collecting online data.
The problem is that automated HTTP requests reveal your public IP address, which can expose your identity and location. That compromises your privacy. To enhance your security and privacy, one of the most effective methods is to use a proxy server to [hide your IP address](/blog/how-tos/five-ways-to-hide-your-ip-address).
Bright Data controls the best proxy servers in the world, serving Fortune 500 companies and 50,000+ customers. Its offer includes a wide range of proxy types:
- [Datacenter proxies](/proxy-types/datacenter-proxies) – 1,300,000+ datacenter IPs.
- [Residential proxies](/proxy-types/residential-proxies) – 400M+ residential IPs in more than 195 countries.
- [ISP proxies](/proxy-types/isp-proxies) – 1,300,000+ ISP IPs.
Create a free Bright Data account today to test our scraping solutions and proxies!
Contact usStart free trial
No credit card required
[ ](https://www.linkedin.com/in/antonello-zanini/)
Antonello Zanini
Technical Writer
5.5 years experience
Antonello Zanini is a technical writer, editor, and software engineer with 5M+ views. Expert in technical content strategy, web development, and project management.
Expertise
Web Development Web Scraping AI Integration
[ View all articles ](https://brightdata.com/blog/authors/antonello-zanini)
Table of Contents
Dedicated Scraper APIs & No-Code Scrapers
Over 1000 scrapers for all popular domains. Simplify your web scraping.
[See pricing](/pricing/web-scraper "See pricing")
Just want data? Skip scraping.
Hundreds of ready-to-use datasets from all popular domains.
[See pricing](/pricing/datasets "See pricing")
[ ](https://news.ycombinator.com/submitlink?t=Web+Scraping+With+HTTPX+in+Python&u=https://brightdata.com/blog/web-data/web-scraping-with-httpx) [ ](https://www.linkedin.com/shareArticle?mini=true&title=Web+Scraping+With+HTTPX+in+Python&url=https://brightdata.com/blog/web-data/web-scraping-with-httpx) [ ](http://www.reddit.com/submit?title=Web+Scraping+With+HTTPX+in+Python&url=https://brightdata.com/blog/web-data/web-scraping-with-httpx)
## You might also be interested in
[ ](https://brightdata.com/blog/ai/openhuman-with-bright-data "Production-Ready Web Access in OpenHuman Through the Bright Data CLI")
[AI
Antonello Zanini
Technical Writer
### Production-Ready Web Access in OpenHuman Through the Bright Data CLI
Integrate Bright Data CLI with OpenHuman to enable production-ready web access and data collection for AI agents.
09-Sep-2026
12 min read
](https://brightdata.com/blog/ai/openhuman-with-bright-data)
[ ](https://brightdata.com/blog/ai/minimax-m3-with-bright-data "Giving self-hosted MiniMax M3 agents live web access with Bright Data")
[AI
Satyam Tripathi
Technical Writer
### Giving self-hosted MiniMax M3 agents live web access with Bright Data
Self-hosted MiniMax M3 agents get live web access using Bright Data’s search and scraping tools. Bypass blocks and CAPTCHAs.
09-Sep-2026
54 min read
](https://brightdata.com/blog/ai/minimax-m3-with-bright-data)
[ ](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax "Multimodal Web Scraping with MiniMax")
[Web Data
Antonello Zanini
Technical Writer
### Multimodal Web Scraping with MiniMax
Pair Bright Data Web Unlocker with MiniMax M3 vision to extract structured data from images and web page screenshots.
09-Sep-2026
4 min read
](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax)