---
title: "Managing Failed Requests in Python"
slug: retry-failed-requests-python
date: 2025-02-02T15:08:40+00:00
modified: 2025-09-16T16:37:43+00:00
permalink: https://brightdata.com/blog/web-data/retry-failed-requests-python
type: blog
---

[ Blog ](https://brightdata.com/blog "Blog") / [Web Data](https://brightdata.com/blog/web-data)







 [Web Data](https://brightdata.com/blog/web-data)

# Managing Failed Requests in Python

Learn to handle failed HTTP requests in Python with effective retry strategies and custom logic.

 8 min read





 [ ](https://brightdata.com/blog/authors/jake-nulty)

 [Jake Nulty

Technical Writer

 ](https://brightdata.com/blog/authors/jake-nulty)





 ![Managing Failed Requests in Python blog image](https://media.brightdata.com/2025/02/Managing-Failed-Requests-in-Python-blog-image.svg)





Whenever you’re dealing with HTTP, failed requests are an inevitable fact of reality that need to be dealt with. In web development, a status 200 indicates a *good response*. However, we don’t always get a 200, and this guide will help you understand how to handle these non-200 status codes.

According to [Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status), status codes can be broken down into the following categories:

- **100-199**: Informational Responses
- **200-299**: Successful Responses
- **300-399**: Redirection Messages
- **400-499**: Client Error Messages
- **500-599**: Server Error Messages

## <a></a>What Are Status Codes?

[Error codes](/blog/proxy-101/proxy-error-codes) are important. When building client side programs like web scrapers, we primarily need to focus on status codes in the 400+ and 500+ range. Codes in the 400s generally cover errors on the client side such as authentication issues, rate limiting, timeouts, and the infamous 404: File Not Found error. In the 500s, we’re generally looking at server issues.

For decades, Mozilla has been documenting web development standards from the [W3C](https://www.w3.org/) and [IETF](https://www.ietf.org/). Below is a list of common error codes you might encounter. This list is non-exhaustive. These errors come from Mozilla’s [official documentation](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses). Depending on your target site, your codes might differ slightly but the logic should remain the same.

**Status Code****Meaning****Description**400Bad RequestCheck your request format401UnauthorizedCheck your API key403ForbiddenYou cannot access this data404Not FoundSite/Endpoint doesn’t exist408Request TimeoutRequest timed out, try again429Too Many RequestsSlow down your requests500Internal Server ErrorGeneric server error, retry request501Not ImplementedServer doesn’t support this yet502Bad GatewayFailed response from an upstream server503Service UnavailableServer is temporarily down, retry later504Gateway TimeoutTimed out waiting for an upstream server## <a></a>Retry Strategies

When implementing a retry mechanism, you can use pre-built libraries such as HTTPAdapter and Tenacity. Depending on your case, you might even want to write your own retry logic.

Typically, we want a retry limit and a strategy for backing off. We need a limit so we don’t get caught in an infinite loop of retries. We need back off little by little in order to respect the host server. When you’re requests come too fast, they get you blocked, or they overwhelm the server.

- **Retry Limits**: You need to set a limit. After X amount of retries, your scraper will give up.
- **Backoff Algorithm**: This one is relatively simple. You want to start with a small back off and increase it with each retry. We want to start with 0.3, then increase to 0.6, and 1.2 and so on and so forth.

**We want to retry our requests up to a certain limit. After each failed request, we want to wait a little more time.**

## <a></a>HTTPAdapter

With HTTPAdapter, we need to configure three things: `total`, `backoff_factor`, and `status_forcelist`. `allowed_methods` isn’t really a requirement, but it does make our code safer by helping define our retry conditions. In the code below, we use [httpbin](https://httpbin.org/) to automatically force an error and trigger our retry logic.

```none
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

#create a session
session = requests.Session()

#configure retry settings
retry = Retry(
    total=3,                  #maximum retries
    backoff_factor=0.3,       #time between retries (exponential backoff)
    status_forcelist=(429, 500, 502, 503, 504), #status codes to trigger a retry
    allowed_methods={"GET", "POST"}
)

#mount the adapter with our custom settings
adapter = HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)

#actually make a request with our retry logic
try:
    print("Making a request with retry logic...")
    response = session.get("https://httpbin.org/status/500")
    response.raise_for_status()
    print("✅ Request successful:", response.status_code)
except requests.exceptions.RequestException as e:
    print("❌ Request failed after retries:", e)

```

Once we’ve created a `Session` object, we do the following:

- Create a `Retry` object and define the following:
    - `total`: The maximum limit for retrying a request.
    - `backoff_factor`: Time to wait between retries. This adjusts exponentially as our retries increase.
    - `status_forcelist`: A list of bad status codes. Any codes in this list will automatically trigger a retry.
- Create an `HTTPAdapter` object with our `retry` variable: `adapter = HTTPAdapter(max_retries=retry)`.
- Once we’ve created the `adapter`, we mount it to the HTTP and HTTPS methods using `session.mount()`.

When you run this code, our three retries (`total=3`) will run and then you’ll get the following output.

```none
Making a request with retry logic...
❌ Request failed after retries: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /status/500 (Caused by ResponseError('too many 500 error responses'))
```

## <a></a>Tenacity

You can also use [Tenacity](https://tenacity.readthedocs.io/en/latest/), a popular open source retry library for Python. It’s not limited to HTTP, but it gives us an expressive, understandable way to implement retries.

First, you need to install it.

```none
pip install tenacity

```

Once installed, we create a *decorator* and use it to wrap a requests function. With our `@retry` decorator, we add the `stop`, `wait`, and `retry` arguments.

```none
import requests
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type, RetryError

#define a retry strategy
@retry(
    stop=stop_after_attempt(3),  #retry up to 3 times
    wait=wait_exponential(multiplier=0.3),  #exponential backoff
    retry=retry_if_exception_type(requests.exceptions.RequestException),  #retry on request failures
)

def make_request():
    print("Making a request with retry logic...")
    response = requests.get("https://httpbin.org/status/500")
    response.raise_for_status()
    print("✅ Request successful:", response.status_code)
    return response

# Attempt to make the request
try:
    make_request()
except RetryError as e:
    print("❌ Request failed after all retries:", e)

```

The logic and settings here are very similar to our first example with HTTPAdapter.

- `stop=stop_after_attempt(3)`: This tells `tenacity` to give up after 3 failed retries.
- `wait=wait_exponential(multiplier=0.3)` uses the same wait that we used before. It also backs off exponentially, just like before.
- `retry=retry_if_exception_type(requests.exceptions.RequestException)` tells `tenacity` to use this logic every time a `RequestException` occurs.
- `make_request()` makes a request to our error endpoint. It receives all of the traits from the decorator we created above it.

When you run this code, you get a similar output.

```none
Making a request with retry logic...
Making a request with retry logic...
Making a request with retry logic...
❌ Request failed after all retries: RetryError[<Future at 0x75e762970760 state=finished raised HTTPError>]
```

## <a></a>Build Your Own Retry Mechanism

You can also build your own retry mechanism. When dealing with custom code, this can often be the best approach. With a relatively small amount of code, we can achieve the same effect that we get from these libraries.

In the code below, we need to import `sleep` for our exponential backoff. We once again set our configuration: `total`, `backoff_factor` and `bad_codes`. We then use a `while` loop to hold our retry logic. `while` we still have tries and we haven’t succeeded, we attempt the request.

```none
import requests
from time import sleep

#create a session
session = requests.Session()

#define our retry settings
total = 3
backoff_factor = 0.3
bad_codes = [429, 500, 502, 503, 504]

#try counter and success boolean
current_tries = 0
success = False

#attempt until we succeed or run out of tries
while current_tries < total and not success:
    try:
        print("Making a request with retry logic...")
        response = session.get("https://httpbin.org/status/500")
        if response.status_code in bad_codes:
            raise requests.exceptions.HTTPError(f"Received {response.status_code}, triggering retry")
        print("✅ Request successful:", response.status_code)
        success = True
    except requests.exceptions.RequestException as e:
        print(f"❌ Request failed: {e}, retries left: {total-current_tries}")
        sleep(backoff_factor)
        backoff_factor = backoff_factor * 2
        current_tries+=1
```

The actual logic here is handled by a simple `while` loop.

- If `response.status_code` is in our list of `bad_codes`, we throw an exception.
- If a request fails, we:
    - Print an error message to the console.
    - `sleep(backoff_factor)` waits before sending the next request.
    - `backoff_factor = backoff_factor * 2` doubles our `backoff_factor` for the next try.
    - We increment `current_tries` so we don’t stay in the loop indefinitely.

Here’s the output from our custom retry logic.

```none
Making a request with retry logic...
❌ Request failed: Received 500, triggering retry, retries left: 3
Making a request with retry logic...
❌ Request failed: Received 500, triggering retry, retries left: 2
Making a request with retry logic...
❌ Request failed: Received 500, triggering retry, retries left: 1
```

## <a></a>Getting Past Blocks

In the wild, some sites are going to block you. It’s best practice to always [use a proxy with Python requests](/blog/proxy-101/proxy-with-python-requests). With a proxy, your request gets routed through a different machine. This will protect your identity and prevent your IP address from getting blocked by your target site. We even have a [detailed guide](/blog/proxy-101/how-to-bypass-an-ip-ban) on getting past IP blocks. Our [residential proxies](/proxy-types/residential-proxies) are built to get you past these challenges.

## <a></a>Conclusion

Now you know how to handle failed HTTP requests in Python. Whether you’re writing a scraper, an API client, or automation tools, you know how to handle these issues. To avoid all kinds of failed requests, we’ve developed products like the [Web Unlocker API](/products/web-unlocker) and [Scraping Browser](/products/scraping-browser). These tools automatically handle anti-bot measures, [CAPTCHA challenges](/blog/web-data/what-is-a-captcha), and IP blocks, ensuring seamless and efficient web scraping for even the most challenging websites.

Sign up now and start your free trial today.



Contact usStart free trial

No credit card required











 [ ](https://www.linkedin.com/in/jacob-nulty-682803187/)

Jake Nulty

 Technical Writer



  6 years experience



Jacob Nulty is a Detroit-based software developer and technical writer exploring AI and human philosophy, with experience in Python, Rust, and blockchain.



Expertise

  Data Structures   Python   Rust



 [ View all articles ](https://brightdata.com/blog/authors/jake-nulty)











 Table of Contents







Dedicated Scraper APIs &amp; No-Code Scrapers

Over 1000 scrapers for all popular domains. Simplify your web scraping.

[See pricing](/pricing/web-scraper "See pricing")

Just want data? Skip scraping.

Hundreds of ready-to-use datasets from all popular domains.

[See pricing](/pricing/datasets "See pricing")







 [ ](https://news.ycombinator.com/submitlink?t=Managing+Failed+Requests+in+Python&u=https://brightdata.com/blog/web-data/retry-failed-requests-python) [ ](https://www.linkedin.com/shareArticle?mini=true&title=Managing+Failed+Requests+in+Python&url=https://brightdata.com/blog/web-data/retry-failed-requests-python) [ ](http://www.reddit.com/submit?title=Managing+Failed+Requests+in+Python&url=https://brightdata.com/blog/web-data/retry-failed-requests-python)







##  You might also be interested in

 [ ](https://brightdata.com/blog/ai/openhuman-with-bright-data "Production-Ready Web Access in OpenHuman Through the Bright Data CLI")

 [AI





Antonello Zanini

Technical Writer





### Production-Ready Web Access in OpenHuman Through the Bright Data CLI

Integrate Bright Data CLI with OpenHuman to enable production-ready web access and data collection for AI agents.



 09-Sep-2026

 12 min read

 ](https://brightdata.com/blog/ai/openhuman-with-bright-data)

 [ ](https://brightdata.com/blog/ai/minimax-m3-with-bright-data "Giving self-hosted MiniMax M3 agents live web access with Bright Data")

 [AI





Satyam Tripathi

Technical Writer





### Giving self-hosted MiniMax M3 agents live web access with Bright Data

Self-hosted MiniMax M3 agents get live web access using Bright Data’s search and scraping tools. Bypass blocks and CAPTCHAs.



 09-Sep-2026

 54 min read

 ](https://brightdata.com/blog/ai/minimax-m3-with-bright-data)

 [ ](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax "Multimodal Web Scraping with MiniMax")

 [Web Data





Antonello Zanini

Technical Writer





### Multimodal Web Scraping with MiniMax

Pair Bright Data Web Unlocker with MiniMax M3 vision to extract structured data from images and web page screenshots.



 09-Sep-2026

 4 min read

 ](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax)