Beautifulsoup Proxy Integration
This guide may be outdated. For an up-to-date guide please see our documentation.
Beautifulsoup proxy integration with Bright Data
- Begin by going to your Bright Data Dashboard and clicking ‘create a Zone’.
- Choose ‘Network type’ and click save.
- Within Beautifulsoup fill in the ‘Proxy IP:Port’ in the ‘proxy-server’ value, for example brd.superproxy.io:33335.
- Under ‘page.authenticate’ input your Bright Data account ID and proxy Zone name in the ‘username’ value, for example: brd-customer-CUSTOMER-zone-YOURZONE and your Zone password found in the Zone settings.
- For example:
from bs4 import BeautifulSoup
import requests
import json
proxies = {
'http': 'http://brd-customer-[ACCOUNT_ID]-zone-[ZONE_NAME]:[ZONE_PASSWORD]@brd.superproxy.io:33335',
'https': 'http://brd-customer-[ACCOUNT_ID]-zone-[ZONE_NAME]:[ZONE_PASSWORD]@brd.superproxy.io:33335',
}
url = "http://lumtest.com/myip.json"
res = requests.get(url, proxies=proxies, verify=False)
res.raise_for_status()
res_body = res.text
soup = BeautifulSoup(res_body, "html.parser")
html_content = str(soup)
data = {
"IP": html_content,
}
print(json.dumps(data, indent=4))
Proxy Manager and Beautifulsop integration
- Begin by going to your Bright Data Dashboard and clicking ‘create a Zone’.
- Choose ‘Network type’ and click save.
- Within Beautifulsoup fill in the ‘Proxy IP:Port’ in the ‘proxy-server’ value, for example brd.superproxy.io:33335.
- Under ‘page.authenticate’ input your Bright Data account ID and proxy Zone name in the ‘username’ value, for example: brd-customer-CUSTOMER-zone-YOURZONE and your Zone password found in the Zone settings.
- For example:
from bs4 import BeautifulSoup
import requests
import json
proxies = {
'http': '127.0.0.1:24000',
'https': '127.0.0.1:24000',
}
url = "http://lumtest.com/myip.json"
res = requests.get(url, proxies=proxies, verify=False)
res.raise_for_status()
res_body = res.text
soup = BeautifulSoup(res_body, "html.parser")
html_content = str(soup)
data = {
"IP": html_content,
}
print(json.dumps(data, indent=4))
⚠️warning:Important note: If you are using Bright Data’s Residential Proxies, Web Unlocker or SERP API, you need to install an SSL certificate to enable end-to-end secure connections to your target website(s). This is a simple process, see https://docs.brightdata.com/general/account/ssl-certificate#installation-of-the-ssl-certificate for instructions.
Get proxies for Beautifulsoup
Powered by an award-winning proxy infrastructure
Over 400 million residential IPs, best-in-class technology and the ability to target any country, city, ZIP Code, carrier, & ASN make our premium proxy services a top choice for developers.
A proxy for every development path
Mix-and-match networks, peers, and IPs to optimize your continuous flow of web data.
Proxy Network Pricing
Using AWS? You can now pay through the AWS Marketplace The best customer experience in the industry
New feature releases every day
To answer any questions right when you need it
Real-time network performance dashboard
To optimize your performance
To meet your data collection goals
The category leader in proxies and data collection
650TB of public data collected every day
New feature releases every day
Serving 7/10 of the world’s leading universities
