Beautifulsoup Proxy Integration

Beautifulsoup proxy integration with Bright Data

Begin by going to your Bright Data Dashboard and clicking ‘create a Zone’.
Choose ‘Network type’ and click save.
Within Beautifulsoup fill in the ‘Proxy IP:Port’ in the ‘proxy-server’ value, for example brd.superproxy.io:33335.
Under ‘page.authenticate’ input your Bright Data account ID and proxy Zone name in the ‘username’ value, for example: brd-customer-CUSTOMER-zone-YOURZONE and your Zone password found in the Zone settings.
For example:

      from bs4 import BeautifulSoup

import requests

import json



proxies = {

   'http': 'http://brd-customer-[ACCOUNT_ID]-zone-[ZONE_NAME]:[ZONE_PASSWORD]@brd.superproxy.io:33335',

   'https': 'http://brd-customer-[ACCOUNT_ID]-zone-[ZONE_NAME]:[ZONE_PASSWORD]@brd.superproxy.io:33335',

}



url = "http://lumtest.com/myip.json"



res = requests.get(url, proxies=proxies, verify=False)

res.raise_for_status()



res_body = res.text



soup = BeautifulSoup(res_body, "html.parser")



html_content = str(soup)



data = {

   "IP": html_content,

  

}



print(json.dumps(data, indent=4))

Proxy Manager and Beautifulsop integration

Begin by going to your Bright Data Dashboard and clicking ‘create a Zone’.
Choose ‘Network type’ and click save.
Within Beautifulsoup fill in the ‘Proxy IP:Port’ in the ‘proxy-server’ value, for example brd.superproxy.io:33335.
Under ‘page.authenticate’ input your Bright Data account ID and proxy Zone name in the ‘username’ value, for example: brd-customer-CUSTOMER-zone-YOURZONE and your Zone password found in the Zone settings.
For example:

      from bs4 import BeautifulSoup

import requests

import json



proxies = {

   'http': '127.0.0.1:24000',

   'https': '127.0.0.1:24000',

}



url = "http://lumtest.com/myip.json"



res = requests.get(url, proxies=proxies, verify=False)

res.raise_for_status()



res_body = res.text



soup = BeautifulSoup(res_body, "html.parser")



html_content = str(soup)



data = {

   "IP": html_content,

  

}



print(json.dumps(data, indent=4))

⚠️warning:Important note: If you are using Bright Data’s Residential Proxies, Web Unlocker or SERP API, you need to install an SSL certificate to enable end-to-end secure connections to your target website(s). This is a simple process, see https://docs.brightdata.com/general/account/ssl-certificate#installation-of-the-ssl-certificate for instructions.

Get proxies for Beautifulsoup

Start free trial

Powered by an award-winning proxy infrastructure

Over 72 million residential IPs, best-in-class technology and the ability to target any country, city, ZIP Code, carrier, & ASN make our premium proxy services a top choice for developers.