Scrapy Proxy Integration
This guide may be outdated. For an up-to-date guide please see our documentation.
What is Scrapy?
Scrapy is a Python framework for web crawling and scraping, which allows users to extract structured data from websites. It is open-source, fast, and extensible. Scrapy can be used for various purposes, such as data mining, monitoring, and automated testing.
Scrapy integration with Bright Data proxies
Open your preferred IDE and start a new scrapy project, type in the command line :
scrapy startproject <project_name>
This will create a new folder with the project name, within the folder open a python file.
- Go to your Bright Data Control Panel and clicking the ‘Proxies & Scraping Infra’ icon
- Create a new proxy zone by clicking ‘Add’, choosing a network type, configuring the proxy, and clicking save
- Under your proxy-zone’s ‘Access parameters’ tab, you will find your ‘USERNAME’ and ‘PASSWORD’ values.
- In your scrapy spider code file, within the request’s meta parameter set the ‘proxy’ value to be the following, using the ‘USERNAME’ and ‘PASSWORD’ values from before: “http://USERNAME:[email protected]:33335”
- For Example:
import scrapy
class BrightdatascrapyexampleSpider(scrapy.Spider):
name = "BrightDataScrapyExample"
def start_requests(self):
request = scrapy.Request(url="http://example.com",callback=self.parse)
request.meta['proxy'] = "http://USERNAME:[email protected]:33335"
yield request
def parse(self, response):
print(response.body)
Then run the following command in your command line :
scrapy runspider <Pythonfilename.py>
How To Use Bright Data Proxy Manger With Scrapy
- Create a proxy zone same as in the direct integration above
- Install the Proxy Manager
- Click ‘add new port’ and configure it for your use case
- In your Scrapy spider code file, within the request’s meta parameter set the ‘proxy’ value to be the following: “http://IP:PORTNUMBER”
- The local host IP is 127.0.0.1 – this is the value you need to use if the proxy manager is installed on your machine. If the proxy manager is installed on an external server, input that server’s IP address
- The port created in the Proxy Manager is 24XXX, for example, 24000 – the default first port number
- For example:
import scrapy
class BrightdatascrapyexampleSpider(scrapy.Spider):
name = "BrightDataScrapyExample"
def start_requests(self):
request = scrapy.Request(url="http://example.com",callback=self.parse)
request.meta['proxy'] = "http://127.0.0.1:24000"
yield request
def parse(self, response):
print(response.body)
⚠️Important note: If you are using Bright Data’s Residential Proxies, Web Unlocker or SERP API, you need to install an SSL certificate to enable end-to-end secure connections to your target website(s). This is a simple process, see https://docs.brightdata.com/general/account/ssl-certificate#installation-of-the-ssl-certificate for instructions.
Get proxies for Scrapy
Powered by an award-winning proxy infrastructure
Over 400 million residential IPs, best-in-class technology and the ability to target any country, city, ZIP Code, carrier, & ASN make our premium proxy services a top choice for developers.
A proxy for every development path
Mix-and-match networks, peers, and IPs to optimize your continuous flow of web data.
Proxy Network Pricing
Using AWS? You can now pay through the AWS Marketplace The best customer experience in the industry
New feature releases every day
To answer any questions right when you need it
Real-time network performance dashboard
To optimize your performance
To meet your data collection goals
The category leader in proxies and data collection
650TB of public data collected every day
New feature releases every day
Serving 7/10 of the world’s leading universities
