Web Crawler
Crawl websites and extract public data from the most popular websites. With over 1500 crawlers, collecting web data has never been easier.
- Retrieve real-time data via API
- Pay only for successfully delivered results
- Bulk request handling, up to 5K URLs
- Retrieve results in multiple formats
Trusted by 50,000+ customers worldwide
Web Crawler API Library
Web Crawler Pricing
Only pay for what’s successfully delivered. No hidden fees, no charges for failed deliveries.
Every plan gives you full access - pay less per record as you scale
- Automated proxy management
- Full browser rendering
- CAPTCHA solving
- Unlimited concurrency
- Batch & scheduled collection
- Job management APIs
- Data validation & discovery
- Data parsing (JSON or CSV)
- Webhook or API delivery
CODE EXAMPLES
Dedicated endpoints for 800+ domains.
curl -H "Authorization: Bearer API_TOKEN" -H "Content-Type: application/json" -d '[{"url":"https://www.linkedin.com/in/elad-moshe-05a90413/"},{"url":"https://www.linkedin.com/in/jonathan-myrvik-3baa01109"},{"url":"https://www.linkedin.com/in/aviv-tal-75b81/"},{"url":"https://www.linkedin.com/in/bulentakar/"},{"url":"https://www.linkedin.com/in/nnikolaev/"}]' "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json&uncompressed_webhook=true"
[
{
"db_source": "1789608697830",
"timestamp": "2026-09-17",
"id": "lil***-ot***o-1*********",
"name": "Lilian O****o",
"city": "Kenya",
"country_code": "KE",
"position": "Marketting Executive at Standard Chartered Bank",
"about": null
},
{
"db_source": "1789608697830",
"timestamp": "2026-09-17",
"id": "die***ham***0ba******",
"name": "Diep P**m",
"city": "Greater Sydney Area",
"country_code": "AU",
"position": "Acting Process Leader at Australia Post",
"about": null
},
{
"db_source": "1789608697830",
"timestamp": "2026-09-17",
"id": "夏-空***171***8",
"name": "******",
"city": "Guangzhou, Guangdong, China",
"country_code": "CN",
"position": "Renaissance Technologies LLC",
"about": null
},
{
"db_source": "1789608697830",
"timestamp": "2026-09-17",
"id": "nur***ami***che*********b61***",
"name": "Nurul A***a *** ***",
"city": "Pahang, Malaysia",
"country_code": "MY",
"position": "Assistant It Manager at Pusat Internet 1Malaysia",
"about": null
},
{
"db_source": "1789608697830",
"timestamp": "2026-09-17",
"id": "olg***eat***-ma*********b4a******",
"name": "Olga B*****z M*****a",
"city": "Buenos Aires, Buenos Aires Province, Argentina",
"country_code": "AR",
"position": "--",
"about": null
}
]
curl -H "Authorization: Bearer API_TOKEN" -H "Content-Type: application/json" -d '[{"url":"https://www.amazon.com/Quencher-FlowState-Stainless-Insulated-Smoothie/dp/B0CRMZHDG8","asin":"B0CRMZHDG8","origin_url":"https://www.amazon.com/Quencher-FlowState-Stainless-Insulated-Smoothie/dp/B0CRMZHDG8","zipcode":"94107","language":""},{"url":"https://www.amazon.com/KitchenAid-Protective-Dishwasher-Stainless-8-72-Inch/dp/B07PZF3QS3","asin":"B07PZF3QS3","zipcode":"","language":""},{"url":"https://www.amazon.com/TruSkin-Naturals-Vitamin-Topical-Hyaluronic/dp/B01M4MCUAF","asin":"","origin_url":"https://www.amazon.com/TruSkin-Naturals-Vitamin-Topical-Hyaluronic/dp/B01M4MCUAF","zipcode":"94124","language":""}]' "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l7q7dkf244hwjntr0&format=json&uncompressed_webhook=true"
[
{
"timestamp": "2026-09-17",
"title": "PEUTIER 3D Carved Tooth Fairy Box, Wood Tooth Storage Box for Lost Teeth",
"seller_name": "PAR***ON",
"brand": "PEUTIER",
"description": "About this item Package List: You will receive 1 piece of tooth fairy box, suitable for boys and girls who are changing ...",
"initial_price": 6.99,
"currency": "USD",
"availability": "In Stock"
},
{
"timestamp": "2026-09-17",
"title": "Cradle Cap Brush for Babies, Set of 5 Baby Bath Essentials with Bath Sponge Silicone Scrubber Long Handle Comb for Newbo...",
"seller_name": "******",
"brand": "YESYBC",
"description": "Gentle \u0026 Safe Grooming: This kit includes two bath sponges and bristle brushes, two silicone cradle cap brushes, and a f...",
"initial_price": 6.99,
"currency": "USD",
"availability": "In Stock"
},
{
"timestamp": "2026-09-17",
"title": "URMYWO Baby Tissue Box Toy - Montessori Baby Toys for 6-12 Months 1 Year Old Boy Girl - Soft Crinkle Sensory Toys for In...",
"seller_name": "******",
"brand": "URMYWO",
"description": "polyster Fun Montessori Toys: Babies and young children are naturally curious and inquisitive about the world around the...",
"initial_price": 9.99,
"currency": "USD",
"availability": "In Stock"
},
{
"timestamp": "2026-09-17",
"title": "Sesame Street Elmo’s Sing \u0026 Learn Playphone",
"seller_name": "Ama***.co***",
"brand": "Sesame Street",
"description": "Get ready for giggles and engaging conversation with the Sesame Street Sing and Learn Playphone. This pretend play phone...",
"initial_price": 36.57,
"currency": "USD",
"availability": "In Stock"
},
{
"timestamp": "2026-09-17",
"title": "Mioglrie Toddler Baby Girl Fall Winter Clothes Long Sleeve Sweatsuit 3-24M",
"seller_name": "Baby M******e",
"brand": "Mioglrie",
"description": "Baby girl clothes is soft, comfortable and breathable, ensuring that your baby stays cozy all day long. Toddler girls sw...",
"initial_price": 24.98,
"currency": "USD",
"availability": "In Stock"
}
]
curl -H "Authorization: Bearer API_TOKEN" -H "Content-Type: application/json" -d '[{"url":"https://www.zillow.com/homedetails/2506-Gordon-Cir-South-Bend-IN-46635/77050198_zpid/?t=for_sale"}]' "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_lfqkr8wm13ixtbd8f5&format=json&uncompressed_webhook=true"
[
{
"db_source": "1789357753648",
"timestamp": "2026-09-14",
"zpid": 89387956,
"city": "Canadensis",
"state": "PA",
"homeStatus": "OTHER",
"address:city": "Canadensis",
"address:streetAddress": "189 Mozzette Rd"
},
{
"db_source": "1789357753648",
"timestamp": "2026-09-14",
"zpid": 60807724,
"city": "Green Bay",
"state": "WI",
"homeStatus": "OTHER",
"address:city": "Green Bay",
"address:streetAddress": "401 13th Ave"
},
{
"db_source": "1789357753648",
"timestamp": "2026-09-14",
"zpid": 68012101,
"city": "Clearfield",
"state": "UT",
"homeStatus": "OTHER",
"address:city": "Clearfield",
"address:streetAddress": "59 Villa Dr"
},
{
"db_source": "1789357753648",
"timestamp": "2026-09-14",
"zpid": 441176909,
"city": "Wellington",
"state": "FL",
"homeStatus": "FOR_SALE",
"address:city": "Wellington",
"address:streetAddress": "11122 Polo Park Trail"
},
{
"db_source": "1789357753648",
"timestamp": "2026-09-14",
"zpid": 2062951799,
"city": "Jersey City",
"state": "NJ",
"homeStatus": "OTHER",
"address:city": "Jersey City",
"address:streetAddress": "93 Bayview Ave #2E"
}
]
curl -H "Authorization: Bearer API_TOKEN" -H "Content-Type: application/json" -d '[{"url":"https://www.instagram.com/p/Cuf4s0MNqNr","country":""},{"url":"https://www.instagram.com/p/DP861NijuwE","country":""}]' "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_lk5ns7kz21pck8jpis&format=json&uncompressed_webhook=true"
[
{
"db_source": "1789600973253",
"timestamp": "2026-09-16",
"url": "https:\/\/www.instagram.com\/p\/DbOCJ7VCITL\/",
"user_posted": "hodgetwins",
"description": "New reporting has linked Sahra Noor, the sister of Rep. Ilhan Omar, to Minnesota’s massive Feeding Our Future fraud sche...",
"hashtags": null,
"num_comments": 379,
"date_posted": "2026-07-25T14:30:34.000Z"
},
{
"db_source": "1789600973253",
"timestamp": "2026-09-16",
"url": "https:\/\/www.instagram.com\/p\/DdEWN0PFqZ8\/",
"user_posted": "abbie_hathcock",
"description": "We’re getting married!!!! \n\nThe Lord is so faithful and His timing is perfect! I can’t wait to spend all my days with my...",
"hashtags": null,
"num_comments": 237,
"date_posted": "2026-09-09T13:16:13.000Z"
},
{
"db_source": "1789600973253",
"timestamp": "2026-09-16",
"url": "https:\/\/www.instagram.com\/reel\/CrrbuVAsQmF\/",
"user_posted": "_reddickk",
"description": "Every day, in every way, I’m getting better and better.",
"hashtags": null,
"num_comments": 8,
"date_posted": "2023-04-30T23:26:56.000Z"
},
{
"db_source": "1789600973253",
"timestamp": "2026-09-16",
"url": "https:\/\/www.instagram.com\/reel\/DdXIZJLlV5D\/",
"user_posted": "kienyke",
"description": "Un perro se convirtió en protagonista de un intento de robo ocurrido en el barrio Primero de Mayo, en la localidad de Sa...",
"hashtags": [
"#Robo",
"#SanCristóbal",
"#Perro"
],
"num_comments": 3,
"date_posted": "2026-09-16T20:21:39.000Z"
},
{
"db_source": "1789600973253",
"timestamp": "2026-09-16",
"url": "https:\/\/www.instagram.com\/reel\/Dcv8c2sDmu9\/",
"user_posted": "iam_bnotehunned",
"description": "SELF LOVE #wordsoftheday",
"hashtags": [
"#wordsoftheday"
],
"num_comments": 1,
"date_posted": "2026-09-01T15:06:19.000Z"
}
]
Easy to start. Easier to scale.
Unmatched Stability
Ensure consistent performance and minimize failures by relying on the world’s leading proxy infrastructure.
Simplified Web Scraping
Put your scraping on auto-pilot using production-ready APIs, saving resources and reducing maintenance.
Unlimited Scalability
Effortlessly scale your scraping projects to meet data demands, maintaining optimal performance.
Never worry about proxies and CAPTCHAs again
- Automatic IP Rotation
- CAPTCHA Solver
- User Agent Rotation
- Custom Headers
- JavaScript Rendering
- Residential Proxies
API for Seamless Web Crawler Data Access
Comprehensive, Scalable, and Compliant Web Data Extraction
API for Seamless Web Crawler Data Access
Tailored to your workflow
Get structured data in JSON, NDJSON, or CSV files through Webhook or API delivery.
Built-in infrastructure and unblocking
Get maximum control and flexibility without maintaining proxy and unblocking infrastructure. Easily scrape data from any geo-location while avoiding CAPTCHAs and blocks.
Battle-proven infrastructure
Bright Data’s platform powers 50,000+ companies worldwide, offering peace of mind with 99.99% uptime, access to 400M+ real user IPs covering 195 countries.
Industry leading compliance
Our privacy practices comply with data protection laws, including the EU data protection regulatory framework, GDPR, and CCPA.