Web Crawler
Crawl websites and extract public data from the most popular websites. With over 1500 crawlers, collecting web data has never been easier.
1892 scrapers
- Retrieve real-time data via API
- Pay only for successfully delivered results
- Bulk request handling, up to 5K URLs
- Retrieve results in multiple formats
Trusted by 50,000+ customers worldwide
Web Crawler API
Web Crawler API Library
Remove the need to develop and maintain the infrastructure. Simply extract high volume web data, and ensure scalability and reliability using our Web Scraper API.
131.9K+
15.3K+
Web Crawler Pricing
Only pay for what’s successfully delivered. No hidden fees, no charges for failed deliveries.
We accept these payment methods:
Every plan gives you full access - pay less per record as you scale
Data Collection
- Automated proxy management
- Full browser rendering
- CAPTCHA solving
Performance at Scale
- Unlimited concurrency
- Batch & scheduled collection
- Job management APIs
Data Delivery
- Data validation & discovery
- Data parsing (JSON or CSV)
- Webhook or API delivery
CODE EXAMPLES
Dedicated endpoints for 800+ domains.
Input
JSON
curl -H "Authorization: Bearer API_TOKEN" -H "Content-Type: application/json" -d '[{"url":"https://www.linkedin.com/in/elad-moshe-05a90413/"},{"url":"https://www.linkedin.com/in/jonathan-myrvik-3baa01109"},{"url":"https://www.linkedin.com/in/aviv-tal-75b81/"},{"url":"https://www.linkedin.com/in/bulentakar/"},{"url":"https://www.linkedin.com/in/nnikolaev/"}]' "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l1viktl72bvl7bjuj0&format=json&uncompressed_webhook=true"
Output
JSON
[
{
"timestamp": "2026-10-22",
"id": "mar***la-***p-2*********",
"name": "Marcella R**p",
"city": "Coal, Pennsylvania, United States",
"country_code": "US",
"position": null,
"about": null,
"posts": null
},
{
"timestamp": "2026-10-22",
"id": "gar***ure***b11******",
"name": "Gary T***k",
"city": "Rock Hill, South Carolina, United States",
"country_code": "US",
"position": null,
"about": null,
"posts": null
},
{
"timestamp": "2026-10-22",
"id": "pau***erg***n-b*********",
"name": "Paul B******n",
"city": "Harrellsville, North Carolina, United States",
"country_code": "US",
"position": null,
"about": null,
"posts": null
},
{
"timestamp": "2026-10-22",
"id": "tod***ppe***8b1******",
"name": "Todd A***l",
"city": "Glenwood, Minnesota, United States",
"country_code": "US",
"position": null,
"about": null,
"posts": null
},
{
"timestamp": "2026-10-22",
"id": "jam***sco***615*********",
"name": "Jamie S***t",
"city": "United States",
"country_code": "US",
"position": null,
"about": "Jam***Sco*** BS*********N *********",
"posts": null
}
]
Input
JSON
curl -H "Authorization: Bearer API_TOKEN" -H "Content-Type: application/json" -d '[{"url":"https://www.amazon.com/Quencher-FlowState-Stainless-Insulated-Smoothie/dp/B0CRMZHDG8","asin":"B0CRMZHDG8","origin_url":"https://www.amazon.com/Quencher-FlowState-Stainless-Insulated-Smoothie/dp/B0CRMZHDG8","zipcode":"94107","language":""},{"url":"https://www.amazon.com/KitchenAid-Protective-Dishwasher-Stainless-8-72-Inch/dp/B07PZF3QS3","asin":"B07PZF3QS3","zipcode":"","language":""},{"url":"https://www.amazon.com/TruSkin-Naturals-Vitamin-Topical-Hyaluronic/dp/B01M4MCUAF","asin":"","origin_url":"https://www.amazon.com/TruSkin-Naturals-Vitamin-Topical-Hyaluronic/dp/B01M4MCUAF","zipcode":"94124","language":""}]' "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_l7q7dkf244hwjntr0&format=json&uncompressed_webhook=true"
Output
JSON
[
{
"timestamp": "2026-10-21",
"title": "Tree Hut Palm Grove Foaming Body Wash, 18 Fl Oz",
"seller_name": "Ama***.co***",
"brand": "Tree Hut",
"description": "Start your shower with the luxurious lather of Tree Hut Foaming Gel Wash in the relaxing, earthy and long-lasting scent ...",
"initial_price": 7.34,
"currency": "USD",
"availability": "Only 10 left in stock - order soon."
},
{
"timestamp": "2026-10-21",
"title": "Tree Hut Jelly Bear Foaming Body Wash, 18 Fl Oz",
"seller_name": "Ama***.co***",
"brand": "Tree Hut",
"description": "Elevate your shower routine with Tree Hut Jelly Bear Foaming Gel Body Wash. This rich gel transforms into a creamy foam ...",
"initial_price": 8.94,
"currency": "USD",
"availability": "In Stock"
},
{
"timestamp": "2026-10-21",
"title": "Tree Hut Tangerine Foaming Body Wash, 18 Fl Oz",
"seller_name": "Ama***.co***",
"brand": "Tree Hut",
"description": "Start your shower with the luxurious lather of Tree Hut Foaming Gel Wash in the bright, long-lasting scent of Tangerine....",
"initial_price": 11.99,
"currency": "USD",
"availability": "In Stock"
},
{
"timestamp": "2026-10-21",
"title": "Tree Hut Lotus Water Foaming Body Wash, 18 Fl Oz",
"seller_name": "Ama***.co***",
"brand": "Tree Hut",
"description": "Start your shower with the luxurious lather of Tree Hut Foaming Gel Wash in the fresh, long-lasting scent of Lotus Water...",
"initial_price": 8.94,
"currency": "USD",
"availability": "In Stock"
},
{
"timestamp": "2026-10-21",
"title": "Tree Hut Golden Vanilla Foaming Body Wash, 18 Fl Oz",
"seller_name": "Ama***.co***",
"brand": "Tree Hut",
"description": "Elevate your shower routine with Tree Hut Golden Vanilla Foaming Gel Body Wash. This rich gel transforms into a creamy f...",
"initial_price": 8.94,
"currency": "USD",
"availability": "In Stock"
}
]
Input
JSON
curl -H "Authorization: Bearer API_TOKEN" -H "Content-Type: application/json" -d '[{"url":"https://www.zillow.com/homedetails/2506-Gordon-Cir-South-Bend-IN-46635/77050198_zpid/?t=for_sale"}]' "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_lfqkr8wm13ixtbd8f5&format=json&uncompressed_webhook=true"
Output
JSON
[
{
"timestamp": "2026-10-22",
"zpid": 250203948,
"city": "Nacogdoches",
"state": "TX",
"homeStatus": "OTHER",
"address:city": "Nacogdoches",
"address:streetAddress": "N Ritchie St",
"address:zipcode": "75963"
},
{
"timestamp": "2026-10-22",
"zpid": 443184609,
"city": "Orangeville",
"state": "ON",
"homeStatus": "OTHER",
"address:city": "Orangeville",
"address:streetAddress": "200 Broadway #607",
"address:zipcode": "L9W5G3"
},
{
"timestamp": "2026-10-22",
"zpid": 90212473,
"city": "North Charleston",
"state": "SC",
"homeStatus": "OTHER",
"address:city": "North Charleston",
"address:streetAddress": "8685 Grassy Oak Trl",
"address:zipcode": "29420"
},
{
"timestamp": "2026-10-22",
"zpid": 1050148,
"city": "Bessemer",
"state": "AL",
"homeStatus": "OTHER",
"address:city": "Bessemer",
"address:streetAddress": "604 Castlewood Dr",
"address:zipcode": "35020"
},
{
"timestamp": "2026-10-22",
"zpid": 43291503,
"city": "Hollywood",
"state": "FL",
"homeStatus": "OTHER",
"address:city": "Hollywood",
"address:streetAddress": "800 S Hollybrook Dr APT 301",
"address:zipcode": "33025"
}
]
Input
JSON
curl -H "Authorization: Bearer API_TOKEN" -H "Content-Type: application/json" -d '[{"url":"https://www.instagram.com/p/Cuf4s0MNqNr","country":""},{"url":"https://www.instagram.com/p/DP861NijuwE","country":""}]' "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_lk5ns7kz21pck8jpis&format=json&uncompressed_webhook=true"
Output
JSON
[
{
"timestamp": "2026-10-22",
"url": "https:\/\/www.instagram.com\/reel\/Dcmugb7PC5k\/",
"user_posted": "irreelbis_jiyugaoka",
"description": "おはようございます。\n秋を感じる朝です。\nここ最近、原点回帰といいますか、やっぱり伝統的なワインが更に美味しく感じます。\n流行り廃りを乗り越え伝統と革新を繰り返し愛され続ける理由はあります。\nイレールビス自由が丘のワインは、厳選したワインを...",
"hashtags": [
"#イレールビス自由が丘",
"#自由が丘ランチ",
"#自由が丘ワイン",
"#自由が丘ディナー",
"#自由が丘レストラン"
],
"num_comments": 0,
"date_posted": "2026-08-29T01:20:33.000Z",
"likes": 18
},
{
"timestamp": "2026-10-22",
"url": "https:\/\/www.instagram.com\/reel\/DZlwExdyf71\/",
"user_posted": "sushigyosaitoto",
"description": "渋谷・幡ヶ谷にある「鮨 魚菜 魚魚」で、落ち着いた店内で至高のひとときをお楽しみください。本日入荷の希少なヤナギダコは、鮨やお刺身としてご提供可能です。お好みに応じた調理も承ります。渋谷・幡ヶ谷でお待ちしております。\n\n#鮨魚菜魚魚\n\n#幡...",
"hashtags": [
"#鮨魚菜魚魚",
"#幡ヶ谷駅鮨",
"#幡ヶ谷駅寿司",
"#幡ヶ谷駅グルメ",
"#幡ヶ谷駅ディナー"
],
"num_comments": 0,
"date_posted": "2026-06-15T02:33:13.000Z",
"likes": 10
},
{
"timestamp": "2026-10-22",
"url": "https:\/\/www.instagram.com\/reel\/DeWMy5QyJgP\/",
"user_posted": "ayapegahaku",
"description": ".\n🎬動画でグルメ記録💕\n\n📍名物家 恵比寿店\n\n＼今日の推し！No.32✨／\n\n🌸今日の推し！\n\n今回は、豪華宴会コース10品＆2時間飲み放題を楽しんできました😋💕\n\n前菜や海鮮サラダ、お刺身7点盛りなどのお魚料理から、名物手...",
"hashtags": [
"#恵比寿居酒屋",
"#恵比寿グルメ",
"#PR",
"#イラストレーター",
"#フリーランス"
],
"num_comments": 0,
"date_posted": "2026-10-11T08:12:14.000Z",
"likes": 9
},
{
"timestamp": "2026-10-22",
"url": "https:\/\/www.instagram.com\/reel\/DdLeza8zwYL\/",
"user_posted": "bar314_yutenji",
"description": "今日のオススメ^_^牛すじの赤ワイン煮です。\n#祐天寺バー#祐天寺ビストロ#祐天寺ワイン#祐天寺314",
"hashtags": [
"#祐天寺バー",
"#祐天寺ビストロ",
"#祐天寺ワイン",
"#祐天寺314"
],
"num_comments": 0,
"date_posted": "2026-09-12T07:47:00.000Z",
"likes": 16
},
{
"timestamp": "2026-10-22",
"url": "https:\/\/www.instagram.com\/p\/C_m60stzOIJ\/",
"user_posted": null,
"description": null,
"hashtags": null,
"num_comments": 0,
"likes": 0,
"photos": null
}
]
BEST-IN-CLASS DX
Easy to start. Easier to scale.
Unmatched Stability
Ensure consistent performance and minimize failures by relying on the world’s leading proxy infrastructure.
Simplified Web Scraping
Put your scraping on auto-pilot using production-ready APIs, saving resources and reducing maintenance.
Unlimited Scalability
Effortlessly scale your scraping projects to meet data demands, maintaining optimal performance.
UNDER THE HOOD
Never worry about proxies and CAPTCHAs again
- Automatic IP Rotation
- CAPTCHA Solver
- User Agent Rotation
- Custom Headers
- JavaScript Rendering
- Residential Proxies
API for Seamless Web Crawler Data Access
Comprehensive, Scalable, and Compliant Web Data Extraction
API for Seamless Web Crawler Data Access
Comprehensive, Scalable, and Compliant Web Data Extraction
Tailored to your workflow
Get structured data in JSON, NDJSON, or CSV files through Webhook or API delivery.
Built-in infrastructure and unblocking
Get maximum control and flexibility without maintaining proxy and unblocking infrastructure. Easily scrape data from any geo-location while avoiding CAPTCHAs and blocks.
Battle-proven infrastructure
Bright Data’s platform powers 50,000+ companies worldwide, offering peace of mind with 99.99% uptime, access to 400M+ real user IPs covering 195 countries.
Industry leading compliance
Our privacy practices comply with data protection laws, including the EU data protection regulatory framework, GDPR, and CCPA.
Bright Data is used by world's top brands
