--- title: "Building Web Scraping Agents with CrewAI & Bright Data’s Model Context Protocol (MCP)" slug: crewai-with-bright-data-mcp-web-scraping date: 2025-06-15T12:48:23+00:00 modified: 2025-09-16T16:33:10+00:00 permalink: https://brightdata.com/blog/ai/crewai-with-bright-data-mcp-web-scraping type: blog --- [ Blog ](https://brightdata.com/blog "Blog") / [AI](https://brightdata.com/blog/ai) [AI](https://brightdata.com/blog/ai) # Building Web Scraping Agents with CrewAI & Bright Data’s Model Context Protocol (MCP) Learn how CrewAI and Bright Data MCP streamline web scraping with intelligent agent-based frameworks. 11 min read [ ](https://brightdata.com/blog/authors/satyam-tripathi) [Satyam Tripathi Technical Writer ](https://brightdata.com/blog/authors/satyam-tripathi) ![CrewAI + Bright Data's MCP Web Scraping](https://media.brightdata.com/2025/06/CrewAI-Bright-Datas-MCP-Web-Scraping.png) Web scraping is at a turning point, as traditional methods are being thwarted by [sophisticated anti-bot defenses](/blog/web-data/anti-scraping-techniques), and developers are constantly patching brittle scripts. Although they still work, their limitations are clear, especially next to [modern, AI-native scraping infrastructures](/blog/ai/mcp-vs-traditional-scraping) that deliver resilience and scalability. With the [AI-agent market](https://www.marketsandmarkets.com/PressReleases/ai-agents.asp) set to grow from $7.84 billion to $52.62 billion by 2030, the future of data access lies in intelligent, autonomous systems. By combining CrewAI’s autonomous-agent framework with Bright Data’s robust infrastructure, you get a scraping stack that reasons and overcomes anti-bot barriers. In this tutorial, you’ll build an AI-powered scraping agent that delivers reliable, real-time data. ## The Limits of Old-School Scraping Traditional scraping is brittle – it relies on static [CSS or XPath](/blog/web-data/xpath-vs-css-selectors) selectors that break with any front-end tweak. Key challenges include: - **Anti-bot defenses.** CAPTCHAs, IP throttling, and fingerprinting block simple crawlers. - **JavaScript-heavy pages.** React, Angular, and Vue build the DOM in-browser, so raw HTTP calls miss most content. - **Unstructured HTML.** Inconsistent HTML and scattered inline data demand heavy parsing and post-processing before use. - **Scaling bottlenecks.** Orchestrating proxies, retries, and continual patching turns into an exhausting, never-ending operational burden. ## How CrewAI + Bright Data Streamline Scraping Building an autonomous scraper hinges on two pillars: an adaptive “brain” and a resilient “body”. - **CrewAI (The Brain).** An open-source multi-agent runtime where you spin up a “crew” of agents that can plan, reason, and coordinate end-to-end scraping jobs. - **Bright Data MCP (The Body).** A live-data gateway that routes each request through Bright Data’s Unlocker stack – rotating IPs, solving CAPTCHAs, and running headless browsers – so LLMs receive clean HTML or JSON in one shot. Bright Data’s implementation is the industry-leading source of [reliable data for AI agents](/ai/data-for-ai-agents). Together, this brain-and-body combo lets your agents think, retrieve, and adapt on practically any site. ## What Is CrewAI? [CrewAI](https://github.com/crewAIInc/crewAI) is an open-source framework for orchestrating cooperative AI agents. You define each agent’s role, goal, and tools, then group them into a *crew* to run multi-step workflows. **Core components:** - **Agent.** An LLM-driven worker with a *role*, *goal*, and optional back-story, giving the model domain context. - **Task.** A single, well-scoped job for one agent, plus an *expected\_output* that serves as the quality gate. - **Tool.** Any callable the agent can invoke – an HTTP fetch, a DB query, or Bright Data’s MCP endpoint for scraping. - **Crew.** The collection of agents and their tasks working toward one objective. - **Process.** The execution plan – *sequential*, *parallel*, or *hierarchical –* that controls task order, delegation, and retries. This mirrors a real team: specialists handle their slice, hand results forward, and escalate when needed. ## What Is the Model Context Protocol (MCP)? MCP is an open JSON-RPC 2.0 standard that lets AI agents call external tools and data sources through a single, structured interface. Think of it as a USB-C port for models – one plug, many devices. [Bright Data’s MCP server](/ai/mcp-server) turns that standard into practice by wiring an agent directly into Bright Data’s scraping stack, making [web scraping with MCP](/blog/ai/web-scraping-with-mcp) not just more powerful but far simpler than traditional stacks: - **Anti-bot bypass.** Requests flow through [Web Unlocker](/products/web-unlocker) and a pool of [400M+ rotating residential IPs](/proxy-types/residential-proxies) spanning 195 countries. - **Dynamic-site support.** A purpose-built [Scraping Browser](/products/scraping-browser) renders JavaScript, so agents see the fully loaded DOM. - **Structured results.** Many tools return clean JSON, cutting out custom parsers. The server publishes [50+ ready-made tools](https://github.com/brightdata/brightdata-mcp/blob/main/assets/Tools.md) – from generic URL fetches to site-specific scrapers – so your CrewAI agent can grab product prices, SERP data, or DOM snapshots with one call. ## Building Your First AI Scraping Agent Let’s build a CrewAI agent that extracts details from an Amazon product page and returns them as structured JSON. You can easily redirect the same stack to another site by tweaking just some lines. ### Prerequisites - **Python 3.11** – Recommended for stability. - **Node.js + npm** – Required to run the Bright Data MCP server; download from the [official site](https://nodejs.org/en/download). - **Python virtual environment** – Keeps dependencies isolated; see the [`venv` docs](https://docs.python.org/3/library/venv.html). - **Bright Data account** – [Sign up](/) and create an API token (free trial credits are available). - **Google Gemini API key** – Create a key in [Google AI Studio](https://aistudio.google.com/app/apikey) (click **+ Create API key**). The free tier allows 15 requests per minute and 500 requests per day. No billing profile is required. ### Architecture Overview ```none Environment Setup → LLM Config → MCP Server Init → Agent Definition → Task Definition → Crew Execution → JSON Output ``` ### Step 1. Environment Setup & Imports ```none mkdir crewai-bd-scraper && cd crewai-bd-scraper python -m venv venv # macOS/Linux: source venv/bin/activate # Windows: venv\Scripts\activate pip install "crewai-tools[mcp]" crewai mcp python-dotenv from crewai import Agent, Task, Crew, Process from crewai_tools import MCPServerAdapter from mcp import StdioServerParameters from crewai.llm import LLM import os from dotenv import load_dotenv load_dotenv() # Load credentials from .env ``` ### Step 2. Configure API Keys & Zones Create a `.env` file in your project root: ```none BRIGHT_DATA_API_TOKEN="…" WEB_UNLOCKER_ZONE="…" BROWSER_ZONE="…" GEMINI_API_KEY="…" ``` You need: 1. **API token.** [Generate](https://docs.brightdata.com/api-reference/authentication) a new API token. 2. **Web Unlocker zone.** [Create](https://docs.brightdata.com/scraping-automation/web-unlocker/quickstart) a new Web Unlocker zone. If omitted, a default zone called `mcp_unlocker` is created for you. 3. **Browser API zone.** [Create](https://docs.brightdata.com/scraping-automation/scraping-browser/quickstart) a new Browser API zone. Needed only for JavaScript‑heavy targets. Copy the *username* string shown in the zone’s *Overview* tab. 4. **Google Gemini API key.** Already created in Prerequisites. ### Step 3. LLM Configuration (Gemini) Configure the LLM (Gemini 1.5 Flash) for deterministic output: ```none llm = LLM( model="gemini/gemini-1.5-flash", api_key=os.getenv("GEMINI_API_KEY"), temperature=0.1, ) ``` ### Step 4. Bright Data MCP Setup Configure the Bright Data MCP server. This tells CrewAI how to launch the server and pass credentials: ```none server_params = StdioServerParameters( command="npx", args=["@brightdata/mcp"], env={ "API_TOKEN": os.getenv("BRIGHT_DATA_API_TOKEN"), "WEB_UNLOCKER_ZONE": os.getenv("WEB_UNLOCKER_ZONE"), "BROWSER_ZONE": os.getenv("BROWSER_ZONE"), }, ) ``` This launches `*npx @brightdata/mcp*` as a subprocess and exposes 50+ tools (≈ 57 at the time of writing) via the MCP standard. ### Step 5. Agent and Task Definition Here, we define the agent’s persona and the specific job it needs to do. Effective CrewAI implementations follow the 80/20 rule: spend 80% effort on task design, 20% on agent definition. ```none def build_scraper_agent(mcp_tools): return Agent( role="Senior E-commerce Data Extractor", goal=( "Return a JSON object with snake_case keys containing: title, current_price, " "original_price, discount, rating, review_count, last_month_bought, " "availability, product_id, image_url, brand, and key_features for the " "target product page. Ensure strict schema validation." ), backstory=( "Veteran web-scraping engineer with years of experience reverse-" "engineering Amazon, Walmart, and Shopify layouts. Skilled in " "Bright Data MCP, proxy rotation, CAPTCHA avoidance, and strict " "JSON-schema validation." ), tools=mcp_tools, llm=llm, max_iter=3, verbose=True, ) def build_scraping_task(agent): return Task( description=( "Extract product data from https://www.amazon.in/dp/B071Z8M4KX " "and return it as structured JSON." ), expected_output="""{ "title": "Product name", "current_price": "$99.99", "original_price": "$199.99", "discount": "50%", "last_month_bought": 150, "rating": 4.5, "review_count": 1000, "availability": "In Stock", "product_id": "ABC123", "image_url": "https://example.in/image.jpg", "brand": "BrandName", "key_features": ["Feature 1", "Feature 2"], }""", agent=agent, ) ``` Here is what each parameter does: - **role –** Short job title, CrewAI injects into every system prompt. - **goal –** North-star objective; CrewAI compares it after each loop to decide whether to stop. - **backstory –** Domain context that guides tone and reduces hallucinations. - **tools –** List of `BaseTool` objects (e.g., MCP `search_engine`, `scrape_as_markdown`). - **llm –** Model CrewAI uses for each *think → plan → act → answer* cycle. - **max\_iter –** Hard cap on the agent’s internal loops (default 20 in v0.30 +). - **verbose –** Streams every prompt, thought, and tool call to stdout (useful for debugging). - **description –** Action-oriented instruction is injected each turn. - **expected\_output –** Formal contract of a valid answer (use strict JSON, no trailing comma). - **agent –** Binds this task to a specific `Agent` instance for `Crew.kickoff()`. ### Step 6. Crew Assembly and Execution This part assembles the agent and task into a `Crew` and runs the workflow. ```none def scrape_product_data(): """Assembles and runs the scraping crew.""" with MCPServerAdapter(server_params) as mcp_tools: scraper_agent = build_scraper_agent(mcp_tools) scraping_task = build_scraping_task(scraper_agent) crew = Crew( agents=[scraper_agent], tasks=[scraping_task], process=Process.sequential, verbose=True ) return crew.kickoff() if __name__ == "__main__": try: result = scrape_product_data() print("\n[SUCCESS] Scraping completed!") print("Extracted product data:") print(result) except Exception as e: print(f"\n[ERROR] Scraping failed: {str(e)}") ``` ### Step 7. Running the Scraper Execute the script from your terminal. You will see the agent’s thought process in the console as it plans and executes the task. The final output will be a clean JSON object: ```none { "title": "Boat BassHeads 100 in-Ear Headphones with Mic (Black)", "current_price": "₹349", "original_price": "₹999", "discount": "-65%", "rating": 4.1, "review_count": 419630, "last_month_bought": 5000, "availability": "In stock", "product_id": "B071Z8M4KX", "image_url": "https://m.media-amazon.com/images/I/513ugd16C6L._SL1500_.jpg", "brand": "boAt", "key_features": [ "10mm dynamic driver", "HD microphone", "1.2 m cable", "Comfortable fit", "1 year warranty" ] } ``` ## Adapting to Other Targets The real strength of an agent-based design is its flexibility. Want to scrape LinkedIn posts instead of Amazon products? Just update the agent’s *role*, *goal*, and *backstory*, plus the task’s *description* and *expected\_output*. Everything else – including the underlying code and infrastructure – stays exactly the same. ```none role = "Senior LinkedIn Post Extractor" goal = ( "Return a JSON object containing: author_name, author_title, " "author_profile_url, post_content, post_date, likes_count, " "and comments_count" ) backstory = ( "Seasoned social-data engineer specializing in LinkedIn data " "extraction using Bright Data MCP. Produces clean, structured " "JSON output." ) description = ( "Extract post data from LinkedIn post (ID: orlenchner_agents-" "brightdata-activity-7336402761892122625-h5Oa) and return " "structured JSON." ) expected_output = """{ "author_name": "Post author's full display name", "author_title": "Author's job title/headline", "author_profile_url": "Author's profile URL", "post_content": "Complete post text with formatting", "post_date": "ISO 8601 UTC timestamp", "likes_count": "Number of post likes", "comments_count": "Number of post comments", }""" ``` The output will be a clean JSON object: ```none { "author_name": "Or Lenchner", "author_title": "CEO at Bright Data - Keeping public web data, public.", "author_profile_url": "https://il.linkedin.com/in/orlenchner", "post_content": "NEW PRODUCT! There’s a consensus that the future internet will be run by automated #Agents , automating the activity on behalf of “their” humans. AI solved the automation part (or at least shows strong indications), but the number one problem is ensuring smooth access to every website at scale without being blocked. browser.ai is the solution → Your Agent always gains access to any website with a simple prompt. Agents using Bright Data are already executing hundreds of millions of web actions daily on our browser infrastructure. #BrightData has long been the go-to for major LLM companies, providing the tools and scale they need to train and deploy such technologies. With browser.ai , we’re taking that foundation and tailoring it specifically for AI agents, optimizing our APIs, proxy networks, and serverless browsers to handle their unique demands. The web isn’t fully prepared for this shift yet, but we are. browser.ai immediate focus is to ensure *smooth* access to any website (DONE!), while phase two will be all about *fast* access (wip). https://browser.ai/", "post_date": "2026-06-05T14:45:22.155Z", "likes_count": 119, "comments_count": 7 } ``` ## Cost Optimization Bright Data’s MCP is usage-based, so every extra request adds to your bill. A few design choices keep costs in check: - **Targeted scraping.** Request only the fields you need instead of crawling entire pages or datasets. - **Caching.** Enable CrewAI’s tool-level cache (`cache_function`) to skip calls when content hasn’t changed, saving both time and credits. - **Efficient tool selection.** Default to the *Web Unlocker* zone and switch to a *Browser API* zone only when JavaScript rendering is essential. - **Set `max_iter`.** Give every agent a sensible upper bound so it can’t loop forever on a broken page. (You can also throttle requests with `max_rpm`.) Follow these practices and your CrewAI agents will stay secure, reliable, and cost-efficient, ready for production workloads on Bright Data MCP. ## What’s Next The MCP ecosystem keeps expanding: **OpenAI’s Responses API** and **Google DeepMind’s Gemini SDK** now speak MCP natively, guaranteeing long-term compatibility and continued investment. CrewAI is rolling out multimodal agents, richer debugging, and enterprise RBAC, while **Bright Data’s MCP server exposes 60-plus ready-made tools** and still growing. Together, agent frameworks and standardized data access unlock a new wave of web intelligence for [AI-powered applications](/ai/web-access). Guides on plugging [MCP into the OpenAI Agents SDK](https://hackernoon.com/mcp-openai-agents-sdk-how-to-build-a-powerful-ai-agent) underline how essential rock-solid data pipes have become. Ultimately, you’re not just building a scraper – you’re orchestrating an adaptive data workflow built for the future web. **Need more scale?** Skip scraper upkeep and block-fighting – just request structured data: - [Crawl API](/products/crawl-api) – full-site extraction at scale. - [Web Scraper APIs](/products/web-scraper) – 120-plus domain-specific endpoints. - [SERP API](/products/serp-api) – hassle-free search-engine scraping. - [Dataset Marketplace](/products/datasets) – fresh, validated datasets on demand. **Ready to build next-gen AI apps?** Explore Bright Data’s full [AI product suite](/ai) and see what seamless, live web access does for your agents. For deeper dives, check our MCP guides for [Qwen-Agent](/blog/ai/qwen-agent-with-bright-data-mcp-server) and [Google ADK](/blog/ai/google-adk-with-bright-data-mcp-server). Contact usStart free trial No credit card required [ ](https://www.linkedin.com/in/triposat/) Satyam Tripathi Technical Writer 5 years experience Satyam Tripathi helps SaaS and data startups turn complex tech into actionable content, boosting developer adoption and user understanding. Expertise Python Developer Education Technical Writing [ View all articles ](https://brightdata.com/blog/authors/satyam-tripathi) Table of Contents Data for AI Supercharge your AI with instant and reliable access to web data. No blockers. No hassle. Talk to an expert Bright Data MCP Get started with Bright Data’s Web MCP Server today with 5000 free monthly requests and unlock your AI’s full potential. Start free now [ ](https://news.ycombinator.com/submitlink?t=Building+Web+Scraping+Agents+with+CrewAI+%26amp%3B+Bright+Data%E2%80%99s+Model+Context+Protocol+%28MCP%29&u=https://brightdata.com/blog/ai/crewai-with-bright-data-mcp-web-scraping) [ ](https://www.linkedin.com/shareArticle?mini=true&title=Building+Web+Scraping+Agents+with+CrewAI+%26amp%3B+Bright+Data%E2%80%99s+Model+Context+Protocol+%28MCP%29&url=https://brightdata.com/blog/ai/crewai-with-bright-data-mcp-web-scraping) [ ](http://www.reddit.com/submit?title=Building+Web+Scraping+Agents+with+CrewAI+%26amp%3B+Bright+Data%E2%80%99s+Model+Context+Protocol+%28MCP%29&url=https://brightdata.com/blog/ai/crewai-with-bright-data-mcp-web-scraping) ## You might also be interested in [ ](https://brightdata.com/blog/ai/openhuman-with-bright-data "Production-Ready Web Access in OpenHuman Through the Bright Data CLI") [AI Antonello Zanini Technical Writer ### Production-Ready Web Access in OpenHuman Through the Bright Data CLI Integrate Bright Data CLI with OpenHuman to enable production-ready web access and data collection for AI agents. 09-Sep-2026 12 min read ](https://brightdata.com/blog/ai/openhuman-with-bright-data) [ ](https://brightdata.com/blog/ai/minimax-m3-with-bright-data "Giving self-hosted MiniMax M3 agents live web access with Bright Data") [AI Satyam Tripathi Technical Writer ### Giving self-hosted MiniMax M3 agents live web access with Bright Data Self-hosted MiniMax M3 agents get live web access using Bright Data’s search and scraping tools. Bypass blocks and CAPTCHAs. 09-Sep-2026 54 min read ](https://brightdata.com/blog/ai/minimax-m3-with-bright-data) [ ](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax "Multimodal Web Scraping with MiniMax") [Web Data Antonello Zanini Technical Writer ### Multimodal Web Scraping with MiniMax Pair Bright Data Web Unlocker with MiniMax M3 vision to extract structured data from images and web page screenshots. 09-Sep-2026 4 min read ](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax)