In this article, you will see:
- What the xpander.ai platform is and what it offers for building AI agents.
- Why AI agents need access to the web to be truly effective.
- How to integrate the built-in Bright Data tools into an xpander.ai agent to give it web scraping functionality.
Let’s dive in!
What Is xpander.ai?
Xpander.ai is a Backend-as-a-Service platform for building autonomous AI agents. It is a no-code solution designed to help enterprise developers efficiently build, test, and deploy AI agents. It also comes with an open-source SDK to programmatically build and run AI agents.
The platform provides a visual environment to define multi-agent workflows and collaboration. That also lets you attach tools and integrate with enterprise systems. Plus, you can simulate and test agent behavior before going live.
Key features include:
- Agent Graph System: Ensures reliable multi-step agent execution.
- Agentic Connectors: Allows agents to connect to various third-party systems and APIs.
The Importance of Fresh Web Data for Accurate AI Agents
No matter which AI agent-building platform, library, or tool you choose, there is a fundamental limitation. That is, LLMs can only carry out tasks and answer questions based on the data they were trained on. This is a major hurdle when building AI agents, which are expected to perform actions that go beyond the static capabilities of a typical LLM.
Thus, to be more accurate, operational, and effective, AI agents need access to the web. They should be able to read web pages and use their content to ground their responses and decisions. After all, the web is one of the richest and most up-to-date sources of data.
However, simply having access to the web is not enough. Most sites implement anti-scraping and anti-bot measures to block automated AI crawlers. This means your agents need powerful tools that let them extract data from web pages, and in a format optimized for AI processing.
That is precisely what Bright Data’s AI infrastructure—available in xpander.ai via integrated connectors—is devised to offer. Among its many features, it lets your no-code AI agents to pull fresh data from over 50 popular platforms in structured JSON format.
Bright Data takes care of handling CAPTCHAs, IP bans, rate limits, and more. Combined with xpander.ai’s AI agent builder, you can integrate all of this without writing a single line of code. The result? A production-ready AI agent with real-time access to reliable web data!
How to Integrate the Bright Data Connector for Building an xpander.ai Scraping Agent
In this guide, you will learn how to build an AI agent in xpander.ai. Specifically, you will use the Bright Data connector to give your agent the ability to scrape data from the web.
We will show how to create a web scraping agent that can ground its responses by retrieving live data from the Internet. This is just one example of what is possible with the xpander.ai + Bright Data integration. You can easily adapt this approach to many other use cases.
Note: In a sense, this example works like a RAG agentic workflow. The reason is that the Bright Data connector acts as the retrieval component, fetching up-to-date data for the agent to use.
Follow the steps below to create your no-code AI scraping agent in xpander!
Prerequisites
To reproduce this tutorial, you need the following:
- An xpander.ai account: A free account is sufficient for simple tests. For more advanced use cases, you will require a paid plan.
- A Bright Data API key: You can create one for free, as explained in the official guide.
If you do not have these yet, click on the links above and follow the setup instructions. Let’s get started!
Step #1: Create a New Agent
Log in to xpander.ai and go to your profile dashboard. Click on “Agent” in the menu on the left, then press the “New Agent” button to add a new agent:
You will reach the following page, with a form where you can set up your new agent. Give it a name like “Web Scraper Agent”:
Leave all other settings in the “General” tab as they are. The defaults are enough for a simple setup like this one. By default, xpander.ai will use OpenAI’s GPT-4o as the LLM model.
Great! You now have a new blank AI agent ready in xpander.ai.
Step #2: Set Up Bright Data Tools for Web Scraping
Right now, your agent can only perform actions offered by the selected LLM provider. It is time to supercharge it with web scraping capabilities using Bright Data.
To do this, go to the “Tools” tab on your agent’s page, then click the “Add tools” button:
A panel titled “Add tools” will appear on the right. Search for “bright data” and select the Bright Data integration:
The following modal will show up:
Fill it out as follows:
- Connector name: Give your Bright Data connector a name (such as “Bright Data Connector”).
- Authentication mode: Select the “API Key” option.
- Authentication scope:
- Select the “Integration user” option.
- Paste your Bright Data API key.
- Select the “Bearer” option. This will pass the API key in the
Authorizationheader via theBearerpattern, which is an authentication method supported by the Bright Data APIs.
Once everything is filled in, press the “Save” button.
Now, you will be prompted to select the specific Bright Data tools you want to enable in your agent:
We recommend selecting all tools to unlock full web scraping capabilities. As of this writing, the available tools are:
- Start Data Collection Job by Dataset ID: Launches a scraping job for a specified dataset using the Web Scraper APIs.
- Execute Proxy Request by URL: Sends an HTTP request through Bright Data’s proxy network for accessing the content of any web page.
- Download Dataset Snapshot by ID: Downloads a snapshot of a dataset in various formats, passing the data to the AI.
Once you have selected the desired tools, click the “Add to agent” button in the bottom-right corner:
The “Tools” tab of your agent will now show the Bright Data connector with the tools you configured:
Notice that you can click on any tool to view or adjust its configuration.
Fantastic! Your AI agent is now fully integrated with Bright Data tools and ready to scrape the web.
Step #3: Specialize Your AI Scraping Agent
Now that your agent has access to the Bright Data tools for web scraping, give it a custom system prompt. This tells the agent what it is and how it should operate.
To do this, click on the “Instructions” tab and paste something like the following into the “System prompt” textarea:
For more specialized agents, you can also add custom rules and goals.
Amazing! Your xpander scraping agent is ready.
Step #4: Put It All Together
Click on the “Agent graph” button to view your current AI agent workflow:
You will see a single agent with access to the three configured Bright Data tools for web scraping.
Well done! All that is left is to test the agent and see it in action.
Step #5: Test the Web Scraper Agent
Go back to the “Tester Chat” tab and try out your agent with a prompt like this:
This instructs your web scraping agent to dynamically look online for the top 3 headphones priced under $100 and retrieve information directly from their product detail pages (PDPs).
As you can imagine, a standard LLM would be able to handle this kind of task without access to dedicated scraping tools like those provided by Bright Data.
Paste the prompt into the chat input and send it to your agent:
The agent uses the LLM and Bright Data tools to:
- Perform a web search and find the top 3 headphones.
- For each product, start a data collection job and download data from Amazon.
- Summarize the information into a short, accurate response, complete with real-world links to the Amazon product detail pages.
If you expand one of the tool sections in the interface, you will see something like this:
This proves that, behind the scenes, the AI agent automatically detected which Bright Data tools to use to complete the task. In detail, it called them with the right parameters to fetch fresh scraped data (in this case, directly from Amazon product pages).
Et voilà! You now have a fully functional scraping agent on xpander.ai, powered by Bright Data’s AI data infrastructure.
Next Steps
Now that you have a Bright-Data-powered xpander scraping agent in place, you can:
- Deploy your agent: Run your AI Agent directly on the xpander.ai platform, or deploy it on your own infrastructure for greater control.
- Call your agent via API with the
XpanderClient: Utilize the xpander SDK to manage agents and get access to utility functions for working with LLM responses. - Explore xpander’s official workshops: Engage in hands-on workshops that will guide you through building complete AI agent solutions using the xpander.ai platform.
Conclusion
In this article, you learned how to use xpander.ai to build a no-code AI scraping agent. This was made possible by the Bright Data connector, which exposes advanced scraping tools for integration in xpander agents.
This was just a simple example, but you might want to build more sophisticated AI agents. For that, you need trusted solutions to fetch, validate, and transform web content. That is exactly what you can find in the Bright Data AI agent infrastructure.
Create a free Bright Data account today and start exploring our AI-ready data tools!
Technical Writer
5.5 years experience
Antonello Zanini is a technical writer, editor, and software engineer with 5M+ views. Expert in technical content strategy, web development, and project management.
