Help! I Don’t Know Code, But I Want To Collect Data Off The Web (Hint: Use Bright Data + Integromat)

This post will introduce you to “low-code” development and its potential for empowering businesses with limited resources, including step-by-step instructions on how to integrate Bright Data with Integromat.
semi transparent graphic of browsers, codes, settings and trust ratings illustrating the frustration of doing data collection without knowing how to code - using bright data's integromat
Itamar Abromovich
Itamar Abramovich | Director of Product Managment
03-Jun-2021
Share:

In this article you’ll learn:

  • What “low-code” means and why it matters for businesses
  • How low-code development can help a business become a data-driven powerhouse with few resources
  • Step-by-step walkthrough of how to integrate Bright Data and Integromat to collect data from the Web with ZERO coding knowledge (it takes minutes)

Understanding “low-code” and why it’s a huge deal for businesses

The best business decisions are data-based decisions. And there is no greater source of data than the Web. But if you need to collect data off public websites, and you can’t code or afford an army of engineers, what’s next?

Welcome to the world of “low-code.”

I’ll save you the Google search. “Low-code” refers to “low-code development,” which in English means: developing an app or software through a visual user interface, rather than through coding and programming.

In other words, it’s perfect for people who don’t know how to code, but want to digitize how they operate–quickly–so they can work more efficiently, deliver products faster, connect better with customers, and save on development costs.

The implications of low code for businesses are huge. Low code removes the complexity from building powerful high-performing business apps. And it enables businesses to quickly adapt and respond to fast-changing business conditions. Small wonder that so many are high on “low-code” these days: Forrester and Gartner predict 75% of all enterprise software will be built with low-code technology in 2021.

Collecting Web data with low-code solutions

Low-code is revolutionizing our capacity to collect web data. A business used to require tremendous resources to get around constantly changing websites, blockades, misleading and fake information, and integrating data on a huge scale with an existing business infrastructure.

Today there are 2 brilliant companies, Bright Data and Integromat, that when used together, enable you to collect, store, and analyze web data on a massive scale without having to develop your own team or sophisticated tech platform (full disclosure: I work at one of these companies).

Together these low-code solutions have revolutionized how businesses across sectors can collect and then leverage web data in their industries. Bright Data is the industry powerhouse in data collection solutions, while Integromat lets you connect apps and automate workflows in a few clicks. This is how you can more easily automate business processes and workflows, while making data-based decisions in real-time.

By integrating Bright Data’s Data Collector with Integromat, you will be moving and analyzing web data without effort so you can focus on growing your business.

The rest of this article will teach you how to integrate Bright Data and Integromat in a few minutes. For real, anyone can do this!

Setting up your first Data Collection with Bright Data and Integromat

1. Sign up at Brightdata.com – it’s easy and there is a free trial.

2. Select “Data Collector” from the menu bar of the Bright Data Control Panel, and then click the “Add new collector” button.

Screen Shot of Bright Data's Data collector control panel ADD NEW COLLECTOR button

Image source: Bright Data’s Dashboard

3. Now you’ll choose which collector to use depending on your needs. (By the way – if Bright Data hasn’t created a collector yet for the site you need, you can have one made for you fast!)

Select the collector you want by entering a URL or by use case. For the sake of example, we’ll use “Instagram.”

Data Collector control pane Which website would you like to collect data from menuImage source: Bright Data’s Dashboard

4. As you enter “Instagram” into the search bar, you’ll see a variety of Instagram collectors in the drop-down menu. Each collector focuses on different needs, so play around to find what’s most useful to you. Say for example, you want to collect profiles and posts related to specific hashtags like “#Summertravel2021.” Select the “Instagram hashtag” Data Collector.

5. After you select your collector, you’ll see information about it on the right, including an overview of what it collects and examples of its data output, as well as API and source code (we’ll come back to this). When you’re ready, select “Choose this template” to move forward.

choosing the right data collection template for data collection with the Bright Data collector - control panel viewImage source: Bright Data’s Dashboard

6. Now it’s time to set your data delivery preferences: when you’d like to receive your data and in what format.

IMPORTANT: for now, keep your delivery method as “delivery my data by email.” (We’ll change it later in the setup when we’re ready)

7. After you’ve set your delivery preferences, click “Next.”

data delivery preferences in the data collectorImage source: Bright Data’s Dashboard

8. Now you have the option to customize your output settings, so you can choose what data items you prefer to receive with each collection. When you’re finished customizing your output, click “Next.”

customizing the data collectors outputsImage source: Bright Data’s Dashboard

9. Next you’ll see the “Initiate by API” screen. Keep this screen open, while you open a new tab or window and go to integromat.com. We’ll return to the “Initiate by API” page later in the flow.

initiate the API on the data collector Image source: Bright Data’s Dashboard

10. In Integromat, choose “Create a new scenario” (if you haven’t signed up yet for Integromat, do so).

data collection - data collector control panel Create a New Scenario Image source: Integromat’s Dashboard

11. Type “req” in the search, select HTTP from the results.

12. Select the HTTP icon.

13. Click the Continue button.

integrating the bright data collection platformImage source: Integromat’s Dashboard

14. Click on HTTP in the pop-up.

selecting http from the pop upImage source: Integromat’s Dashboard

15. Select “Make a request” to send HTTP requests to a specific URL.

make a request selection in the control panelImage source: Integromat’s Dashboard

16. Now it’s time to copy several lines of code from Bright Data to Integromat (Notice split screen below). Return to the “Initiate by API” screen in Bright Data Collector and copy the URL code. Then paste that code into the slot for URL in Integromat.

17. Select Post for Method in Integromat.

18. Copy the code for “Authorization” from “Initiate by API” in Bright Data Collector and paste it in the “Headers” slot in Integromat.

19. Select “Raw” for body type in Integromat.

20. Select “JSON” for Content type in Integromat.

21. Copy the code for “account” from “Initiate by API” in Bright Data Collector and paste it in the “Request content” slot in Integromat.

IMPORTANT: When copying the format from the API code, do not copy the “-d” before the code

NOTE: You can specify what exact data inputs you’d like to collect within this slot.
Copy the code from API, but feel free to change the input of the code to fit your needs. For example, in the picture below, you’d copy the code with the words “bright_data,” but you could replace those words with any profile name.”

22. When finished, click “OK” to move forward.

initiating the data collection apiImage source: Bright Data and Integromat Dashboards

23. Select “Run once” to initiate the integration.

Click on Run Once to get the data collection process startedImage source: Integromat’s Dashboard

24. Once you’ve initiated a run with Integromat, you’ll see the following screen on Bright Data Collector, as your data is being collected:

How the control panel screen should look when the data collector is running the data collection processImage source: Bright Data’s Dashboard

25. When your data is ready, you’ll see a green check-marked message “Your data set is ready” and you’ll receive your data here in the format you chose.

how it looks when the collected data is readyImage source: Bright Data’s Dashboard

How to Integrate Bright Data and Integromat with Webhook

The following are instructions on how to set up another Integromat scenario so that you can work with webook to analyze the data collected in the previous scenario.

1. From the Bright Data Data Collector dashboard, select the edit button next to “Delivery method” to change delivery configurations.

changing the way the collected data is deliveredImage source: Bright Data’s Dashboard

2. Select “Webhook” from the “Deliver my data by” options. Keep your screen open, but open a new tab or window and go to integromat.com.

selecting deliver my data by webhookImage source: Bright Data’s Dashboard

3. In Integromat, click on “Create a new scenario.”

create new scenario pageImage source: Integromat’s Dashboard

4. Type “Webhook” to search for service you want to integrate.

5. Select the “Webhooks” icon.

6. Click “Continue” to move forward.

selecting webhooks as the service to integrateImage source: Integromat’s Dashboard

7. On the next screen, select “Webooks.”

and now select webhooksImage source: Integromat’s Dashboard

8. Next, select “Custom Webhook.”

make sure you select webhookImage source: Integromat’s Dashboard

9. Click on Add.

10. Give your webhook a personalized name.

11. Click “Save.”

saving the new parameters you just set for integrationImage source: Integromat’s Dashboard

12. After you’ve added a personalized name (step 11), you should see a link in blue under your name (see picture below). You’ll need to copy this link and paste it into Bright Data (step 13).

13. Paste your link from Webhook (step 12) here in the slot for URL.

14. Click “Update” to move forward.

update the new configurationImage source: Bright Data and Integromat Dashboards

15. Click “Redetermine data structure” so that you’ll be able to use your data with ease.

16. Next you’re asked by Integromat to send your data sample to the webhook address. Go to Bright Data’s dashboard to do this (see next steps). Leave the Integromat window open.

configure integromat to send your collected data to webhooksImage source: Integromat’s Dashboard

17. Go back to Bright Data’s Dashboard and click “Initiate manually” in your Collector’s drop-down settings menu.

Initiate manually in Data Collectors dashboardImage source: Bright Data’s Dashboard

18. Click the “Start” button.

click START to start the data collectingImage source: Bright Data’s Dashboard

19. Here you’ll see your results in the format of your choice when your data is ready in Bright Data.

20. In Integromat, you’ll receive a “Successfully determined” message when your data structure has been determined.

confirmation message that your data has been determinedImage source: Bright Data and Integromat Dashboards

As an example of how to integrate your data in a useful way, let’s use Google Sheets.

21. Next you’ll want to configure your data for Google Sheets. Click on the “plus” sign in the menu bar at the bottom of your Integromat dashboard, and then select “Google Sheets.

configure the data for Google sheetsImage source: Integromat’s Dashboard

22. Next select “Add a Row.”

add row in output google sheetsImage source: Integromat’s Dashboard

23. Click “add” to create a new connection to your google sheet app

24. Give it a name and follow the process by allowing google sheet and integromat to integrate.

naming the google sheet for the integration processImage source: Integromat’s Dashboard

25. Select spreadsheet and sheet in “Mode.”

26. We created a spreadsheet example named “Instagram,” but you can select any spreadsheet to work with.

27. In these values, you’ll be able to choose the data fields from your data collection to include in your spreadsheet.

28. These are the data fields to choose from your collection.

29. When you’re ready, click OK. You’re all set!

click OK and you are all set. Now YOU are the data collectorImage source: Integromat’s Dashboard

To finish this integration, you should configure integromat to pull this input from the business system you have running

Your integration is ready to go and you’re ready to enjoy the benefits of data collection for business insights!

Thoughts? Was this helpful? Let me know below.

Itamar Abromovich
Itamar Abramovich | Director of Product Managment

Itamar Abramovich is Director of Product Management at Bright Data.
With a deep knowledge of SaaS products, he helps businesses create scalable, efficient, and cost-effective data collection processes to support cross-company growth. itamar@brightdata.com

Share: