Help! I Don’t Know Code, But I Want To Collect Data Off The Web (Hint: Use Bright Data + Integromat)
In this article you’ll learn:
- What “low-code” means and why it matters for businesses
- How low-code development can help a business become a data-driven powerhouse with few resources
- Step-by-step walkthrough of how to integrate Bright Data and Integromat to collect data from the Web with ZERO coding knowledge (it takes minutes)
Understanding “low-code” and why it’s a huge deal for businesses
The best business decisions are data-based decisions. And there is no greater source of data than the Web. But if you need to collect data off public websites, and you can’t code or afford an army of engineers, what’s next?
Welcome to the world of “low-code.”
I’ll save you the Google search. “Low-code” refers to “low-code development,” which in English means: developing an app or software through a visual user interface, rather than through coding and programming.
In other words, it’s perfect for people who don’t know how to code, but want to digitize how they operate–quickly–so they can work more efficiently, deliver products faster, connect better with customers, and save on development costs.
The implications of low code for businesses are huge. Low code removes the complexity from building powerful high-performing business apps. And it enables businesses to quickly adapt and respond to fast-changing business conditions. Small wonder that so many are high on “low-code” these days: Forrester and Gartner predict 75% of all enterprise software will be built with low-code technology in 2021.
Collecting Web data with low-code solutions
Low-code is revolutionizing our capacity to collect web data. A business used to require tremendous resources to get around constantly changing websites, blockades, misleading and fake information, and integrating data on a huge scale with an existing business infrastructure.
Today there are 2 brilliant companies, Bright Data and Integromat, that when used together, enable you to collect, store, and analyze web data on a massive scale without having to develop your own team or sophisticated tech platform (full disclosure: I work at one of these companies).
Together these low-code solutions have revolutionized how businesses across sectors can collect and then leverage web data in their industries. Bright Data is the industry powerhouse in data collection solutions, while Integromat lets you connect apps and automate workflows in a few clicks. This is how you can more easily automate business processes and workflows, while making data-based decisions in real-time.
By integrating Bright Data’s Data Collector with Integromat, you will be moving and analyzing web data without effort so you can focus on growing your business.
The rest of this article will teach you how to integrate Bright Data and Integromat in a few minutes. For real, anyone can do this!
Setting up your first Data Collection with Bright Data and Integromat
1. Sign up at Brightdata.com – it’s easy and there is a free trial.
2. Select “Data Collector” from the menu bar of the Bright Data Control Panel, and then click the “Add new collector” button.
3. Now you’ll choose which collector to use depending on your needs. (By the way – if Bright Data hasn’t created a collector yet for the site you need, you can have one made for you fast!)
Select the collector you want by entering a URL or by use case. For the sake of example, we’ll use “Instagram.”
4. As you enter “Instagram” into the search bar, you’ll see a variety of Instagram collectors in the drop-down menu. Each collector focuses on different needs, so play around to find what’s most useful to you. Say for example, you want to collect profiles and posts related to specific hashtags like “#Summertravel2021.” Select the “Instagram hashtag” Data Collector.
5. After you select your collector, you’ll see information about it on the right, including an overview of what it collects and examples of its data output, as well as API and source code (we’ll come back to this). When you’re ready, select “Choose this template” to move forward.
6. Now it’s time to set your data delivery preferences: when you’d like to receive your data and in what format.
IMPORTANT: for now, keep your delivery method as “delivery my data by email.” (We’ll change it later in the setup when we’re ready)
7. After you’ve set your delivery preferences, click “Next.”
8. Now you have the option to customize your output settings, so you can choose what data items you prefer to receive with each collection. When you’re finished customizing your output, click “Next.”
9. Next you’ll see the “Initiate by API” screen. Keep this screen open, while you open a new tab or window and go to integromat.com. We’ll return to the “Initiate by API” page later in the flow.
10. In Integromat, choose “Create a new scenario” (if you haven’t signed up yet for Integromat, do so).
11. Type “req” in the search, select HTTP from the results.
12. Select the HTTP icon.
13. Click the Continue button.
14. Click on HTTP in the pop-up.
15. Select “Make a request” to send HTTP requests to a specific URL.
16. Now it’s time to copy several lines of code from Bright Data to Integromat (Notice split screen below). Return to the “Initiate by API” screen in Bright Data Collector and copy the URL code. Then paste that code into the slot for URL in Integromat.
17. Select Post for Method in Integromat.
18. Copy the code for “Authorization” from “Initiate by API” in Bright Data Collector and paste it in the “Headers” slot in Integromat.
19. Select “Raw” for body type in Integromat.
20. Select “JSON” for Content type in Integromat.
21. Copy the code for “account” from “Initiate by API” in Bright Data Collector and paste it in the “Request content” slot in Integromat.
IMPORTANT: When copying the format from the API code, do not copy the “-d” before the code
NOTE: You can specify what exact data inputs you’d like to collect within this slot.
Copy the code from API, but feel free to change the input of the code to fit your needs. For example, in the picture below, you’d copy the code with the words “bright_data,” but you could replace those words with any profile name.”
22. When finished, click “OK” to move forward.
23. Select “Run once” to initiate the integration.
24. Once you’ve initiated a run with Integromat, you’ll see the following screen on Bright Data Collector, as your data is being collected:
25. When your data is ready, you’ll see a green check-marked message “Your data set is ready” and you’ll receive your data here in the format you chose.
How to Integrate Bright Data and Integromat with Webhook
The following are instructions on how to set up another Integromat scenario so that you can work with webook to analyze the data collected in the previous scenario.
1. From the Bright Data Data Collector dashboard, select the edit button next to “Delivery method” to change delivery configurations.
2. Select “Webhook” from the “Deliver my data by” options. Keep your screen open, but open a new tab or window and go to integromat.com.
3. In Integromat, click on “Create a new scenario.”
4. Type “Webhook” to search for service you want to integrate.
5. Select the “Webhooks” icon.
6. Click “Continue” to move forward.
7. On the next screen, select “Webooks.”
8. Next, select “Custom Webhook.”
9. Click on Add.
10. Give your webhook a personalized name.
11. Click “Save.”
12. After you’ve added a personalized name (step 11), you should see a link in blue under your name (see picture below). You’ll need to copy this link and paste it into Bright Data (step 13).
13. Paste your link from Webhook (step 12) here in the slot for URL.
14. Click “Update” to move forward.
15. Click “Redetermine data structure” so that you’ll be able to use your data with ease.
16. Next you’re asked by Integromat to send your data sample to the webhook address. Go to Bright Data’s dashboard to do this (see next steps). Leave the Integromat window open.
17. Go back to Bright Data’s Dashboard and click “Initiate manually” in your Collector’s drop-down settings menu.
18. Click the “Start” button.
19. Here you’ll see your results in the format of your choice when your data is ready in Bright Data.
20. In Integromat, you’ll receive a “Successfully determined” message when your data structure has been determined.
As an example of how to integrate your data in a useful way, let’s use Google Sheets.
21. Next you’ll want to configure your data for Google Sheets. Click on the “plus” sign in the menu bar at the bottom of your Integromat dashboard, and then select “Google Sheets.
22. Next select “Add a Row.”
23. Click “add” to create a new connection to your google sheet app
24. Give it a name and follow the process by allowing google sheet and integromat to integrate.
25. Select spreadsheet and sheet in “Mode.”
26. We created a spreadsheet example named “Instagram,” but you can select any spreadsheet to work with.
27. In these values, you’ll be able to choose the data fields from your data collection to include in your spreadsheet.
28. These are the data fields to choose from your collection.
29. When you’re ready, click OK. You’re all set!
To finish this integration, you should configure integromat to pull this input from the business system you have running
Your integration is ready to go and you’re ready to enjoy the benefits of data collection for business insights!
Thoughts? Was this helpful? Let me know below.