Walmart is the world’s largest company in terms of both revenue and number of employees. And contrary to popular opinion, Walmart is much more than just a retail corporation. In fact, it’s one of the largest e-commerce websites in the world, making it a great source of information about products. However, due to the vast portfolio of products, it’s impossible for any person to collect this data manually, which is why it’s the ideal use case for web scraping.
With web scraping, you can quickly retrieve data about thousands of Walmart products (such as the name of the product, price, description, images, and ratings) and store it in any format that you find useful. Scraping Walmart data will enable you to monitor the prices of different products and their stock level, analyze market movements and customer behavior, and create different applications.
In this article, you’ll learn two completely different methods of scraping Walmart.com. First, you’ll follow step-by-step instructions to learn how to scrape Walmart using Python and Selenium, a tool primarily used for automating web applications for testing purposes. Second, you’ll learn how you can more easily use the Bright Data Walmart Scraper to do the same thing.
Scraping Walmart
As you may know, there are many different ways to scrape websites, including Walmart. One such method involves utilizing Python and Selenium.
Instructions on Scraping Walmart with Python and Selenium
Python is one of the most popular programming languages when it comes to web scraping. Meanwhile, Selenium is mainly used to automate testing. However, it can also be used for web scraping due to its ability to automate web browsers.
In essence, Selenium simulates manual actions in a web browser. With Python and Selenium, you can simulate opening a web browser and any web page and then scrape information from that particular page. It does this by utilizing a WebDriver, which is used for controlling web browsers.
If you don’t already have Selenium installed, you need to install both the Selenium library and a browser driver. Directions to do so are available in the Selenium documentation.
Due to its popularity, the ChromeDriver will also be used in this article, but the steps are the same regardless of the driver.
Now take a look at how you can use Python and Selenium to perform some common web scraping tasks:
Search for Products
To start using Selenium to simulate searching for Walmart products, you need to import it. You can do so with the following piece of code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import ServiceAfter importing Selenium, the next step is to use it to open a web browser, in this case, Chrome. However, you can choose whatever browser you prefer. Once you open a browser, the steps are the same regardless. Opening a browser is very straightforward, and you can do so by running the following piece of code as a Python script or from a Jupyter Notebook:
s=Service('/path/to/chromedriver')
driver = webdriver.Chrome(service=s)This simple piece of code will do nothing else except open Chrome. Here’s its output:
Now that you’ve opened Chrome, you need to go to Walmart’s home page, which you can do with the following code:
driver.get("https://www.walmart.com")
As you can see from the screenshot, this will simply open Walmart.com.
The next step is to manually look at the page’s source code with the Inspect tool. This tool enables you to inspect any specific element on a web page. With it, you can view (and even edit) the HTML and CSS of any web page.
Since you want to search for a product, you need to navigate to the search bar, right-click on it, and click Inspect. Locate the input tag with the type attribute equal to search. This is the search bar where you need to input your search term. Then you need to find the name attribute and look at its value. In this case, you can see that the name attribute has the value q:
In order to input a query in the search bar, you can use the following piece of code:
search = driver.find_element("name", "q")
search.send_keys("Gaming Laptops")This code will input the query Gaming Laptops, but you can input any phrase you want by replacing the term “Gaming Laptops” with any other term:
Please note that the previous code only inputted the search term into the search bar, and it didn’t actually search for it. In order to actually search for the term, you need the following line of code:
search.send_keys(Keys.ENTER)And this is what the output will look like:
Now you should get all the results for the search term you entered. And if you want to search for a different term, you only need to run the last two lines of code with the new search term you want.
Navigate to a Product’s Page and Scrape Product Info
Another common task you can perform with Selenium is to open the page of a specific product and scrape information about it. For instance, you can scrape the product’s name, description, price, rating, or reviews.
Let’s say that you’ve chosen a product whose information you want to be scraped. Begin by opening the product’s page, which you can do with the following code (assuming you’ve already installed and imported Selenium in the first example):
url = "https://www.walmart.com/ip/Acer-Nitro-5-15-6-Full-HD-IPS-144Hz-Display-11th-Gen-Intel-Core-i5-11400H-NVIDIA-GeForce-RTX-3050Ti-Laptop-GPU-16GB-DDR4-512GB-NVMe-SSD-Windows-11-Ho/607988022?athbdg=L1101"
driver.get(url)
Once the page is opened, you’ll need to utilize the Inspect tool. In essence, you need to navigate to any element whose information you want to scrape, right-click on it, and click Inspect. For example, once you inspect the product title, you’ll notice that the title is in an H1 tag. Since this is the only H1 tag on the page, you can get it with the following piece of code:
title = driver.find_element(By.TAG_NAME, "h1")
print(title.text)
>>>'Acer Nitro 5 , 15.6" Full HD IPS 144Hz Display, 11th Gen Intel Core i5-11400H, NVIDIA GeForce RTX 3050Ti Laptop GPU, 16GB DDR4, 512GB NVMe SSD, Windows 11 Home, AN515-57-5700'In a similar way, you can locate and scrape the price, rating, and number of reviews of the product:
price = driver.find_element(By.CSS_SELECTOR, '[itemprop="price"]')
print(price.text)
>>> '$899.00'
rating = driver.find_element(By.CLASS_NAME,"rating-number")
print(rating.text)
>>> '(4.6)'
number_of_reviews = driver.find_element(By.CSS_SELECTOR, '[itemprop="ratingCount"]')
print(number_of_reviews.text)
>>> '108 reviews'One important thing to keep in mind is that Walmart makes it extremely difficult to scrape data in the way shown here. This is because Walmart has antispam systems that actively try to block web scrapers. So if you find your web scraping efforts being consistently blocked, know that it’s probably not your fault, and there’s not much you can do about it. However, using the solution shown in the next section should prove much more effective.
Step-by-Step Instructions on Scraping Walmart with Bright Data
As you can see, scraping Walmart with Python and Selenium isn’t really straightforward. There’s a much easier way to scrape Walmart’s website, which involves the use of the Bright Data Web Scraper IDE. With this tool, you can more easily and efficiently perform the same tasks shown previously. Another advantage of using the Web Scraper IDE is that Walmart won’t be able to instantly block your efforts.
To use the Web Scraper IDE, you first need to sign up for a Bright Data account. Once you’re registered and logged in, you’ll see the following screen. Click on the Datasets & Web Scraper IDE button on the left:
That will take you to the following screen. From there, navigate to the My scrapers field:
This will show your existing web scrapers (if you have any) and will give you the option to develop a web scraper (IDE). Assuming that this is your first time using Bright Data, you won’t have any web scrapers, so you should click on Develop a web scraper (IDE):
You’ll be given the option to use one of the existing templates or to start the code from scratch. To scrape Walmart.com specifically, click on Start from scratch. That will open the Bright Data Web Scraper IDE:
The Web Scraper IDE consists of several different windows. On the upper-left part, you’ll find the Interaction code window. As its name suggests, you’ll use this window to interact with a website, including navigating and scrolling through the website, clicking buttons, as well as doing various other actions. Below that is the Parser code window, which will enable you to parse the HTML results from the interaction with the website. On the right side, you can preview and test your code.
Additionally, in the code settings at the top right corner you can choose between different worker types. You can toggle between a code (the default option) and a browser worker to navigate and crawl the data:
Now see how you can scrape the same data for the same product as was done with Python and Selenium. To begin, navigate to the product page, which you can do with the following line of code in the Interaction code window:
navigate('https://www.walmart.com/ip/Acer-Nitro-5-15-6-Full-HD-IPS-144Hz-Display-11th-Gen-Intel-Core-i5-11400H-NVIDIA-GeForce-RTX-3050Ti-Laptop-GPU-16GB-DDR4-512GB-NVMe-SSD-Windows-11-Ho/607988022?athbdg=L1101');Alternatively, you can use a changeable input parameter with navigate(input.url). In that case, add the URLs you want to scrape as an input, as shown here:
Following this, you need to collect the data you want, which you can do with this code:
let data = parse();
collect({
title: data.title,
price: data.price,
rating: data.rating,
number_of_reviews: data.number_of_reviews,
});The last thing you need to do is parse the HTML into structured data. You can do this with the help of the following piece of code in the Parser code window:
return {
title: $('h1').text().trim(),
price: $('span.inline-flex:nth-child(2) > span:nth-child(1)').text(),
rating: $('span.f7').text(),
number_of_reviews: $('a.f7').text(),
};Then you can obtain the data you want right in the Web Scraper IDE. Just click on the play button on the right side (or press Ctrl+Enter), and the results you need will be outputted. You can also download the data directly from the Web Scraper IDE:
In case you’ve chosen browser instead of code as a worker type, this is what the output would look like:
Getting results directly from the Web Scraper IDE is just one of the options you have to obtain the data. You can also set your delivery preferences in the My scrapers dashboard.
Finally, if you find web scraping too difficult even with the Web Scraper IDE, Bright Data provides an available Walmart products data set in the Dataset Marketplace, where there are numerous data sets available with the click of a button:
As shown here, using the Bright Data Web Scraper IDE is easier and more user-friendly than building your own web scraper with Python and Selenium. Even better, the Bright Data Web Scraper IDE makes it possible for beginners to start collecting data from Walmart. In contrast, you would need solid coding knowledge to scrape Walmart.com with Python and Selenium.
Apart from the ease of use, another impressive thing about the Bright Data Walmart Scraper is its scalability. You can scrape data about as many products as you need without any issues.
One key thing to point out when it comes to web scraping is privacy laws. A lot of corporations forbid or restrict what information you can scrape from their website. So if you’re building your own web scraper with Python and Selenium, you need to ensure that you’re not breaking any rules. However, when you use the Bright Data Web Scraper IDE, Bright Data takes on this responsibility and will ensure that the industry’s best practices and all privacy regulations are being followed.
Conclusion
This article discussed why you would want to scrape Walmart data, but more importantly, you actually learned how to scrape Walmart prices, names, number of reviews, and ratings of thousands of Walmart products.
As you learned, you can scrape this data using Python and Selenium; however, this method can be difficult and comes with challenges that can intimidate beginners. There are solutions that allow for much easier scraping of Walmart data, such as the Web Scraper IDE. It provides functions and code templates to scrape many popular websites, enables you to avoid CAPTCHAs, and is in compliance with data protection laws. Bright Data offers several advanced data collection and proxy services.
