` (for document section) and `

` (for paragraph) being some of their most common “children.” In the snippet above, you can see “attributes” associated with each HTML “element”: `lang`, `class`, and `style` are pre-built; the attributes starting with `data-` are custom to Amazon. `class` is of particular interest for web scraping, together with the `id` attribute, as they allow us to target a group of elements and a specific element, respectively. This was originally for styling in CSS. CSS provides the styling of the web page. From coloring to positioning and sizing, you can select any HTML element and assign new values to its styling properties. You can also apply CSS styling inline within the HTML element with the `style` attribute, as you saw in the snippet above: ```none ``` In pure CSS, this would be written as: ```none body {padding-bottom: 0px;} ``` Here, body is the “selector”, `padding-bottom` is the “property”, and `0px` is the “value”. Any `tag`, `class`, or `id` can be used as a CSS selector. Users can dynamically interact with the content displayed on the web page via functionalities provided by the [JavaScript](https://www.w3schools.com/html/html_scripts.asp) programming language through the `script` tag. After a user interaction, the displayed content may change and new content may appear; advanced web scrapers can mimic user interactions, as we’ll discuss later. ### Understanding DevTools Major web browsers provide built-in developer tools that allow for the collection and live updating of technical information on a web page for logging, debugging, testing, and performance analysis. For this tutorial, we’ll be using [Chrome’s DevTools](https://developer.chrome.com/docs/devtools/). Developer Tools are accessible from the upper-right corner of the browser, in More Tools: Process of opening Developer Tools in ChromeIn DevTools, you can scroll through the raw HTML in the Elements tab. As you scroll through any of the HTML lines, you’ll see the corresponding element rendered in the web page highlighted in blue: Process of accessing rendered elements in the web page from Developer Tools in ChromeConversely, you can click on the icon in the top-left corner and select any rendered element from the web page to be redirected to its raw HTML counterpart, again highlighted in blue. These two processes are all you need to extract the CSS Descriptors for our hands-on tutorial. ## Deep-Dive into Web Scraping in R: Tutorial In this section, we’ll explore how to web scrape the [Amazon URL](https://www.amazon.co.uk/Xbox-Elite-Wireless-Controller-2/dp/B07SR4R8K1/) to [extract product reviews](/products/web-scraper/amazon/reviews). ### Prerequisites Ensure you have the following installed in your Rstudio environment: - R = 4.2.2 - rvest = 1.0.3 - tidyverse = 1.3.2 ### Interactively Exploring the Web Page You can use Chrome’s DevTools to explore the HTML of your URL and create a list of all the classes and IDs of the HTML elements that contain the information we’re interested in scraping, i.e., the product reviews: Customer review from the Amazon product page selected with DevTools in ChromeEach customer review belongs to a `div` with an `id` in the format: ```none customer_review_$INTERNAL_ID. ``` The HTML content of the div corresponding to the customer review in the screenshot above is the following: ```none

Very good controller if a little overpriced

In all honesty I'm not sure why the price is quite as high ….

…

… …

…

``` Each piece of content you’re interested in for the customer reviews has its unique class: `review-title-content` for the title, `review-text-content` for the body, and `review-rating` for the rating. You could check that the class is unique in the document and directly use the “simple selector”. A more foolproof approach is to use the CSS Descriptor instead, which will stay unique even if the class is assigned to new elements in the future. Simply retrieve the CSS Descriptor by right-clicking on the element in the DevTools and selecting Copy Selector: Process of copying the CSS selector for an element from Developer Tools in ChromeYou can define your three selectors as: - `customer_review-R2U9LWUSIPY0GS > div:nth-child(2) > a.a-size-base.a-link-normal.review-title.a-color-base.review-title-content.a-text-bold > span` for the title - `customer_review-R2U9LWUSIPY0GS > div.a-row.a-spacing-small.review-data > span > div > div.a-expander-content.reviewText.review-text-content.a-expander-partial-collapse-content > span` for the body - `customer_review-R2U9LWUSIPY0GS > div:nth-child(2) > a:nth-child(1) > i.review-rating > span` for the rating \*`.review-rating` was manually added for better consistency. #### CSS Selector vs. XPath for Web Scraping For this tutorial, we’ve decided to use a CSS selector to identify elements for web scraping. The other common approach is using XPath, i.e., the XML Path, which identifies an element through its full path in the DOM. You can extract the full XPath following the same procedure as for the CSS selector. For example, the review title is: ```none /html/body/div[2]/div[3]/div[6]/div[32]/div/div/div[2]/div/div[2]/span[2]/div/div/div[3]/div[3]/div/div[1]/div/div/div[2]/a[2]/span ``` A CSS selector is slightly faster, while XPath has slightly better backward compatibility. Outside of these small differences, selecting one over the other depends more on personal preferences than technical implications. ### Programmatically Extracting Information from the Web Page While we could use the Console directly to start exploring how to web scrape the URL, we’ll instead create a script for traceability and reproducibility and run it via the Console using the `source()` command. After creating the script, the first step is to load the installed libraries: ```none library(”rvest”) library(”tidyverse”) ``` Then, you can programmatically extract the content you’re interested in as follows. First, create a variable where you’ll store the URL to search: ```none HtmlLink <- "https://www.amazon.co.uk/Xbox-Elite-Wireless-Controller-2/dp/B07SR4R8K1/ref=sr_1_1_sspa?crid=3F4M36E0LDQF3" ``` Next, extract the Amazon Standard Identification Number (ASIN) from the URL to use it as a unique product ID: ```none ASIN <- str_match(HtmlLink, "/dp/([A-Za-z0-9]+)/")[,2] ``` Using [RegEx](https://cran.r-project.org/web/packages/stringr/vignettes/regular-expressions.html) to clean the text extracted via web scraping is common and recommended to ensure data quality. Now, download the HTML content of the web page: ```none HTMLContent <- read_html(HtmlLink) ``` The read\_html() function is part of the xml2 package. If you `print()` the content, we’ll see it matches the raw HTML structure previously analyzed: ```none {html_document} [1] n` ” connector from the string; you can also take advantage of the html\_nodes() and html\_text() functionalities of rvest to save the HTML content in separate objects. The following commands will extract the review titles: ```none review_title <- HTMLContent %>% html_nodes("div:nth-child(2) a.a-size-base.a-link-normal.review-title.a-color-base.review-title-content.a-text-bold span") %>% html_text() ``` An example of an entry in `review_title` is “Very good controller if a little overpriced.”. The code below will extract the review body: ```none review_body <- HTMLContent %>% html_nodes("div.a-row.a-spacing-small.review-data span div div.a-expander-content.reviewText.review-text-content.a-expander-partial-collapse-content span") %>% html_text() ``` An example of an entry in `review_body` begins with “In all honesty, I’m not sure why the price…”. And you can use the following commands to extract the review rating: ```none review_rating <- HTMLContent %>% html_nodes("div:nth-child(2) a:nth-child(1) i.review-rating span") %>% html_text() ``` An example of an entry in `review_rating` is “4.0 out of 5 stars”. To improve the quality of this variable, extract the rating “4.0” only and convert it to int: ```none review_rating <- substr(review_rating, 1, 3) %>% as.integer() ``` The pipe functionality %>% is provided by the magrittr toolkit. Now it’s time to export the scraped content inside a tibble for data analysis. [tibble](https://tibble.tidyverse.org/) is an R package that also belongs to the tidyverse collection and is used to manipulate and print data frames. ```none df <- tibble(review_title, review_body, review_rating) ``` The output dataframe is as follows: The scraped data as outputted by tibbleFinally, it’s a good idea to refactor the code into the function `scrape_amazon` <- `function(HtmlLink)` to comply with best practices and best prepare the code for scaling to multiple URLs. ## Scaling to Multiple URLs After the web scraping template has been created, you can create a list of URLs for all top competitors’ products on Amazon via [web crawling and scraping](/blog/leadership/web-crawling-vs-web-scraping). When scaling to multiple URLs to productionize the solution, you need to outline the application’s technical requirements. Having well-defined technical requirements will ensure you correctly support business requirements and seamlessly integrate with your existing systems. Depending on the specific technical requirements, the scraping function needs to be updated to support a combination of the following: - Real-time or batch process - Output format(s), such as JSON, NDJSON, CSV, or XLSX - Output target(s), such as email, API, webhook, or cloud storage We’ve already mentioned that you can extend rvest with [polite](https://dmi3kno.github.io/polite/) to scrape multiple web pages. polite creates and manages a web harvesting session via the use of three main functionalities, in full compliance with the web host’s robots.txt file and with built-in rate limiting and response caching: - `bow()` creates the scraping session for a specific URL, i.e., it introduces you to the web host and asks for permission to scrape. - `scrape()` accesses the HTML of the URL; you can pipe the function to `html_nodes()` and `html_text()` from rvest to retrieve specific content. - `nod()` updates the session’s URL to the next page, without the need to recreate a session. Quoting directly from their website, “The three pillars of a polite session are **seeking permission, taking slowly and never asking twice.**” ## Next Step: Pre-Built vs. Self-Built? To develop a state-of-the-art web scraper that can extract good data for a business, a few capabilities need to be available: - A team of data specialists with expertise in web data extraction - A team of DevOps engineers with expertise in proxy management and anti-bot circumvention to allow getting past CAPTCHAs and unlocking less publicly accessible websites - A team of data engineers with expertise in creating infrastructure for real-time and batch data extraction - A team of legal experts to understand data protection legal requirements for privacy (such as GDPR and CCPA) Content for the web comes in varied formats, and it is hard to find two websites with the exact same structure. The more complex a website is and the more features and data there are to scrape, the more advanced the programming knowledge required will be, not to mention the additional time and resources needed for the solution. Typically, you’d want to at least implement the following advanced functionalities: 1. **Minimize the chances of CAPTCHAs and bot detection:** A simple approach here is adding a random sleep() to avoid overloading web servers and regular request patterns. A more effective approach is using a user\_agent or proxy server to spread requests among different IPs. 2. **Scrape Javascript-powered websites:** In our Amazon example, the URL does not change when selecting a specific product variant. This is acceptable for scraping reviews, as they are shared, but not for scraping product specifications. To mimic user interactions in dynamic web pages, you could use a tool such as [RSelenium](https://cran.r-project.org/web/packages/RSelenium/index.html) to automate the navigation of the web browser. When wanting to gain access to web data with limited resources, ensure data quality, or unlock more advanced use cases, a pre-built web scraper can be the right choice. [Bright Data’s Web Scraper](/products/web-scraper) provides templates for many websites powered by state-of-the-art functionalities, including a much more advanced implementation of the demoed [Amazon Scraper](/products/web-scraper/amazon)! Don’t want to deal with data collection? Check out [our datasets](/products/datasets), free samples available. Contact usStart free trial No credit card required Aviv Besinsky Aviv is a lead product manager at Bright Data. He has been a driving force in taking data collection technology to the next level - developing technological solutions in the realms of data unblocking, static proxy networks, and more. Sharing his data crawling know-how is one of his many passions. [ View all articles ](https://brightdata.com/blog/authors/aviv-besinsky) Table of Contents Dedicated Scraper APIs & No-Code Scrapers Over 1000 scrapers for all popular domains. Simplify your web scraping. [See pricing](/pricing/web-scraper "See pricing") Just want data? Skip scraping. Hundreds of ready-to-use datasets from all popular domains. [See pricing](/pricing/datasets "See pricing") [ ](https://news.ycombinator.com/submitlink?t=A+Hands-On+Guide+to+Web+Scraping+in+R&u=https://brightdata.com/blog/how-tos/web-scraping-with-r) [ ](https://www.linkedin.com/shareArticle?mini=true&title=A+Hands-On+Guide+to+Web+Scraping+in+R&url=https://brightdata.com/blog/how-tos/web-scraping-with-r) [ ](http://www.reddit.com/submit?title=A+Hands-On+Guide+to+Web+Scraping+in+R&url=https://brightdata.com/blog/how-tos/web-scraping-with-r) ## You might also be interested in [ ](https://brightdata.com/blog/ai/openhuman-with-bright-data "Production-Ready Web Access in OpenHuman Through the Bright Data CLI") [AI Antonello Zanini Technical Writer ### Production-Ready Web Access in OpenHuman Through the Bright Data CLI Integrate Bright Data CLI with OpenHuman to enable production-ready web access and data collection for AI agents. 09-Sep-2026 12 min read ](https://brightdata.com/blog/ai/openhuman-with-bright-data) [ ](https://brightdata.com/blog/ai/minimax-m3-with-bright-data "Giving self-hosted MiniMax M3 agents live web access with Bright Data") [AI Satyam Tripathi Technical Writer ### Giving self-hosted MiniMax M3 agents live web access with Bright Data Self-hosted MiniMax M3 agents get live web access using Bright Data’s search and scraping tools. Bypass blocks and CAPTCHAs. 09-Sep-2026 54 min read ](https://brightdata.com/blog/ai/minimax-m3-with-bright-data) [ ](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax "Multimodal Web Scraping with MiniMax") [Web Data Antonello Zanini Technical Writer ### Multimodal Web Scraping with MiniMax Pair Bright Data Web Unlocker with MiniMax M3 vision to extract structured data from images and web page screenshots. 09-Sep-2026 4 min read ](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax)