--- title: "A Guide to Playwright Web Scraping in 2026" slug: playwright-web-scraping date: 2023-05-19T08:14:55+00:00 modified: 2026-08-24T13:48:39+00:00 permalink: https://brightdata.com/blog/how-tos/playwright-web-scraping type: blog --- [ Blog ](https://brightdata.com/blog "Blog") / [How Tos](https://brightdata.com/blog/how-tos) [How Tos](https://brightdata.com/blog/how-tos) # A Guide to Playwright Web Scraping in 2026 Web scraping with Playwright: A step-by-step guide to extracting data using this powerful tool. 17 min read [ ](https://brightdata.com/blog/authors/antonello-zanini) [Antonello Zanini Technical Writer ](https://brightdata.com/blog/authors/antonello-zanini) ![Web Scraping with Playwright blog image](https://media.brightdata.com/2023/05/Web-Scraping-with-Playwright.svg) In this blog post, you will learn: - What Playwright is - The key features it offers for web scraping - How to use it to extract data from a dynamic site - Advanced Playwright web scraping techniques - How it compares to other alternatives - Its main limitations and how to overcome them Let’s dive in! ## What Is Playwright? [Playwright](https://playwright.dev/) is a Node.js library for browser automation developed by Microsoft, designed to control Chromium-based, Firefox, and WebKit browsers. While primarily used for testing, it is also highly effective for web scraping JavaScript-heavy sites, SPAs, and any dynamic content page. Playwright is officially available in JavaScript, Python, C#, and Java, with additional language support provided by the community. This makes its API not only cross-browser but also easily adaptable across programming languages. Key features that make Playwright a modern browser automation tool for web scraping include headless mode, network interception, and browser context isolation. ## Playwright Web Scraping Features These are some of the most important Playwright features that support web scraping: - **Playwright Library API**: A powerful browser automation API to interact with web pages as in E2E tests. - **SPAs and PWA-ready**: Playwright provides all the necessary features to [scrape data from modern JavaScript-based Single Page Applications (SPAs) and Progressive Web Apps (PWAs)](https://hackernoon.com/how-to-scrape-modern-spas-pwas-and-ai-driven-dynamic-sites). - **Multiple locator methods**: Support for various selector syntaxes, including CSS selectors, XPath expressions, element text content, and more. - **Auto-waiting mechanism**: It automatically waits for elements to be visible, stable, and ready for interaction, reducing errors caused by partial data loads. - **Screenshots**: Capture full-page or element-specific screenshots to aid in debugging and visual data extraction. ## Web Scraping With Playwright: Step-By-Step Guide In this guided section, you will learn how to perform web scraping with Playwright. The target site will be the following [dynamic page from Scraping Course](https://www.scrapingcourse.com/javascript-rendering): By inspecting the “Network” tab in DevTools, you can see that this page loads data dynamically via an AJAX request: Then, the page renders that data in the browser using JavaScript. This makes it a perfect example of a dynamic site that requires a browser automation tool like Playwright for scraping. A simple HTTP client combined with an HTML parser would not be enough. In detail, the Playwright web scraping script will: - Navigate to the target page - Wait for the products to load - Scrape the required data - Export the data to CSV Time to build it! **Note:** Playwright provides the same browser automation API in Python, C#, and Java. While this guide uses JavaScript—the primary language supported by Playwright along with TypeScript—you can easily adapt the script we are going to build to any of the other supported programming languages. ### Step #1: Project Setup First, make sure you have the latest LTS version of Node.js installed on your machine. Otherwise, [download it from the official site](https://nodejs.org/en/download) and follow the wizard. Create a folder for your JavaScript project and enter it in the terminal: ```none mkdir playwright-web-scraper cd playwright-web-scraper ``` Launch the command below to initialize a Node.js project inside it: ```none npm init -y ``` Open the Playwright web scraping project with your favorite JavaScript IDE, and add `index.js` to it: Wonderful! You are ready to get started with Playwright. ### Step #2: Playwright Installation and Setup Install Playwright by adding the [`playwright`](https://www.npmjs.com/package/playwright) npm package to your project’s dependencies: ```none npm install playwright ``` This will take a while, so be patient. Now, keep in mind that each version of Playwright needs specific versions of browser binaries to work. Install them with: ```none npx playwright install ``` The [Playwright CLI](https://playwright.dev/docs/test-cli) will download Chromium, Firefox, and other required system dependencies for you. To get started with Playwright, import `chromium` in `index.js`: ```none const { chromium } = require("playwright"); ``` Playwright is typically initialized in Test mode, which provides a fully managed end-to-end test runner along with additional features. That is not necessary for web scraping. Instead, using the [Playwright library](https://playwright.dev/docs/library) as above is sufficient. This offers unified APIs for launching and interacting with browsers—such as the `chromium` object in this case. Then, create an `async` function where to create a browser instance: ```none const { chromium } = require("playwright"); (async () => { // launch a new Chromium browser instance const browser = await chromium.launch(); const page = await browser.newPage(); // browser automation for web scraping... // close the browser and release resources await browser.close(); })(); ``` `launch()` creates a new browser instance, [running in headless mode](/blog/proxy-101/what-is-a-headless-browser) by default. Then, the `newPage()` method opens a new page within that browser context. Finally, `close()` is used to properly shut down the browser and free up resources. To run the browser in headed mode—useful for debugging and visually tracking the script’s actions—set the `headless` option to `false`: ```none const browser = await chromium.launch({ headless: false, }); ``` Great! Your Playwright scraping setup is complete. ### Step #3: Visit the Target Page Use the `goto()` method to instruct the browser to connect to the target page: ```none await page.goto("https://www.scrapingcourse.com/javascript-rendering"); ``` Configure the script to work in headed mode. Next, add a breakpoint before the `close()` method and run the script in the debugger. This is what you should be seeing: Amazing! The Playwright scraping navigation logic works like a charm. ### Step #4: Inspect the Target Page Before writing the scraping logic, you need to familiarize yourself with the site’s structure and the DOM of the target page. To do so, open the website in incognito mode in your browser. Since you want to extract data from each product on the page, inspect a product’s HTML element. Right-click on the first product and select the “Inspect” option. This will open the DevTools panel below: Take a close look at the HTML of the product element. You will notice that all elements have a data-testid attribute. These attributes are mostly used for testing and tend to remain stable over time, making them reliable selectors for web scraping. On this page, notice that all products are contained within `[data-testid="product-grid"]`. In particular, each product is represented by a `[data-content="product-item"]` node. Each of these includes: - The product URL in `[data-testid="product-link"]`. - The product image in `[data-testid="product-image"]`. - The product name in `[data-content="product-name"]`. - The product price in `[data-testid="product-price"]`. Perfect! You now have all the necessary information to do web scraping with Playwright. ### Step #5: Implement the Scraping Logic Since you want to scrape a list of products, first initialize an array where to store the scraped data: ```none products = [] ``` Remember, the products are loaded dynamically via AJAX. Initially, the HTML elements for all products are present on the page, but they do not contain any data: To ensure the AJAX request has been completed and the data has been rendered, you can wait for the `href` attribute of the `` tag inside `[data-content="product-item"]` to be populated. Here is how you can achieve that ```none await page.waitForSelector('[data-content="product-item"] a[href]:not([href=""])'); ``` The `waitForSelector()` method waits for the the specified selector to be valid on the page’s DOM. The page renders all products at once, so waiting for the first one to appear is enough. On more complex pages, you may need to customize this logic to ensure all products are fully loaded before proceeding. Now, use the `locator()` method to select all product items on the page: ```none const productItemsLocator = page.locator('[data-content="product-item"]'); ``` The `locator()` method applies the specified locator strategy and can reference one or more DOM elements. Playwright supports both [CSS selectors and XPath expression](/blog/web-data/xpath-vs-css-selectors), and it auto-detects them if you omit the `css=` or `xpath=` prefix. **Note**: Locators are more efficient and stable for interacting with the page compared to traditional element selector methods (e.g., `$()` and `$$()`), [which are currently deprecated](https://playwright.dev/docs/api/class-page#page-query-selector-all). Use the `all()` method to access the list of elements in the locator and iterate through them: ```none for (const productLocator of await productItemsLocator.all()) { // scraping logic... } ``` Now that you have the current product element, you can extract the required data (URL, image, name, price) as below: ```none const urlElement = await productLocator.locator( '[data-testid="product-link"]' ); const url = await urlElement.getAttribute("href"); const imageElement = await productLocator.locator( '[data-testid="product-image"]' ); const image = await imageElement.getAttribute("src"); const nameElement = await productLocator.locator( '[data-content="product-name"]' ); const name = await nameElement.textContent(); const priceElement = await productLocator.locator( '[data-testid="product-price"]' ); const price = await priceElement.textContent(); ``` As the names suggest: - `textContent()` retrieves the text inside an element; - `getAttribute()` accesses the data within an element’s attribute. Once you have extracted the data, create an object for the product and add it to the `products` array: ```none const product = { "url": url, "image": image, "name": name, "price": price, }; products.push(product); ``` Terrific! The data extraction logic is finalized. ### Step #6: Export to CSV Node.js provides the tools to work with CSV files, but using a specialized library like [`csv-writer`](https://www.npmjs.com/package/csv-writer) makes the process much easier. To add it to your project’s dependencies, run the following command: ```none npm install csv-writer ``` Then, import it in your `index.js` file: ```none const createCsvWriter = require("csv-writer").createObjectCsvWriter; ``` Next, use it to populate a CSV file with the scraped data: ```none const csvWriter = createCsvWriter({ path: "products.csv", header: [ { id: "url", title: "URL" }, { id: "image", title: "Image" }, { id: "name", title: "Name" }, { id: "price", title: "Price" }, ], }); // write the products data to the CSV await csvWriter.writeRecords(products); ``` Fantastic! You started with raw data from a webpage, and now you have it neatly organized in a CSV file. ### Step #7: Put It All Together Your final Playwright web scraping script should contain: ```none const { chromium } = require("playwright"); const createCsvWriter = require("csv-writer").createObjectCsvWriter; (async () => { // launch a new Chromium browser instance const browser = await chromium.launch({ headless: false, // comment out in production }); const page = await browser.newPage(); // connect to the target page await page.goto("https://www.scrapingcourse.com/javascript-rendering"); /// where to store the scraped data const products = []; // wait for the procuts to be rendered await page.waitForSelector( '[data-content="product-item"] a[href]:not([href=""])' ); // get the list of product items const productItemsLocator = page.locator('[data-content="product-item"]'); // iterate over all product elements for (const productLocator of await productItemsLocator.all()) { // scraping logic const urlElement = await productLocator.locator( '[data-testid="product-link"]' ); const url = await urlElement.getAttribute("href"); const imageElement = await productLocator.locator( '[data-testid="product-image"]' ); const image = await imageElement.getAttribute("src"); const nameElement = await productLocator.locator( '[data-content="product-name"]' ); const name = await nameElement.textContent(); const priceElement = await productLocator.locator( '[data-testid="product-price"]' ); const price = await priceElement.textContent(); // populate a new product object // and add it to the array const product = { "url": url, "image": image, "name": name, "price": price, }; products.push(product); } // configurations for the CSV output file const csvWriter = createCsvWriter({ path: "products.csv", header: [ { id: "url", title: "URL" }, { id: "image", title: "image" }, { id: "name", title: "name" }, { id: "price", title: "price" }, ], }); // write the products data to the CSV await csvWriter.writeRecords(products); // close the browser and release resources await browser.close(); })(); ``` Way to go! In less than 100 lines of code, you just built a script to perform web scraping with Playwright. Time to test it by launching this command: ```none node index.js ``` The script will take a while, then a `products.csv` file will appear in the project’s directory. Open it and you will see: Et voilà! The output CSV file stores the same data as on the target page, but in a structured format. ## Playwright Scraping: Advanced Features You just learned the basics of web scraping using Playwright through a guided tutorial. Still, many more advanced scenarios are possible. Time time to explore them! ### User Interaction Over the years, web pages have become more complex than ever. They now feature advanced navigation and interaction patterns such as dynamic dropdowns, “Load More” buttons, and infinite scrolling. Find out more in our guide on [how to scrape sites with complex navigation](/blog/web-data/scraping-complex-navigation). Playwright is designed to handle most of those scenarios through dedicated methods. For example, you can select a “Load More” button and click it with: ```none await page.locator(':has-text("Load More")').click(); ``` The `click()` method simulates a click on the specified element, triggering any associated events just as if a real user had clicked it. **Note**: Various checks are performed before interacting with elements. Playwright automatically waits for these conditions to be met before executing an action. If the conditions are not satisfied within the specified `timeout`, the action fails with a `TimeoutError`. For example, when calling `page.click()`, Playwright ensures that: - The selector resolves to exactly one element - The element is visible - The element is stable (not animating or has completed its animation) - The element can receive events (not hidden behind other elements) - The element is enabled Next, you can fill out and submit a form with: ```none await page.fill('input[name="email"]', "[email protected]"); await page.fill('input[name="password"]', "mysecurepassword"); await page.click('button[type="submit"]'); ``` The `fill()` method sets the value of an input field, while `click()` is used to submit the form by clicking the submit button. Additionally, you can select an option from a dropdown using `selectOption()`: ```none await page.selectOption("#country", "USA"); ``` `selectOption()` selects a dropdown item by matching it with the provided value, label, or index. The effect on the page will be the same as a user manually choosing the specified option from the list. ### JavaScript Script Execution Some sites require complex interactions that cannot be handled with standard Playwright methods alone. In such cases, you can run custom JavaScript directly on the page to simulate user behavior. For example, to scroll down the page, you can execute the following script: ```none window.scrollTo(0, document.body.scrollHeight); ``` Playwright enables you to launch JavaScript scripts on the page via the `evaluate()` method: ```none await page.evaluate(() => { window.scrollTo(0, document.body.scrollHeight); }); ``` This injects and runs JavaScript inside the page, scrolling to the bottom just like a real user would. Custom scripts like that help you interact with pages that require particularly complex UI behaviors. ### User-Agent Customization `User-Agent` is one of the most important [HTTP headers for web scraping](/blog/web-data/http-headers-for-web-scraping). By default, Playwright uses the `User-Agent` string of the controlled browser, which typically looks like this: ```none "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/133.0.6943.16 Safari/537.36" ``` The problem is that “HeadlessChrome” in the string can flag your requests as automated because real users do not run a browser without a GUI. To avoid detection, override the `User-Agent` like this: ```none const browser = await chromium.launch(); // set a custom user agent const customUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.6834.160 Safari/537.36"; const context = await browser.newContext({ userAgent: customUserAgent, }); // create a new page within the customized context const page = await context.newPage(); ``` Note that you must set the `User-Agent` header in a dedicated browser context and use that to open a new page. ### Request Interception On most dynamic sites, data is retrieved through AJAX requests instead of being directly embedded in the initial HTML. Playwright allows you to intercept these requests and inspect their responses through the [Network API](https://playwright.dev/docs/network). For example, intercept a request to `/ajax/products/json` and print its data with the following code: ```none await page.route("**/ajax/products/json", async (route, request) => { // access the requests's response and print it const response = await route.fetch(); const jsonData = await response.json(); console.log("intercepted data:", jsonData); // continue the request without modification await route.continue(); }); ``` `route()` intercepts the AJAX request and enables you to access its responses. Intercepted responses usually contain the data that is dynamically loaded onto the product page. By accessing this data directly from the site’s server, you can retrieve information without waiting for the page to fully render. With this method, you can also modify the behavior of these requests or even block them entirely. ### Proxy Integration To protect your IP and identity while using Playwright, you can integrate a [proxy server](/blog/proxy-101/what-is-proxy-server) into your script. This helps mask your real IP address and bypass rate limiters. Use the following code to configure Playwright to route traffic through a proxy: ```none const browser = await chromium.launch({ proxy: { server: "http://your-proxy-server.com:PORT", // proxy server URL username: "your-username", // optional proxy username credential password: "your-password", // optional proxy password credential }, }); ``` For more guidance, refer to our [Playright proxy integration guide](/integration/playwright). If you are seeking reliable proxies, keep in mind that Bright Data’s proxy network is trusted by Fortune 500 companies and 50,000+ customers worldwide. This extensive network includes: - [Datacenter proxies](/proxy-types/datacenter-proxies): 1,300,000+ datacenter IPs. - [Residential proxies](/proxy-types/residential-proxies): 400M+ residential IPs in more than 195 countries. - [ISP proxies](/proxy-types/isp-proxies): 1,300,000+ ISP IPs. ## Playwright Alternatives for Web Scraping The two most popular Playwright alternatives when it comes to web scraping are: - [**Puppeteer**](/blog/how-tos/web-scraping-puppeteer): A JavaScript browser automation library maintained by Google. It is well-suited for tasks such as automated testing, scraping, and generating screenshots. Puppeteer provides a powerful API for controlling headless Chrome, with [Firefox support that has been added recently](https://hacks.mozilla.org/2024/08/puppeteer-support-for-firefox/). - [**Selenium**](/blog/how-tos/using-selenium-for-web-scraping): A widely-used browser automation tool that supports multiple browsers, including Chrome, Firefox, Internet Explorer, and Safari. It is used for web scraping, automation, and testing, and has been around for a lot of years. Although it offers broad language support, it is typically slower and less modern compared to Playwright and Puppeteer. You can compare these [browser automation solutions](/blog/web-data/best-headless-browsers) in the following table: **Feature****Playwright****Puppeteer****Selenium****GitHub Stars**69k stars89.6k stars31.5k stars**Developed By**MicrosoftGoogleCommunity and Sponsors**Browser Support**Chromium-based browsers, Firefox, WebKitChromium-based browsers, FirefoxChromium-based browsers, Firefox, IE, WebKit**Automation API**RichQuite richFocused on the basics**Developer Experience**GreatGoodDecent**Documentation**ExcellentExcellentDecent**Programming Languages**JavaScript, Python, C#, JavaJavaScriptJava, Python, C#, Ruby, JavaScript**Community**MediumLargeVery large**Performance**FastFastMedium/SlowFor more information, refer to these comparison articles: - [Playwright vs. Selenium Guide](/blog/web-data/playwright-vs-selenium) - [Puppeteer vs. Playwright: Comparison Guide](/blog/web-data/puppeteer-vs-playwright) ## Biggest Playwright Limitations Playwright is a feature-rich tool for web scraping, but it comes with a few key limitations: - **Performance issues**: Browsers, even in headless mode, consume significant system resources—like RAM and CPU. Running multiple instances can slow down even high-performance servers and significantly impact efficiency. - **Instability**: The connection with local browsers can become unstable over time, especially in long-running sessions or when managing multiple tabs. - **Anti-Bot systems**: Automated browsers have subtle configuration differences compared to real user sessions. These can trigger anti-bot detection, leading to CAPTCHAs, IP bans, and other blocks. To reduce detection risks, explore our tutorial on the [Playwright Stealth plugin](/blog/how-tos/avoid-bot-detection-with-playwright-stealth) and learn [how to bypass CAPTCHAs with Playwright](/blog/web-data/bypass-captchas-with-playwright). Yet, those tricks may not be enough. Switching between browser automation libraries like Selenium or Puppeteer will not resolve these challenges. The reason is that they stem from the browser itself, not the library. Browsers require high system resources, are prone to detection, and need special configurations for scraping at scale. The real solution lies in using a cloud-based browser optimized for web scraping, such as [Playwright Web Scraping Browser](/products/scraping-browser/playwright). This scalable solution includes built-in anti-bot bypass mechanisms, automatic IP rotation, browser fingerprinting management, CAPTCHA solving, and automated retries—ensuring reliable and efficient data extraction. That is everything you need for seamless and effective web scraping with Playwright! ## Conclusion In this article, you learned why Playwright is a powerful tool for web scraping, especially when handling dynamic sites. With the step-by-step guide provided here, you now understand how to perform web scraping against JavaScript-rendered content. You also explored more advanced Playwright web scraping scenarios. However, no matter how well you configure your script, headless browsers consume significant system resources and can still be detected by anti-bot solutions. This can lead to performance issues and blocks. Avoid these issues with a dedicated browser optimized for web scraping, such as [Scraping Browser](/products/scraping-browser). Create a free Bright Data account today to explore our proxies and scraping solutions! Contact usStart free trial No credit card required [ ](https://www.linkedin.com/in/antonello-zanini/) Antonello Zanini Technical Writer 5.5 years experience Antonello Zanini is a technical writer, editor, and software engineer with 5M+ views. Expert in technical content strategy, web development, and project management. Expertise Web Development Web Scraping AI Integration [ View all articles ](https://brightdata.com/blog/authors/antonello-zanini) Table of Contents Dedicated Scraper APIs & No-Code Scrapers Over 1000 scrapers for all popular domains. Simplify your web scraping. [See pricing](/pricing/web-scraper "See pricing") Just want data? Skip scraping. Hundreds of ready-to-use datasets from all popular domains. [See pricing](/pricing/datasets "See pricing") [ ](https://news.ycombinator.com/submitlink?t=A+Guide+to+Playwright+Web+Scraping+in+2026&u=https://brightdata.com/blog/how-tos/playwright-web-scraping) [ ](https://www.linkedin.com/shareArticle?mini=true&title=A+Guide+to+Playwright+Web+Scraping+in+2026&url=https://brightdata.com/blog/how-tos/playwright-web-scraping) [ ](http://www.reddit.com/submit?title=A+Guide+to+Playwright+Web+Scraping+in+2026&url=https://brightdata.com/blog/how-tos/playwright-web-scraping) ## You might also be interested in [ ](https://brightdata.com/blog/how-tos/avoid-bot-detection-with-playwright-stealth "Avoid Bot Detection With Playwright Stealth") [How Tos Antonello Zanini Technical Writer ### Avoid Bot Detection With Playwright Stealth Outsmart bot detection with the Playwright Stealth plugin. Our quick guide shows you how to integrate Stealth into your Playwright scripts, keeping your web automation smooth and undetectable. 20-Feb-2024 9 min read ](https://brightdata.com/blog/how-tos/avoid-bot-detection-with-playwright-stealth) [ ](https://brightdata.com/blog/how-tos/web-scraping-php "Web Scraping with PHP: a Step-By-Step Guide") [How Tos Aniket Bhattacharyea ### Web Scraping with PHP: a Step-By-Step Guide Learn how to easily create and program your own simple web scraper in PHP, from scratch. 17-Jul-2023 2 min read ](https://brightdata.com/blog/how-tos/web-scraping-php) [ ](https://brightdata.com/blog/how-tos/web-scraping-with-r "A Hands-On Guide to Web Scraping in R") [How Tos Aviv Besinsky ### A Hands-On Guide to Web Scraping in R In this tutorial, we’ll go through all the steps involved in web scraping in R with rvest with the goal of extracting product reviews from one publicly accessible URL from Amazon’s website. 09-Jan-2023 7 min read ](https://brightdata.com/blog/how-tos/web-scraping-with-r)