A Light in the ...
£51.77
` tag. The dot (`.`) at the beginning of the XPath expression specifies to start searching from the `` tag rather than the default starting point. The next line uses the `cssselect` method to extract the price from a `` tag with the class `price_color`. Since `cssselect` returns a list, indexing (`[0]`) accesses the first element, and `text_content()` retrieves the text inside the element. Each extracted title and price pair is then appended to the `books` list as a dictionary, which can be easily stored in a JSON file.
Now that you’ve completed the web scraping process, it’s time to save this data locally. Open your script file and input the following code:
```none
with open("books.json", "w", encoding="utf-8") as file:
json.dump(books ,file)
```
In this code, a new file named `books.json` is created. This file is populated using the `json.dump` method, which takes the `books` list as the source and a `file` object as the destination.
You can test this script by opening the terminal and running the following command:
```none
python static_scrape.py
```
This command generates a new file in your directory with the following output:
All the code for this script is available on [GitHub](https://gist.github.com/vivekthedev/c1c5f0fb0e23cabfa3fa5c364b939f7c).
### Parsing Dynamic HTML Content
Scraping dynamic web content is trickier than scraping static content because JavaScript renders the data continuously rather than all at once. To help scrape dynamic content, you use a browser automation tool called [Selenium](https://www.selenium.dev/), which lets you create and run a browser instance and control it programmatically.
To install Selenium, open the terminal and run the following command:
```none
pip install selenium
```
YouTube is a great example of content rendered using JavaScript. When you visit any channel, only a limited number of videos load initially, with more videos appearing as you scroll down. Here, you scrape data for the top hundred videos from the [freeCodeCamp.org YouTube channel](https://www.youtube.com/c/Freecodecamp) by emulating keyboard presses to scroll the page.
To begin, inspect the HTML code of the web page. When you open **Dev Tools**, you’ll see the following:
The following code identifies the elements responsible for displaying the video title and link:
```none
GitHub Advanced Security Certification – Pass the Exam!
```
The video title is within the `yt-formatted-string` tag with the ID `video-title`, and the video link is located in the `href` attribute of the `a` tag with the ID `video-title-link`.
Once you identify what you want to scrape, create a new file named `dynamic_scrape.py` and add the following code, which imports all the modules required for the script:
```none
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from lxml import html
from time import sleep
import json
```
Here, you begin by importing `webdriver` from `selenium`, which creates a browser instance that you can control programmatically. The next lines import `By` and `Keys`, which select an element on the web and perform some keystrokes on it. The `sleep` function is imported to pause the program execution and wait for the JavaScript to render content on the page.
With all the imports sorted out, you can define the driver instance for the browser of your choice. This tutorial uses [Chrome](https://www.selenium.dev/documentation/webdriver/browsers/chrome/), but Selenium also supports [Edge](https://www.selenium.dev/documentation/webdriver/browsers/edge/), [Firefox](https://www.selenium.dev/documentation/webdriver/browsers/firefox/), and [Safari](https://www.selenium.dev/documentation/webdriver/browsers/safari/).
To define the driver instance for the browser, append the script with the following code:
```none
URL = "https://www.youtube.com/@freecodecamp/videos"
videos = []
driver = webdriver.Chrome()
driver.get(URL)
sleep(3)
```
Similar to the previous script, you declare a `URL` variable containing the web URL that you want to scrape and a `videos` variable that stores all the data as a list. Next, a `driver` variable is declared (*ie* a `Chrome` instance) that you use when you interact with the browser. The `get()` function opens the browser instance and sends a request to the specified `URL`. After that, you call the `sleep` function to wait for three seconds before accessing any element on the web page to make sure all the HTML code gets loaded in the browser.
As mentioned before, YouTube uses JavaScript to load more videos as you scroll to the bottom of the page. To scrape data from a hundred videos, you must programmatically scroll to the bottom of the page after opening the browser. You can do this by adding the following code to your script:
```none
parent = driver.find_element(By.TAG_NAME, 'html')
for i in range(4):
parent.send_keys(Keys.END)
sleep(3)
```
In this code, the `` tag is selected using the `find_element` function. It returns the first element matching the given criteria, which in this case is the `html` tag. The `send_keys` method simulates pressing the `END` key to scroll to the bottom of the page, triggering more videos to load. This action is repeated four times within a `for` loop to ensure enough videos are loaded. The `sleep` function pauses for three seconds after each scroll to allow the videos to load before scrolling again.
Now that you have all the data needed to begin the scraping process, it’s time to use lxml with cssselect to select the elements you want to extract:
```none
html_data = html.fromstring(driver.page_source)
videos_html = html_data.cssselect("a#video-title-link")
for video in videos_html:
title = video.text_content()
link = "https://www.youtube.com" + video.get("href")
videos.append( {"title": title, "link": link} )
```
In this code, you pass the HTML content from the driver’s `page_source` attribute to the `fromstring` method, which builds a hierarchical tree of the HTML. Then, you select all `` tags with the ID `video-title-link` using CSS selectors, where the `#` sign indicates selection using the tag’s ID. This selection returns a list of elements that satisfy the given criteria. The code then iterates over each element to extract the title and link. The `text_content` method retrieves the inner text (the video title), while the `get` method fetches the `href` attribute value (the video link). Finally, the data is stored in a list called `videos`.
At this point, you’re done with the scraping process. The next step involves storing this scraped data locally in your system. To store the data, append the following code in the script:
```none
with open('videos.json', 'w') as file:
json.dump(videos, file)
driver.close()
```
Here, you create a `videos.json` file and use the `json.dump` method to serialize the videos list into JSON format and write it to the file object. Finally, you call the close method on the driver object to safely close and destroy the browser instance.
Now, you can test your script by opening the terminal and running the following command:
```none
python dynamic_scrape.py
```
After running the script, a new file named `videos.json` is created in your directory:
All the code for this script is also available on [GitHub](https://gist.github.com/vivekthedev/36489fbaf896eb7c06ebb9350dec298a).
### Using lxml with Bright Data Proxy
While web scraping is a great technique for automating data collection from various sources, the process isn’t without its challenges. You have to deal with anti-scraping tools implemented by websites, rate-limiting, geoblocking, and a lack of anonymity. [Proxy servers](/proxy-types/proxy-servers) can help with these issues by acting as intermediaries that mask the user’s IP address, allowing scrapers to bypass restrictions and access targeted data without being detected. Bright Data is a top choice for [reliable proxy services](/proxy-types).
The following example highlights how easy it is to work with Bright Data proxies. It involves making some changes to the `script_scrape.py` file to scrape the Books to Scrape website.
To start, you need to obtain proxies from Bright Data by signing up for a free trial, which provides $5 USD worth of proxy resources. After creating a Bright Data account, you’ll see the following dashboard:
Navigate to the **My Zones** option and create a new [residential proxy](/proxy-types/residential-proxies) zone. Creating a new zone reveals your proxy username, password, and host, which you need in the next step.
Open the `static_scrape.py` file and add the following code below the URL variable:
```none
URL = "https://books.toscrape.com/"
# new
username = ""
password = ""
hostname = ""
proxies = {
"http": f"https://{username}:{password}@{hostname}",
"https": f"https://{username}:{password}@{hostname}",
}
content = requests.get(URL, proxies=proxies).text
```
Replace the `username`, `password`, and `hostname` placeholders with your proxy credentials. This code instructs the `requests` library to use the specified proxy. The rest of your script remains unchanged.
Test your script by running the following command:
```none
python static_scrape.py
```
After running this script, you’ll see a similar output to what you received in the previous example.
You can view this entire script on [GitHub](https://gist.github.com/vivekthedev/201f994bc14e4dbc7263b03983f917b3).
## Conclusion
lxml is a robust and easy-to-use tool for extracting data from HTML documents. lxml is optimized for speed and supports XPath and CSS selectors, allowing for efficient parsing of large XML and HTML documents.
In this tutorial, you learned all about web scraping with lxml and scraping both dynamic and static content. You also learned how using [Bright Data](/) proxy servers can help you bypass restrictions imposed by websites against scrapers.
Bright Data is a one-stop solution for all your web scraping projects. It offers features like proxies, scraping browsers, and reCAPTCHAs that enable users to effectively solve web scraping challenges. Bright Data also offers an [in-depth blog](/blog) with tutorials and best practices related to web scraping.
Interested in starting? Sign up now and test our products for free!
Contact usStart free trial
No credit card required
Vivek Kumar Singh
[ View all articles ](https://brightdata.com/blog/authors/vivek-kumar-singh)
Table of Contents
Dedicated Scraper APIs & No-Code Scrapers
Over 1000 scrapers for all popular domains. Simplify your web scraping.
[See pricing](/pricing/web-scraper "See pricing")
Just want data? Skip scraping.
Hundreds of ready-to-use datasets from all popular domains.
[See pricing](/pricing/datasets "See pricing")
[ ](https://news.ycombinator.com/submitlink?t=How+to+Use+lxml+for+Web+Scraping&u=https://brightdata.com/blog/web-data/lxml-web-scraping) [ ](https://www.linkedin.com/shareArticle?mini=true&title=How+to+Use+lxml+for+Web+Scraping&url=https://brightdata.com/blog/web-data/lxml-web-scraping) [ ](http://www.reddit.com/submit?title=How+to+Use+lxml+for+Web+Scraping&url=https://brightdata.com/blog/web-data/lxml-web-scraping)
## You might also be interested in
[ ](https://brightdata.com/blog/ai/openhuman-with-bright-data "Production-Ready Web Access in OpenHuman Through the Bright Data CLI")
[AI
Antonello Zanini
Technical Writer
### Production-Ready Web Access in OpenHuman Through the Bright Data CLI
Integrate Bright Data CLI with OpenHuman to enable production-ready web access and data collection for AI agents.
09-Sep-2026
12 min read
](https://brightdata.com/blog/ai/openhuman-with-bright-data)
[ ](https://brightdata.com/blog/ai/minimax-m3-with-bright-data "Giving self-hosted MiniMax M3 agents live web access with Bright Data")
[AI
Satyam Tripathi
Technical Writer
### Giving self-hosted MiniMax M3 agents live web access with Bright Data
Self-hosted MiniMax M3 agents get live web access using Bright Data’s search and scraping tools. Bypass blocks and CAPTCHAs.
09-Sep-2026
54 min read
](https://brightdata.com/blog/ai/minimax-m3-with-bright-data)
[ ](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax "Multimodal Web Scraping with MiniMax")
[Web Data
Antonello Zanini
Technical Writer
### Multimodal Web Scraping with MiniMax
Pair Bright Data Web Unlocker with MiniMax M3 vision to extract structured data from images and web page screenshots.
09-Sep-2026
4 min read
](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax)
` tag with the class `price_color`. Since `cssselect` returns a list, indexing (`[0]`) accesses the first element, and `text_content()` retrieves the text inside the element. Each extracted title and price pair is then appended to the `books` list as a dictionary, which can be easily stored in a JSON file.
Now that you’ve completed the web scraping process, it’s time to save this data locally. Open your script file and input the following code:
```none
with open("books.json", "w", encoding="utf-8") as file:
json.dump(books ,file)
```
In this code, a new file named `books.json` is created. This file is populated using the `json.dump` method, which takes the `books` list as the source and a `file` object as the destination.
You can test this script by opening the terminal and running the following command:
```none
python static_scrape.py
```
This command generates a new file in your directory with the following output:
All the code for this script is available on [GitHub](https://gist.github.com/vivekthedev/c1c5f0fb0e23cabfa3fa5c364b939f7c).
### Parsing Dynamic HTML Content
Scraping dynamic web content is trickier than scraping static content because JavaScript renders the data continuously rather than all at once. To help scrape dynamic content, you use a browser automation tool called [Selenium](https://www.selenium.dev/), which lets you create and run a browser instance and control it programmatically.
To install Selenium, open the terminal and run the following command:
```none
pip install selenium
```
YouTube is a great example of content rendered using JavaScript. When you visit any channel, only a limited number of videos load initially, with more videos appearing as you scroll down. Here, you scrape data for the top hundred videos from the [freeCodeCamp.org YouTube channel](https://www.youtube.com/c/Freecodecamp) by emulating keyboard presses to scroll the page.
To begin, inspect the HTML code of the web page. When you open **Dev Tools**, you’ll see the following:
The following code identifies the elements responsible for displaying the video title and link:
```none