--- title: "Web Scraping With Scrapy Splash: Step-By-Step Guide" slug: web-scraping-with-scrapy-splash date: 2025-03-19T13:30:33+00:00 modified: 2026-06-30T14:54:07+00:00 permalink: https://brightdata.com/blog/web-data/web-scraping-with-scrapy-splash type: blog --- [ Blog ](https://brightdata.com/blog "Blog") / [Web Data](https://brightdata.com/blog/web-data) [Web Data](https://brightdata.com/blog/web-data) # Web Scraping With Scrapy Splash: Step-By-Step Guide Master Scrapy Splash for web scraping in Python. Learn to render JavaScript, extract data, and handle dynamic pages efficiently. 16 min read [ ](https://brightdata.com/blog/authors/federico-trotta) [Federico Trotta Technical Writer ](https://brightdata.com/blog/authors/federico-trotta) ![Web Scraping with Scrapy Splash blog image](https://media.brightdata.com/2025/03/Web-Scraping-with-Scrapy-Splash.svg) In this guide on Scrapy Splash, you will learn: - What Scrapy Splash is - How to use Scrapy Splash in Python in a step-by-step tutorial - Advanced scraping techniques with Splash in Scrapy - Limitations when scraping websites with this tool Let’s dive in! ## What is Scrapy Splash? [Scrapy Splash](https://github.com/scrapy-plugins/scrapy-splash) refers to the integration between these two tools: - [**Scrapy**](https://scrapy.org/): A Python open-source crawling framework library for extracting the data you need from websites. - [**Splash**](https://github.com/scrapinghub/splash): A lightweight headless browser designed for rendering JavaScript-heavy web pages. You might be wondering why a powerful tool like Scrapy needs Splash. Well, Scrapy can only handle static sites, as it relies on HTML parsing capabilities (specifically, from [Parsel](/blog/web-data/web-scraping-with-parsel)). However, when scraping dynamic websites, you need to deal with JavaScript rendering. A common solution is to use an automated browser, which is exactly what Splash provides. With Scrapy Splash, you can send a special request—known as [`SplashRequest`](https://github.com/scrapy-plugins/scrapy-splash#requests)—to a Splash server. This server fully renders the page by executing JavaScript and returns the processed HTML. So, it lets your Scrapy Spider retrieve data from dynamic pages. In short, you need Scrapy Splash if: - You are working with JavaScript-heavy websites that Scrapy alone cannot scrape. - You prefer a lightweight solution compared to [Selenium](https://www.selenium.dev/) or [Playwright](https://playwright.dev/). - You want to avoid the overhead of running a full browser for scraping. If Scrapy Splash does not meet your needs, consider these alternatives: 1. [**Selenium**](/blog/how-tos/using-selenium-for-web-scraping): A full-fledged browser automation ability for scraping JavaScript-heavy websites, which provides interesting extensions like [Selenium Wire](/blog/web-data/web-scraping-with-selenium-wire). 2. [**Playwright**](/blog/how-tos/playwright-web-scraping): An open-source browser automation tool providing consistent cross-browser automation and robust API, which supports multiple programming languages. 3. [**Puppeteer**](/blog/how-tos/web-scraping-puppeteer): An open-source Node.js library developed that provides a high-level API for automating and controlling Chrome via the [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/). ## Scrapy Splash in Python: A Step-by-Step Tutorial In this section, you will understand how to use Scrapy Splash to retrieve data from a website. The target page will be a special JavaScript-rendered version of the popular “[Quotes to Scrape](https://quotes.toscrape.com/scroll)” site: This is just like the usual “Quotes to Scrape,” but it uses infinite scrolling to load data dynamically via AJAX requests triggered by JavaScript. ### Requirements To replicate this tutorial using Scrapy Splash in Python, your system must match the following requirements: - [Python 3.10.1](https://www.python.org/downloads/) or higher. - [Docker 27.5.1](https://www.docker.com/) or higher. If you do not have those two tools installed on your machine, follow the above links. ### Prerequisites, Dependencies, and Splash Integration Suppose you call the main folder of your project `scrapy_splash/`. At the end of this step, the folder will have the following structure: ```none scrapy_splash/ └── venv/ ``` Where `venv/` contains the virtual environment. You can create the `venv/` [virtual environment](https://docs.python.org/3/library/venv.html) directory like so: ```none python -m venv venv ``` To activate it, on Windows, run: ```none venvScriptsactivate ``` Equivalently, on macOS and Linux, execute: ```none source venv/bin/activate ``` In the activated virtual environment, install the dependencies with: ```none pip install scrapy scrapy-splash ``` As a final prerequisite, you need to [pull the Splash image via Docker](https://hub.docker.com/r/scrapinghub/splash): ```none docker pull scrapinghub/splash ``` Then, start the container: ```none docker run -it -p 8050:8050 --rm scrapinghub/splash ``` For more information, [follow the OS-based Docker integration instructions](https://splash.readthedocs.io/en/latest/install.html). After starting the Docker container, wait until the Splash service logs the message below: ```none Server listening on http://0.0.0.0:8050 ``` The message signals that Splash is now available at `http://0.0.0.0:8050`. Visit that URL in your browser, and you should now be seeing the following page: Depending on your configuration, following the `http://0.0.0.0:8050` URL may not show the Splash service working. In this case, try to use one of the following instead: - `http://localhost:8050` - `http://127.0.0.1:8050` **Note**: Remember that the connection to the Splash server must remain open while using Scrapy-Splash. In other words, if you used the CLI to run the Docker container, keep that terminal open and use a separate terminal for the next steps in this procedure. Wonderful! You now have what you need to scrape web pages with Scrapy Splash. ### Step #1: Start a New Scrapy Project Inside the `scrapy_splash/` main folder, type the command below to launch a new Scrapy project: ```none scrapy startproject quotes ``` With that command, Scrapy will create a `quotes/` folder. Inside it, it will automatically generate all the files you need. This is the resulting folder structure: ```none scrapy_splash/ ├── quotes/ │ ├── quotes/ │ │ ├── spiders/ │ │ ├── __init__.py │ │ ├── items.py │ │ ├── middlewares.py │ │ ├── pipelines.py │ │ └── settings.py │ │ │ └── scrapy.cfg └── venv/ ``` Perfect! You started a new Scrapy project. ### Step #2: Generate The Spider To generate a new spider to crawl the target website, navigate to the `quotes/` folder: ```none cd quotes ``` Then, generate a new spider with: ```none scrapy genspider words https://quotes.toscrape.com/scroll ``` You will obtain the following result: ```none Created spider 'words' using template 'basic' in module: quotes.spiders.words ``` As you can tell, Scrapy has automatically created a `words.py` file inside the `spiders/` folder. The `words.py` file contains the following code: ```none import scrapy class WordsSpider(scrapy.Spider): name = "words" allowed_domains = ["quotes.toscrape.com"] start_urls = ["https://quotes.toscrape.com/scroll"] def parse(self, response): pass ``` This will soon contain the required scraping logic from the dynamic target page. Hooray! You generated the spider to scrape the target website. ### Step #3: Configure Scrapy to Use Splash You now have to configure Scrapy so that it can use the Splash service. To do so, add the following configurations to the file `settings.py`: ```none # Set the Splash local server endpoint SPLASH_URL = "http://localhost:8050" # Enable the Splash downloader middleware DOWNLOADER_MIDDLEWARES = { "scrapy_splash.SplashCookiesMiddleware": 723, "scrapy_splash.SplashMiddleware": 725, "scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware": 810, } # Enable the Splash deduplication argument filter SPIDER_MIDDLEWARES = { "scrapy_splash.SplashDeduplicateArgsMiddleware": 100, } ``` In the above configs: - `SPLASH_URL` sets the endpoint for the local Splash server. This where Scrapy will send requests for JavaScript rendering. - `DOWNLOADER_MIDDLEWARES` enables specific middlewares to interact with Splash. In particular: - `SplashCookiesMiddleware` configures Scrapy to handle cookies when making requests through Splash. - `SplashMiddleware` tells Scrapy to do the actual rendering of dynamic content through Splash. - `HttpCompressionMiddleware` changes how Scrapy deals with HTTP compression for downloaded content. - `SPIDER_MIDDLEWARES` guarantees that requests with the same Splash arguments are not duplicated, which is useful for reducing unnecessary load and improving efficiency. For more detailed information about these configurations, [refer to the official Scrapy-Splash documentation](https://github.com/scrapy-plugins/scrapy-splash?tab=readme-ov-file#configuration). Good! Now Scrapy can connect to Splash and programmatically use it for JavaScript rendering. ### Step #4: Define the Lua Script for JavaScript Rendering Scrapy can now integrate with Splash to render web pages that rely on JavaScript, like the target page of this guide. To define custom rendering and interaction logic, you need to use [Lua scripts](https://www.lua.org/). That is because [Splash relies on Lua scripts](https://splash.readthedocs.io/en/stable/scripting-tutorial.html) to interact with web pages via JavaScript and control the browser’s behavior programmatically. Specifically, add the Lua script below to `words.py`: ```none script = """ function main(splash, args) splash:go(args.url) -- custom rendering script logic... return splash:html() end """ ``` In the above snippet, the `script` variable contains the Lua logic that Splash will execute on the server. In particular, this script instructs Splash to: 1. Navigate to the defined URL with the method [`splash:go()`](https://splash.readthedocs.io/en/stable/scripting-ref.html#splash-go). 2. Return the rendered HTML content with the method [`splash:html()`](https://splash.readthedocs.io/en/stable/scripting-ref.html#splash-html). Use the Lua script above in a `start_requests()` function inside the `WordsSpider` class: ```none def start_requests(self): for url in self.start_urls: yield SplashRequest( url, self.parse, endpoint="execute", args={"lua_source": script} ) ``` The above `start_requests()` method overrides Scrapy’s default [`start_requests()`](https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.start_requests). This way, Scrapy Splash can execute the Lua script to retrieve the JavaScript-rendered HTML of the page. Lua script execution occurs via the `"lua_source": script` argument in the `SplashRequest()` method. Also, note the use of the `"execute"` Splash endpoint (which you will learn more soon). Do not forget to import `SplashRequest` from Scrapy Splash: ```none from scrapy_splash import SplashRequest ``` Your `words.py` file is now equipped with the correct Lua script to access the JavaScript-rendered content on the page! ### Step #5: Define the Data Parsing Logic Before getting started, inspect a quote HTML element on the target page to understand how to parse it: There, you can see that the quote elements can be selected with `.quote`. Given a quote, you can then get: 1. The quote text from `.text`. 2. The quote author from `.author`. 3. The quote tags from `.tags`. The scraping logic to retrieve all the quotes from the target page can be defined through the following `parse()` method: ```none def parse(self, response): # Retrieve CSS selectors quotes = response.css(".quote") for quote in quotes: yield { "text": quote.css(".text::text").get(), "author": quote.css(".author::text").get(), "tags": quote.css(".tags a.tag::text").getall() } ``` `parse()` processes the response returned by Splash. In detail, it: 1. Extracts all `div` elements with the class `quote` using the CSS selector `".quote"`. 2. Iterates over each `quote` element to extract the name, author, and tag for each quote. Very well! The Scrapy Splash scraping logic is complete. ### Step #6: Put It All Together and Run The Script This is what your final `words.py` file should look like: ```none import scrapy from scrapy_splash import SplashRequest # Lua script for JavaScript rendering script = """ function main(splash, args) splash:go(args.url) return splash:html() end """ class WordsSpider(scrapy.Spider): name = "words" start_urls = ["https://quotes.toscrape.com/scroll"] def start_requests(self): for url in self.start_urls: yield SplashRequest( url, self.parse, endpoint="execute", args={"lua_source": script} ) def parse(self, response): quotes = response.css(".quote") for quote in quotes: yield { "text": quote.css(".text::text").get(), "author": quote.css(".author::text").get(), "tags": quote.css(".tags a.tag::text").getall() } ``` Run the script with this command: ```none scrapy crawl words ``` This is the expected result: The desired result can be better visualized like this: ```none 2026-03-18 12:21:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/scroll> {'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']} 2026-03-18 12:21:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/scroll> {'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']} # omitted for brevity... 2026-03-18 12:21:55 [scrapy.core.engine] INFO: Closing spider (finished) ``` Note that the output contains the data of interest. Note that if you remove the `start_requests()` method from the `Words``Spider` class, Scrapy will not return any data. That is because, withour Splash, it cannot render pages that require JavaScript. Very good! You completed your first Scrapy Splash project. ### A Note on Splash Splash is a server that communicates over HTTP. This allows you to scrape web pages with Splash using any HTTP client, by calling its endpoints. The endpoints it provides are: - [`execute`](https://splash.readthedocs.io/en/stable/api.html#execute): Executes a custom Lua rendering script and return its result. - [`render.html`](https://splash.readthedocs.io/en/stable/api.html): Returns the HTML of the javascript-rendered page. - [`render.png`](https://splash.readthedocs.io/en/stable/api.html#render-png): Returns an image (in PNG format) of the javascript-rendered page. - [`render.jpeg`](https://splash.readthedocs.io/en/stable/api.html#render-jpeg): Returns an image (in JPEG format) of the javascript-rendered page. - [`render.har`](https://splash.readthedocs.io/en/stable/api.html#render-har): Returns information about Splash interaction with a website in HAR format. - [`render.json`](https://splash.readthedocs.io/en/stable/api.html#render-json): Returns a JSON-encoded dictionary with information about the javascript-rendered webpage. It can include HTML, PNG, and other information, based on arguments passed. To understand more about how these endpoints work, consider the `render.html` endpoint. Connect to the endpoint with this Python code: ```none # pip install requests import requests import json # URL of the Splash endpoint url = "http://localhost:8050/render.html" # Sending a POST request to the Splash endpoint payload = json.dumps({ "url": "https://quotes.toscrape.com/scroll" # URL of the page to render }) headers = { "content-type": "application/json" } response = requests.request("POST", url, headers=headers, data=payload) print(response.text) ``` This snippet defines: - The Splash instance on localhost as the URL making a call to the `render.html` endpoint. - The target page to scrape inside the `payload`. Run the above code and tou will obtain the rendered HTML of the whole page: ```none Quotes to Scrape ``` While Splash can handle JavaScript-rendered HTML independently, using Scrapy Splash with `SplashRequest` makes web scraping much easier. ## Scrapy Splash: Advanced Scraping Techniques In the previous paragraph, you completed a basic Scrapy tutorial with Splash integration. Time to try some advanced scraping techniques with Scrapy Splash! ### Managing Advanced Scrolling The target page contains quotes that are loaded dynamically via AJAX thanks to infinite scrolling: To manage the interaction with the infinite scrolling, you need to change the Lua script as follows: ```none script = """ function main(splash, args) local scroll_delay = 1.0 -- Time to wait between scrolls local max_scrolls = 10 -- Maximum number of scrolls to perform local scroll_to = 1000 -- Pixels to scroll down each time splash:go(args.url) splash:wait(scroll_delay) local scroll_count = 0 while scroll_count < max_scrolls do scroll_count = scroll_count + 1 splash:runjs("window.scrollBy(0, " .. scroll_to .. ");") splash:wait(scroll_delay) end return splash:html() end """ ``` This modified script relies on these variables: - `max_scrolls` defines the maximum number of scrolls to perform. This value may need to be changed based on how much content you want to scrape from the page. - `scroll_to` specifies the number of pixels to scroll down each time. Its value might need to be adjusted depending on the page’s behavior. - [`splash:runjs()`](https://splash.readthedocs.io/en/latest/scripting-ref.html#splash-runjs) executes the [`window.scrollBy()`](https://developer.mozilla.org/en-US/docs/Web/API/Window/scrollBy) JavaScript function to scroll the page down by the specified number of pixels. - [`splash:wait()`](https://splash.readthedocs.io/en/latest/scripting-ref.html#splash-wait) ensures the script waits before loading new content. The amount of time (in seconds) to wait is defined by the variable `scroll_delay`. In simpler terms, the above Lua script simulates a defined number of scrolls in an infinite scrolling web page scenario. The code in the `words.py` file will look like this: ```none import scrapy from scrapy_splash import SplashRequest # Lua script for infinite scrolling script = """ function main(splash, args) local scroll_delay = 1.0 -- Time to wait between scrolls local max_scrolls = 10 -- Maximum number of scrolls to perform local scroll_to = 1000 -- Pixels to scroll down each time splash:go(args.url) splash:wait(scroll_delay) local scroll_count = 0 while scroll_count < max_scrolls do scroll_count = scroll_count + 1 splash:runjs("window.scrollBy(0, " .. scroll_to .. ");") splash:wait(scroll_delay) end return splash:html() end """ class WordsSpider(scrapy.Spider): name = "words" start_urls = ["https://quotes.toscrape.com/scroll"] def start_requests(self): for url in self.start_urls: yield SplashRequest( url, self.parse, endpoint="execute", args={"lua_source": script} ) def parse(self, response): # Retrieve CSS selectors quotes = response.css("div.quote") for quote in quotes: yield { "text": quote.css("span.text::text").get(), "author": quote.css("span small.author::text").get(), "tags": quote.css("div.tags a.tag::text").getall() } ``` Run the script using the command below: ```none scrapy crawl words ``` The crawler will print all the scraped quotes consistently with the `max_scrolls` variable. This is the expected result: Note that the output now includes significantly more quotes than before. This confirms that the pages have been successfully scrolled down, and new data has been loaded and scraped. Perfect! You have now learned how to manage infinite scrolling with Scrapy Splash. ### Wait For Element Web pages can retrieve data dynamically or render nodes in the browser. This means the final DOM may take time to render. To avoid errors when retrieving data from a website, you should always wait for an element to be loaded on the page before interacting with it. In this example, the element to wait will be the text from the first quote: To implement waiting logic, write a Lua script as follows: ```none script = """ function main(splash, args) splash:go(args.url) while not splash:select(".text") do splash:wait(0.2) print("waiting...") end return { html=splash:html() } end """ ``` This script creates a `while` loop that waits for 0.2 seconds if the `text` element is on the page. To verify if the `.text` element in on the page, you can use the the [`splash:select()`](https://splash.readthedocs.io/en/stable/scripting-ref.html#splash-select) method. ### Wait For Time Since dynamic-content web pages take time to load and render, you can wait for a few seconds before accessing the HTML content. This can be achieved through the method `splash:wait()` like so: ```none script = """ function main(splash, args) splash:wait(args.wait) splash:go(args.url) return { html=splash:html() } end """ ``` In this case, the seconds that the script must wait are expressed in the `SplashRequest()` method with a [Lua script](https://splash.readthedocs.io/en/stable/scripting-ref.html#splash-args) [argument](https://splash.readthedocs.io/en/stable/scripting-ref.html#splash-args). For example, set`"wait" : 2.0` to tell the Lua script to wait 2 seconds: ```none import scrapy from scrapy_splash import SplashRequest script = """ function main(splash, args) splash:wait(args.wait) splash:go(args.url) return { html=splash:html() } end """ class WordsSpider(scrapy.Spider): name = "words" start_urls = ["https://quotes.toscrape.com/scroll"] def start_requests(self): for url in self.start_urls: yield SplashRequest( url, self.parse, endpoint="execute", args={"lua_source": script, "wait": 2.0} # Waiting for 2 seconds ) # ... ``` **Note**: A hard wait (`splash:wait()`) is useful for local testing since it ensures the page loads before proceeding. That approach is not ideal for production as it adds unnecessary delays, hurting performance and scalability. Additionally, you cannot know in advance the right amount of time to wait. Well done! You learned how to wait a certain amount of time in Scrapy Splash. ## Limitations of Using Scrapy Splash In this tutorial, you learned how to use Scrapy Splash to extract data from the Web in different scenarios. While this integration is straightforward, it has some downsides. For instance, setting up Splash requires running a separate Splash server with Docker, adding complexity to your scraping infrastructure. Additionally, Splash’s Lua scripting API is somewhat limited compared to more modern tools like Puppeteer and Playwright. However, as with [all headless browsers](/blog/web-data/best-headless-browsers), the biggest limitation comes from the browser itself. Anti-scraping technologies can detect when a browser is being automated rather than used normally, leading to script blocks. Forget these challenges with [**Scraping Browser**](/products/scraping-browser)—a dedicated cloud-based scraping browser designed for infinite scalability. It includes CAPTCHA solving, browser fingerprint management, and anti-bot bypassing, so you don’t have to worry about getting blocked. ## Conclusion In this article, you learned what Scrapy Splash is and how it works. You started with the basics and then explored more complex scraping scenarios. You also discovered the limitations of the tool, particularly its vulnerability to anti-bot and anti-scraping systems. To overcome these challenges, Scraping Browser is an excellent solution. This is just one of the many Bright Data scraping solutions you can try: - [**Proxy Services**](/proxy-types): Four different types of proxies to bypass location restrictions, including [150 million+ residential IPs](/proxy-types/residential-proxies) - [**Web Scraper APIs**](/products/web-scraper): Dedicated endpoints for extracting fresh, structured web data from over 100 popular domains. - [**SERP API**](/products/serp-api): API to handle all ongoing unlocking management for SERP and extract one page Sign up for Bright Data now and start your free trial to test our scraping solutions. Contact usStart free trial No credit card required [ ](https://it.linkedin.com/in/federico-trotta) Federico Trotta Technical Writer 3 years experience Federico Trotta is a technical writer, editor, and data scientist. Expert in technical content management, data analysis, machine learning, and Python development. Expertise Data Analysis AI Web Scraping [ View all articles ](https://brightdata.com/blog/authors/federico-trotta) Table of Contents Dedicated Scraper APIs & No-Code Scrapers Over 1000 scrapers for all popular domains. Simplify your web scraping. [See pricing](/pricing/web-scraper "See pricing") Just want data? Skip scraping. Hundreds of ready-to-use datasets from all popular domains. [See pricing](/pricing/datasets "See pricing") [ ](https://news.ycombinator.com/submitlink?t=Web+Scraping+With+Scrapy+Splash%3A+Step-By-Step+Guide&u=https://brightdata.com/blog/web-data/web-scraping-with-scrapy-splash) [ ](https://www.linkedin.com/shareArticle?mini=true&title=Web+Scraping+With+Scrapy+Splash%3A+Step-By-Step+Guide&url=https://brightdata.com/blog/web-data/web-scraping-with-scrapy-splash) [ ](http://www.reddit.com/submit?title=Web+Scraping+With+Scrapy+Splash%3A+Step-By-Step+Guide&url=https://brightdata.com/blog/web-data/web-scraping-with-scrapy-splash) ## You might also be interested in [ ](https://brightdata.com/blog/ai/openhuman-with-bright-data "Production-Ready Web Access in OpenHuman Through the Bright Data CLI") [AI Antonello Zanini Technical Writer ### Production-Ready Web Access in OpenHuman Through the Bright Data CLI Integrate Bright Data CLI with OpenHuman to enable production-ready web access and data collection for AI agents. 09-Sep-2026 12 min read ](https://brightdata.com/blog/ai/openhuman-with-bright-data) [ ](https://brightdata.com/blog/ai/minimax-m3-with-bright-data "Giving self-hosted MiniMax M3 agents live web access with Bright Data") [AI Satyam Tripathi Technical Writer ### Giving self-hosted MiniMax M3 agents live web access with Bright Data Self-hosted MiniMax M3 agents get live web access using Bright Data’s search and scraping tools. Bypass blocks and CAPTCHAs. 09-Sep-2026 54 min read ](https://brightdata.com/blog/ai/minimax-m3-with-bright-data) [ ](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax "Multimodal Web Scraping with MiniMax") [Web Data Antonello Zanini Technical Writer ### Multimodal Web Scraping with MiniMax Pair Bright Data Web Unlocker with MiniMax M3 vision to extract structured data from images and web page screenshots. 09-Sep-2026 4 min read ](https://brightdata.com/blog/web-data/multimodal-web-scraping-with-minimax)