Scraping JavaScript Pages With Scrapy

Scrapy's downloader fetches the HTML the server sends and hands it to your callback. Anything a script adds after load (a table filled by an AJAX call, prices from a React app, reviews behind "Load more") is not in that response. The usual symptom is a selector that works in your browser's DevTools and returns nothing in Scrapy.

Check it in the Scrapy shell:

scrapy shell "https://www.scrapethissite.com/pages/ajax-javascript/"
>>> response.css("tr.film").getall()
[]
>>> view(response)   # opens what Scrapy received in your browser

On this practice page, the Oscar films table is empty in the HTML; a click on a year runs a jQuery request to ?ajax=true&year=2015 and fills the table from the JSON it returns. The general method for spotting this is in scrape dynamic web pages.

If the data arrives as JSON, skip the browser. Ask Scrapy for the same URL the page calls:

import scrapy


class FilmsApiSpider(scrapy.Spider):
    name = "films_api"

    async def start(self):
        for year in range(2010, 2016):
            yield scrapy.Request(
                f"https://www.scrapethissite.com/pages/ajax-javascript/?ajax=true&year={year}"
            )

    def parse(self, response):
        for film in response.json():
            yield {"title": film["title"].strip(), "year": film["year"], "awards": film["awards"]}

On 2026-10-23 this spider returned 87 films for the six years in six small requests. This is the fastest and lightest way to scrape a JavaScript page: no browser, a few kilobytes per response, Scrapy's normal concurrency. It fails when the request needs tokens that only the page's script can compute, and that is when you render. Finding and replaying these requests is covered in scraping hidden APIs.

async def start() replaced start_requests() in Scrapy 2.13.0, and from Scrapy 2.16.0 on start_requests() is removed and no longer called (Scrapy release notes). The current Scrapy release is 2.19.0 (PyPI, 2026-10-23).

scrapy-playwright (BSD-3-Clause, Python 3.10+) replaces Scrapy's download handler. Requests you mark with meta={"playwright": True} are loaded in a Playwright browser; all other requests still go through Scrapy's normal downloader.

pip install scrapy scrapy-playwright
playwright install chromium

import scrapy
from scrapy_playwright.page import PageMethod


class FilmsSpider(scrapy.Spider):
    name = "films"
    custom_settings = {
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_DELAY": 2,
    }

    async def start(self):
        for year in (2015, 2014):
            yield scrapy.Request(
                f"https://www.scrapethissite.com/pages/ajax-javascript/#{year}",
                meta={
                    "playwright": True,
                    "playwright_page_methods": [PageMethod("wait_for_selector", "tr.film")],
                },
                cb_kwargs={"year": year},
                dont_filter=True,
            )

    def parse(self, response, year):
        for row in response.css("tr.film"):
            yield {
                "year": year,
                "title": row.css("td.film-title::text").get("").strip(),
                "awards": int(row.css("td.film-awards::text").get("0")),
            }

Run it with scrapy runspider films_spider.py -O films.json. On 2026-10-23 it returned 32 films for the two years. Three details in it:

• PageMethod("wait_for_selector", "tr.film") waits until the script has filled the table. Without a wait, the response is taken when the page's load event fires, and on this page the rows come later. You can chain other page methods the same way: click, fill, evaluate, wait_for_timeout.
• dont_filter=True is needed here because Scrapy's duplicate filter ignores the #2015 fragment, so the second URL would count as a repeat of the first.
• The asyncio reactor has been the default in new Scrapy projects since 2.7; set it anyway in a spider that runs on its own.

When you need to act on the page inside your callback (scroll, click "Load more", read a value from JavaScript), pass "playwright_include_page": True, take the page from response.meta["playwright_page"], and always await page.close(), also in an errback. Open pages count against PLAYWRIGHT_MAX_PAGES_PER_CONTEXT, and a crawl that leaks them can freeze.

Make scrapy-playwright faster and lighter

A browser downloads everything a page references. Block what you don't need, and cap how many browser pages run at once:

def should_abort_request(request):
    return request.resource_type in {"image", "media", "font"}

custom_settings = {
    # ... the settings above, plus:
    "PLAYWRIGHT_ABORT_REQUEST": should_abort_request,
    "PLAYWRIGHT_MAX_CONTEXTS": 4,
    "PLAYWRIGHT_MAX_PAGES_PER_CONTEXT": 4,
    "CONCURRENT_REQUESTS": 8,
}

Render only the pages that need it. A common pattern is a plain Scrapy request for the listing and meta={"playwright": True} only for the detail pages that load their content by script, or the other way around.

Splash is a headless browser that runs as an HTTP service, usually in Docker on port 8050, and scrapy-splash (0.11.1, February 2025) sends Scrapy requests to it. You start the Splash container as its installation docs describe, then add the settings from the scrapy-splash README (SPLASH_URL, its downloader and spider middlewares, the dupe filter and request fingerprinter) and use SplashRequest:

from scrapy_splash import SplashRequest

yield SplashRequest(url, self.parse, args={"wait": 2})

Splash scripts are written in Lua, which lets one request scroll, click and return several values. The trade-offs: the last Splash release is 3.5, from June 16, 2020 (changelog), and its rendering engine is older than current browsers, so some modern sites render incompletely. A render times out after 30 seconds by default and can't exceed 90 unless you start Splash with --max-timeout (Splash HTTP API). Splash still makes sense in a project that already uses it, or when you want rendering on a separate server that several spiders share. For new projects, scrapy-playwright is simpler.

You can combine them, but there is no maintained plugin. scrapy-selenium 0.0.7 dates from January 24, 2019; its middleware passes an executable_path argument and a browser-specific options argument (such as chrome_options) to the WebDriver, while the current Selenium, 4.51.0, accepts only options, service and keep_alive. With current Selenium it fails when the spider starts, unless you pin an old Selenium or patch the middleware.

Two working routes if you need Selenium:

• A small downloader middleware of your own that loads marked requests in a WebDriver and returns HtmlResponse(url, body=driver.page_source, encoding="utf-8", request=request). Selenium's calls block, so Scrapy waits while each page renders; keep concurrency low or run several processes.
• Two stages: Scrapy collects the URLs, and a separate Selenium script renders only the pages that need it.

If Selenium is not a requirement, scrapy-playwright does the same job inside Scrapy's event loop. The broader comparison is in Playwright vs Selenium.