Web Scraping With Playwright

Yes, for JavaScript-rendered pages and multi-step flows. Playwright is Microsoft's open-source browser automation library (Apache-2.0), built for testing but just as useful for scraping (playwright.dev). Its strengths for scraping:

• Auto-waiting. Actions and locators wait until an element is attached, visible and stable, so you write fewer manual waits.
• Browser contexts. One browser can run many isolated sessions, each with its own cookies and its own proxy, at a fraction of the cost of new browsers.
• Network control. You can block images and fonts, or read the JSON responses the page loads, from the same script.
• Four languages, three engines. The same API in Python, Node.js, Java and .NET, on Chromium, Firefox and WebKit.

Use a plain HTTP client instead when the data is in the page source: it is many times faster and uses a fraction of the traffic. What Playwright is, its license and how it differs from Selenium are covered in what Playwright is.

Playwright installs as a library plus the browser builds it is tested against.

# Python
pip install playwright
playwright install chromium

# Node.js
npm i playwright
npx playwright install chromium

playwright install without a name downloads Chromium, Firefox and WebKit; on a fresh Linux server, add --with-deps to install the system libraries the browsers need (playwright.dev: Browsers).

This script opens a practice page whose quotes are written in by JavaScript, waits for them and saves each quote, author and tag list to JSON.

# scrape.py  ->  python scrape.py
import json
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()                     # headless by default
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/", wait_until="domcontentloaded")
    page.locator(".quote").first.wait_for()           # wait for the data, not a timer

    quotes = [
        {
            "text": card.locator(".text").inner_text(),
            "author": card.locator(".author").inner_text(),
            "tags": card.locator(".tag").all_inner_texts(),
        }
        for card in page.locator(".quote").all()
    ]
    browser.close()

json.dump(quotes, open("quotes.json", "w"), indent=2)
print(len(quotes), "quotes saved")

The same scraper in Node.js:

// scrape.mjs  ->  node scrape.mjs
import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://quotes.toscrape.com/js/', { waitUntil: 'domcontentloaded' });
await page.locator('.quote').first().waitFor();
const quotes = await page.locator('.quote').evaluateAll(cards => cards.map(c => ({
  text: c.querySelector('.text').textContent,
  author: c.querySelector('.author').textContent,
})));
console.log(quotes.length);
await browser.close();

To write a script faster, let Playwright record one: playwright codegen https://quotes.toscrape.com/js/ (or npx playwright codegen …) opens a browser and turns your clicks into code with suggested locators (playwright.dev: Codegen).

A locator is a description of how to find an element, evaluated fresh every time you use it, which is why it survives re-renders. Create one with page.locator() or the get_by_* helpers (playwright.dev: Locators):

For scraping, CSS and XPath on stable attributes (IDs, data-* attributes, itemprop) break least often. Role and text locators are the testing default and work well for buttons and links you click. A locator that matches several elements returns a list through .all(), .all_inner_texts() or .evaluate_all(); calling an action such as .click() on it raises a strict-mode error, which protects you from clicking the wrong element.

Most waits in Playwright are automatic: click(), fill() and inner_text() wait for the element first. You need an explicit wait when you read data that arrives later.

• page.goto(url, wait_until=...) takes "commit", "domcontentloaded", "load" (the default) or "networkidle". The Playwright docs discourage networkidle for tests, because many pages never go quiet; for scraping, domcontentloaded plus a locator wait is faster and more reliable.
• locator.wait_for(state="visible") waits until the element is attached and visible; the other states are "attached", "detached" and "hidden". This is the waitFor most scrapers need.
• page.wait_for_load_state("load") waits for a load state after an action that navigates.
• page.wait_for_function("() => window.dataReady === true") waits for any condition in the page.
• page.expect_response(...) waits for a specific network response and gives you its body.

Timeouts are 30 seconds by default for actions and navigation in library scripts. Change them for the whole page with page.set_default_timeout(60_000) and page.set_default_navigation_timeout(60_000), or per call with timeout=. In the Playwright Test runner, each test has a 30-second timeout, expect assertions 5 seconds, and globalTimeout in playwright.config caps the whole run (playwright.dev: Timeouts).

Read the "Next" link's address and goto() it until there is none; a URL can be retried and logged, a click cannot.

from urllib.parse import urljoin

url, rows = "https://quotes.toscrape.com/js/", []
while url:
    page.goto(url, wait_until="domcontentloaded")
    page.locator(".quote").first.wait_for()
    rows += page.locator(".quote .text").all_inner_texts()
    nxt = page.locator("li.next > a")
    url = urljoin(page.url, nxt.get_attribute("href")) if nxt.count() else None

Scroll, wait until more items exist than before, and stop when nothing new arrives or after a fixed number of rounds.

from playwright.sync_api import TimeoutError as PlaywrightTimeout

page.goto("https://quotes.toscrape.com/scroll", wait_until="domcontentloaded")
page.locator(".quote").first.wait_for()
count = page.locator(".quote").count()
for _ in range(30):
    page.mouse.wheel(0, 5000)
    try:
        page.wait_for_function(
            "n => document.querySelectorAll('.quote').length > n", arg=count, timeout=5000)
    except PlaywrightTimeout:
        break                                   # no new items: end of the feed
    count = page.locator(".quote").count()
print(count, "quotes loaded")

Each batch of an infinite feed usually comes from a JSON request. Catch it instead of parsing the page: with page.expect_response(lambda r: "/api/" in r.url) as resp: page.mouse.wheel(0, 5000) and then resp.value.json().

page.screenshot() captures the viewport or, with full_page=True, the whole scrollable page; locator.screenshot() captures one element (playwright.dev: Screenshots).

page.set_viewport_size({"width": 1366, "height": 900})
page.screenshot(path="page.png", full_page=True)
page.locator(".quote").first.screenshot(path="first-quote.png")
page.screenshot(path="clean.png", mask=[page.locator(".cookie-banner")])   # hide an element

A browser page is heavy, so scale by reusing what you can:

page.route("**/*", lambda route: route.abort()
           if route.request.resource_type in {"image", "font", "media"} else route.continue_())

4. Use the browser only where needed. Fetch static pages over HTTP and send only the JavaScript-rendered ones to Playwright.

import asyncio
from playwright.async_api import async_playwright

async def title(browser, sem, url):
    async with sem:
        ctx = await browser.new_context()
        page = await ctx.new_page()
        await page.goto(url, wait_until="domcontentloaded")
        t = await page.title()
        await ctx.close()
        return t

async def main(urls):
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        sem = asyncio.Semaphore(4)                # 4 pages at a time
        print(await asyncio.gather(*(title(browser, sem, u) for u in urls)))
        await browser.close()

asyncio.run(main(["https://quotes.toscrape.com/js/page/1/", "https://quotes.toscrape.com/js/page/2/"]))

• Prefer stable selectors (IDs, data-* attributes, structured data in <script type="application/ld+json">) over long CSS paths.
• Wait for the data element, not for networkidle and not for fixed sleeps.
• Set explicit timeouts and catch TimeoutError; log the URL and save a screenshot on failure.
• Close contexts and pages; a leaked context keeps memory and connections.
• Keep sessions consistent: one context, one IP, one set of cookies for one logical visitor.
• Pace requests per site, and collect only public data the site's terms allow; see is web scraping legal.
• When a site blocks you, read how anti-bot systems detect scrapers before adding more IPs.