Headless Browser Scraping in Python and Go

Headless browser scraping means loading pages in a real browser engine that runs without a window, controlled by code, and reading the page after its scripts have run. The browser downloads the HTML, executes the JavaScript, fetches the data the page asks for and builds the DOM, exactly as it would for a person; your script then reads the rendered elements, clicks and scrolls. "Headless" only means no visible window: the engine is the same Chromium, Firefox or WebKit. What a headless browser is in general, beyond scraping, is covered in what a headless browser is.

When curl is not enough

curl, Requests and other HTTP clients fetch exactly one URL and stop. They do not run JavaScript, load the requests a page makes after it opens, click buttons, scroll, or keep the browser-side state some sites check. If the page you get back is a shell with a <div id="root">, a "please enable JavaScript" notice, or a challenge page, curl is not enough. Before you switch, check the Network tab: many "JavaScript sites" load their data from one JSON request that curl can call.

Playwright: a headless browser in Python

# pip install playwright && playwright install chromium
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)       # headless is the default
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/", wait_until="domcontentloaded")
    page.locator(".quote").first.wait_for()
    print(page.locator(".quote .text").all_inner_texts()[:3])
    browser.close()

Pass headless=False while you develop to watch the browser work. The full scraping guide, with waits, infinite scroll and screenshots, is web scraping with Playwright.

Selenium: headless Chrome in Python

# pip install selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_argument("--window-size=1366,900")
driver = webdriver.Chrome(options=options)          # Selenium Manager finds chromedriver
try:
    driver.get("https://quotes.toscrape.com/js/")
    WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".quote")))
    print([e.text for e in driver.find_elements(By.CSS_SELECTOR, ".quote .text")][:3])
finally:
    driver.quit()

Set a window size in headless mode: the default viewport can be small enough to switch a responsive site to its mobile layout. More on Selenium in what Selenium is and web scraping with Selenium.

Go has no official Puppeteer, but two libraries play the same role by speaking the Chrome DevTools Protocol directly:

• chromedp (github.com/chromedp/chromedp, MIT): drives Chrome with a sequence of actions run in a context.
• rod (github.com/go-rod/rod, MIT): a higher-level API with auto-waiting, and it can download a browser for you.

// go get github.com/chromedp/chromedp
package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"github.com/chromedp/chromedp"
)

func main() {
	ctx, cancel := chromedp.NewContext(context.Background()) // headless Chrome
	defer cancel()
	ctx, cancel = context.WithTimeout(ctx, 30*time.Second)
	defer cancel()

	var quotes []string
	err := chromedp.Run(ctx,
		chromedp.Navigate("https://quotes.toscrape.com/js/"),
		chromedp.WaitVisible(".quote", chromedp.ByQuery),
		chromedp.Evaluate(`[...document.querySelectorAll('.quote .text')].map(e => e.textContent)`, &quotes),
	)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(len(quotes), "quotes")
}

The same with rod:

// go get github.com/go-rod/rod
browser := rod.New().MustConnect()
defer browser.MustClose()
page := browser.MustPage("https://quotes.toscrape.com/js/").MustWaitStable()
for _, el := range page.MustElements(".quote .text") {
	fmt.Println(el.MustText())
}

For static pages in Go, Colly and goquery are much faster; see web scraping with Go.

Why: it sees the page as a user does. It runs the JavaScript that builds the content, follows the redirects and requests a page makes, handles clicks, scrolls and forms, keeps cookies and storage like a normal session, and produces screenshots and PDFs.

The downsides:

• Speed. A page takes seconds instead of milliseconds, because the browser downloads and runs everything.
• Resources. Each browser process takes hundreds of MB of memory and real CPU; a server runs far fewer browsers than HTTP workers.
• Traffic. Scripts, styles, images and fonts are downloaded too, which matters when proxy traffic is billed per GB.
• Fragility. Waits, timeouts and pop-ups add failure modes plain requests do not have.
• Detection. Automated browsers have tell-tale signs, and sites with anti-bot systems look for them; headless mode alone does not make a scraper look human. See how anti-bot systems detect scrapers.