Web Scraping With Puppeteer

Yes, for pages whose data appears only after scripts run: single-page apps, infinite feeds, content behind "Load more" buttons and sites that check for a real browser. Puppeteer is a Node.js library maintained by the Chrome team that controls Chrome (and, in current versions, Firefox) over the DevTools Protocol or WebDriver BiDi (pptr.dev). It is open source under the Apache-2.0 license (npm).

It is the wrong tool when the data is already in the HTML the server sends. A browser downloads every script, style sheet and image, so each page costs more time, memory and proxy traffic than a plain request. Check first: open the page source (Ctrl+U or Cmd+Option+U). If you can see the prices or names there, use fetch with Cheerio (see web scraping with JavaScript and Node.js); if the source is an empty shell, use Puppeteer.

What Puppeteer is and where it came from is covered in what Puppeteer is; this guide is about scraping with it.

Run npm i puppeteer in a Node.js project. The package downloads a matching build of Chrome for Testing the first time, so your scraper and the browser version always fit together. If you want to use a Chrome you already have, install puppeteer-core instead and pass executablePath (or channel: 'chrome') to puppeteer.launch().

mkdir quotes-scraper && cd quotes-scraper
npm init -y
npm pkg set type=module
npm i puppeteer

Use a current Node.js LTS release; the Puppeteer docs list the supported versions for each release (pptr.dev: System requirements).

The script below opens a JavaScript-rendered practice page, waits for the quotes to appear and saves each quote, author and tag list. The practice site quotes.toscrape.com/js/ exists for scraping exercises; its quotes are written into the page by a script, so a plain HTTP request sees none of them.

// scrape.mjs  ->  node scrape.mjs
import puppeteer from 'puppeteer';
import { writeFile } from 'node:fs/promises';

const browser = await puppeteer.launch();            // headless by default
const page = await browser.newPage();

await page.goto('https://quotes.toscrape.com/js/', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('.quote');                // wait for the data, not for a timer

const quotes = await page.$$eval('.quote', cards => cards.map(card => ({
  text: card.querySelector('.text')?.textContent.trim(),
  author: card.querySelector('.author')?.textContent.trim(),
  tags: [...card.querySelectorAll('.tag')].map(t => t.textContent.trim()),
})));

await writeFile('quotes.json', JSON.stringify(quotes, null, 2));
console.log(`Saved ${quotes.length} quotes`);
await browser.close();

What each step does:

1. puppeteer.launch() starts the browser. It runs headless unless you pass headless: false, which is useful while you write selectors.
2. page.goto() loads the URL and resolves when the event named in waitUntil fires.
3. page.waitForSelector('.quote') waits until the first quote is in the DOM, up to the default 30-second timeout.
4. page.$$eval() runs the callback inside the page on every matching element and returns plain data to Node.js.
5. browser.close() ends the browser process; forgetting it leaves Chrome processes running.

waitUntil tells page.goto() (and page.setContent(), page.reload(), page.waitForNavigation()) when to consider the page loaded. The default is load. The two network-idle values count open network connections (pptr.dev: PuppeteerLifeCycleEvent):

networkidle2 is the safer of the two idle options on real sites, because chat widgets, analytics and live updates often keep one or two connections open, and networkidle0 would then wait until the 30-second timeout. Both values are still part of the current API. For scraping, the most reliable wait is usually neither: load with domcontentloaded, then wait for the element that holds your data.

Three more waits you will use:

• page.waitForSelector(selector, { visible: true }) waits for an element to exist and be visible.
• page.waitForNetworkIdle({ idleTime: 500 }) waits for the network to go quiet at any point, for example after a click or a scroll (pptr.dev: waitForNetworkIdle).
• page.waitForResponse(res => res.url().includes('/api/')) waits for a specific data request, and gives you its JSON.

page.waitForTimeout() was removed in Puppeteer 22. If an old tutorial uses it, replace it with one of the waits above, or with await new Promise(r => setTimeout(r, 1000)) when you really need a fixed pause.

Puppeteer gives you four ways to read a page; pick by how much data you need.

• page.$eval(selector, fn) runs fn on the first match: await page.$eval('h1', el => el.textContent).
• page.$$eval(selector, fn) runs fn on the array of all matches, as in the script above.
• page.evaluate(fn) runs any code in the page and returns serializable data; use it to read window variables or embedded JSON.
• page.locator(selector) returns a locator that waits for the element to be ready before it clicks, fills or reads it (pptr.dev: Locators).

Many sites ship their data as JSON inside the page, in a <script type="application/ld+json"> tag or a framework state object. Reading that JSON is more stable than reading the visible layout:

const products = await page.$$eval('script[type="application/ld+json"]',
  tags => tags.map(t => JSON.parse(t.textContent)));

Click with page.click(selector) or a locator; when the click opens a new page, start waiting for the navigation before you click:

await page.locator('button.load-more').click();                 // same page, more items
await Promise.all([page.waitForNavigation(), page.click('li.next > a')]);  // new page
await page.locator('input[name="q"]').fill('running shoes');    // type into a form field

To scrape every page of a list, loop until the "Next" link disappears:

const all = [];
let url = 'https://quotes.toscrape.com/js/';
while (url) {
  await page.goto(url, { waitUntil: 'domcontentloaded' });
  await page.waitForSelector('.quote');
  all.push(...await page.$$eval('.quote .text', els => els.map(e => e.textContent)));
  url = await page.$eval('li.next > a', a => a.href).catch(() => null);  // null on the last page
}

Reading the next page's URL and calling page.goto() is sturdier than clicking, because you can retry a URL and resume from it after a crash.

Scroll to the bottom, wait for the new items to load, and stop when the page height stops growing. The cap on rounds keeps a feed that never ends from running forever.

await page.goto('https://quotes.toscrape.com/scroll', { waitUntil: 'networkidle2' });
let lastHeight = await page.evaluate(() => document.body.scrollHeight);
for (let round = 0; round < 30; round++) {
  await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
  await page.waitForNetworkIdle({ idleTime: 500, timeout: 10_000 }).catch(() => {});
  const height = await page.evaluate(() => document.body.scrollHeight);
  if (height === lastHeight) break;            // nothing new loaded
  lastHeight = height;
}
const texts = await page.$$eval('.quote .text', els => els.map(e => e.textContent));

Infinite feeds usually load each batch from a JSON endpoint. Open the browser's DevTools, Network tab, filter by Fetch/XHR and scroll: if you see a request like /api/quotes?page=2, calling that endpoint directly is faster than scrolling, and you can do it without a browser.

page.screenshot() saves the viewport, the full page or a clipped area as PNG, JPEG or WebP; an element handle can screenshot only itself (pptr.dev: screenshot).

await page.setViewport({ width: 1366, height: 900 });
await page.screenshot({ path: 'page.png', fullPage: true });
const card = await page.$('.quote');
await card.screenshot({ path: 'first-quote.png' });

Screenshots are useful as evidence for price and ad checks, and for debugging a selector that returns nothing on the server but works on your laptop.

A browser is the most expensive way to fetch a page, so most of the speed comes from doing less in it:

For many URLs, puppeteer-cluster is an open-source library outside the Puppeteer project that runs a pool of pages or browser contexts with a queue, retries and a concurrency limit; its latest release (0.25.0, November 2025) lists Puppeteer 24 or later as its peer dependency, so check it against your Puppeteer version before you rely on it.

import { Cluster } from 'puppeteer-cluster';
const cluster = await Cluster.launch({ concurrency: Cluster.CONCURRENCY_CONTEXT, maxConcurrency: 4 });
await cluster.task(async ({ page, data: url }) => {
  await page.goto(url, { waitUntil: 'domcontentloaded' });
  console.log(url, await page.title());
});
['https://quotes.toscrape.com/js/page/1/', 'https://quotes.toscrape.com/js/page/2/'].forEach(u => cluster.queue(u));
await cluster.idle();
await cluster.close();