Web Scraping With JavaScript and Node.js

Yes, in Node.js. JavaScript is a natural fit for scraping because it is the language of the web: the same CSS selectors you test in the browser's DevTools work in your scraper, JSON is native, and Node's asynchronous I/O fetches many pages at once without threads. Node.js also has the two most used browser automation libraries, Puppeteer and Playwright.

Scraping from a script in the browser itself (the DevTools console or a bookmarklet) works only for the page you are on: browsers block a page from reading other sites' responses unless those sites allow it (the same-origin policy and CORS, MDN: CORS). Node.js has no such limit, which is why scrapers run there.

A wider list across languages is in web scraping tools compared.

Open the page, press Ctrl+U (Cmd+Option+U on a Mac) to view the source the server sent, and search for a value you want, such as a price. If it is in the source, the page is static for your purposes: use fetch and Cheerio. If the source is an almost empty <div id="root"> and the data appears only in the rendered page, it is built by JavaScript (React, Vue or AJAX calls): look for its JSON request first (DevTools, Network tab, Fetch/XHR), and use Puppeteer or Playwright only if there is no clean request to call.

The script below collects every book's title, price and stock status from a practice catalog, follows the "next" link through all pages, waits a second between pages, and saves JSON.

mkdir books-scraper && cd books-scraper
npm init -y && npm pkg set type=module
npm i cheerio

// scrape.mjs  ->  node scrape.mjs
import * as cheerio from 'cheerio';
import { writeFile } from 'node:fs/promises';

const books = [];
let url = 'https://books.toscrape.com/catalogue/page-1.html';

while (url) {
  const res = await fetch(url, { headers: { 'user-agent': 'Mozilla/5.0 (compatible; books-demo/1.0)' } });
  if (!res.ok) throw new Error(`HTTP ${res.status} on ${url}`);
  const $ = cheerio.load(await res.text());

  $('article.product_pod').each((_, el) => {
    books.push({
      title: $(el).find('h3 a').attr('title'),
      price: $(el).find('.price_color').text().trim(),
      inStock: $(el).find('.instock').length > 0,
    });
  });

  const next = $('li.next a').attr('href');
  url = next ? new URL(next, url).href : null;     // resolve the relative link
  await new Promise(r => setTimeout(r, 1000));     // one request per second
}

await writeFile('books.json', JSON.stringify(books, null, 2));
console.log(`Saved ${books.length} books`);

How it works: fetch returns the HTML, cheerio.load() turns it into a queryable document, $(selector) finds elements as in jQuery, and new URL(next, url) turns the relative "next" link into an absolute address.

Tables and lists

HTML tables map straight to rows:

const rows = $('table tr').map((_, tr) =>
  [$(tr).find('th, td').map((_, c) => $(c).text().trim()).get()]).get();

Each inner array is one row; write it to CSV by joining cells with commas after escaping quotes, or use a CSV library.

Saving the data

JSON.stringify(data, null, 2) with writeFile is enough for most jobs. For large crawls, write one JSON object per line (JSON Lines) as you go, so a crash does not lose what you already scraped:

import { appendFile } from 'node:fs/promises';
await appendFile('books.jsonl', JSON.stringify(book) + '\n');

A crawler starts from one URL, collects the links on each page, and visits the ones on the same site it has not seen. A queue, a Set of visited URLs and a page limit are all it needs:

import * as cheerio from 'cheerio';

const start = new URL('https://books.toscrape.com/');
const queue = [start.href];
const seen = new Set(queue);
const MAX_PAGES = 200;
let visited = 0;

while (queue.length && visited < MAX_PAGES) {
  const url = queue.shift();
  visited++;
  const res = await fetch(url);
  if (!res.ok || !res.headers.get('content-type')?.includes('text/html')) continue;
  const $ = cheerio.load(await res.text());
  console.log(url, '-', $('title').text().trim());

  $('a[href]').each((_, a) => {
    const link = new URL($(a).attr('href'), url);
    link.hash = '';
    if (link.host === start.host && !seen.has(link.href)) {
      seen.add(link.href);
      queue.push(link.href);
    }
  });
  await new Promise(r => setTimeout(r, 500));
}

Before you crawl a site that is not yours, read its robots.txt and terms, keep the request rate low, and stop at a page limit. For crawls of thousands of pages, a framework such as Crawlee adds persistent queues, retries, concurrency limits and proxy rotation. The general method is in how to crawl a website.

When the data is not in the page source, run the page in a browser. Puppeteer loads it, runs its scripts, and lets you read the rendered DOM:

// npm i puppeteer
import puppeteer from 'puppeteer';

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://quotes.toscrape.com/js/', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('.quote');
const quotes = await page.$$eval('.quote .text', els => els.map(e => e.textContent));
console.log(quotes);
await browser.close();

Waits, infinite scroll, screenshots and performance are covered in web scraping with Puppeteer; the Playwright version is in web scraping with Playwright. For pages built with React or loaded by AJAX, check the Network tab first: the JSON the page requests is usually easier to scrape than the page.

const data = await (await fetch('https://example.com/api/products?page=2')).json();

Everything above works in TypeScript: Cheerio, Puppeteer, Playwright and Crawlee ship their own type definitions. Types help most in scraping where they define the shape of a scraped record, so a selector that returns undefined fails at the type check or the validation step instead of in your database. Run TypeScript files directly with npx tsx scrape.ts, or on Node.js versions that strip types natively.

"Screen scraping" is the older term for reading data from what a program displays; on the web it means the same as browser-based scraping. In Node.js, that is Puppeteer or Playwright reading the rendered page, or taking screenshots with page.screenshot() when you need the visual result. Reading text from images (OCR) is a different job that needs an OCR library.

Benefits:

• One language for the browser, the scraper and often the backend.
• Asynchronous I/O fetches many pages concurrently with little code.
• The DevTools selectors you test by hand work unchanged in Cheerio and Puppeteer.
• First-class browser automation (Puppeteer, Playwright) and crawling frameworks (Crawlee).

Challenges and fixes: