How to Scrape Dynamic Web Pages

A static page arrives from the server with its content already in the HTML. A dynamic page arrives as a shell, and JavaScript in the browser fills it in: it calls an API, receives JSON and builds the list, the prices or the comments on the spot. Single-page apps work this way, and so do many ordinary sites for parts of a page:

• product grids, search results and filters that update without a reload;
• infinite scroll and "Load more" buttons;
• prices, stock or reviews loaded after the main page;
• content that appears only after a click, a hover or a cookie banner.

Four checks, each under a minute:

An empty list, None from a selector or a page that looks like a skeleton almost always means one of these:

Most dynamic pages get their data from a JSON endpoint, and that endpoint is the cleanest thing to scrape: smaller than the page, already structured, and stable across redesigns more often than the HTML. In the Network panel, click the request that holds your data and look at:

• the URL and parameters: page, offset, cursor, limit, filters;
• the headers the endpoint needs: often Accept, Referer, sometimes a token the page got from an earlier request;
• GraphQL: a POST to /graphql with a query and variables in the body; replay the same body with your own variables.

Right-click and Copy as cURL to reproduce it, then move it into code. Call only the endpoints the public page itself uses, at a pace a visitor would.

Frameworks often ship the page's data inside the HTML so the first render is fast. Look in the source for:

• <script id="__NEXT_DATA__" type="application/json"> (Next.js),
• window.__NUXT__ (Nuxt) or window.__INITIAL_STATE__ / __APOLLO_STATE__ assignments,
• <script type="application/ld+json"> with schema.org data such as Product, Offer or JobPosting.

Extract the script's text, parse it as JSON, and you have the data without running a browser. For a Python walkthrough of Methods 1 and 2, see Python dynamic web scraping.

When the data exists only after scripts run, use a browser. This Node.js example uses Playwright (npm playwright 1.64.1, Apache-2.0, 2026-10-23): it routes through a proxy, blocks heavy resources, waits for the product cards, and also captures the JSON the page fetches, which is often the better source.

// npm i playwright && npx playwright install chromium
// save as scrape.mjs (ES module, top-level await)
import { chromium } from 'playwright';

const browser = await chromium.launch({
  proxy: { server: 'http://gate.proxyfield.io:7000',
           username: '<login>-country-us-city-new_york', password: '<password>' },
});
const page = await browser.newPage();
await page.route('**/*', route =>
  ['image', 'media', 'font'].includes(route.request().resourceType())
    ? route.abort() : route.continue());

const apiResponse = page.waitForResponse(r => r.url().includes('/api/products') && r.ok())
  .catch(() => null);                                   // resolves to null if no such request
await page.goto('https://example.com/products', { waitUntil: 'domcontentloaded' });
await page.locator('div.product-card').first().waitFor();

const cards = await page.locator('div.product-card').evaluateAll(els =>
  els.map(e => ({ name: e.querySelector('h2')?.innerText, price: e.querySelector('.price')?.innerText })));
const res = await apiResponse;
const json = res ? await res.json().catch(() => null) : null;   // the data the page itself received

console.log(cards.length, json ? 'API captured' : 'no API response');
await browser.close();

Patterns for the common cases:

• Wait for elements, not seconds. locator.waitFor() or page.waitForSelector(); a fixed waitForTimeout(5000) is flaky and slow.
• Infinite scroll: scroll (page.mouse.wheel(0, 5000)), wait briefly, and stop when the item count stops growing; or call the API the scroll triggers.
• "Load more" buttons and tabs: click until the button disappears, waiting for the new items after each click.
• Dynamic tables: read the table after it renders with evaluateAll, or, better, take the JSON the table was built from.

The same patterns in Puppeteer and Selenium, with proxy setup: Puppeteer proxy setup, Playwright proxy setup, Selenium proxy setup. Headless browser trade-offs in depth: headless browser scraping.

Crawling adds a second problem: the links themselves may be created by JavaScript, so an HTTP crawler finds no URLs to follow. In order of preference:

1. Use the sitemap (/sitemap.xml, or the Sitemap: line in robots.txt): it lists the URLs without rendering anything.
2. Use the listing API to enumerate items, then build each detail URL.
3. Render to discover links: a browser crawler such as Crawlee's PlaywrightCrawler renders each page and enqueues the links it finds. Render the listing pages, and fetch detail pages over plain HTTP if they are static.

Search engines face the same problem: Google renders JavaScript pages in a second pass, after crawling the HTML, as described in Google Search Central: JavaScript SEO basics. Most other crawlers and simple scrapers do not run JavaScript at all.

Scrapy and JavaScript: Scrapy does not execute JavaScript. Its documentation recommends finding the data source first; for pages that truly need rendering, the scrapy-playwright plugin renders only the requests you mark with meta={"playwright": True} (Scrapy: selecting dynamically-loaded content).

Rendering one page is easy; rendering a million is an infrastructure job.

Measure before you size a job: open DevTools' Network panel and compare the transferred size of one page with images, media and fonts allowed and blocked. Then:

• Rotation: per-request rotation for API calls and independent pages; a sticky session (30 seconds to 36 hours) when a browser loads a page and then makes its own API calls, so they all come from one visitor.
• Location: country, region and city targeting for prices and content that change by place. The ProxyField gateway is gate.proxyfield.io:7000; <login>-country-us-city-new_york asks for an IP in New York. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.
• Concurrency: ProxyField has unlimited concurrent threads, so the limit is the site's tolerance and your hardware; see how to make web scraping faster.

Residential proxies cost $2.95/GB for 1-9 GB down to $1.95/GB from 1,000 GB, with no subscription, and the traffic never expires: residential and mobile proxy pricing. If you would rather not run browsers, the ProxyField Web Scraping API fetches any URL at $2.00 per 1,000 successful results with JavaScript rendering, down to $1.00 at volume, or $0.50 with plain HTTP, down to $0.20, and you pay only for successful results; ask for it through Contact sales.

Scraping JavaScript-heavy sites? ProxyField residential proxies: 412M+ residential IPs in 195+ countries, per-request rotation or sticky sessions, from $1.95/GB. Start with the Easy Trial (1 GB); traffic never expires, and all purchases are final.