How to Scrape Dynamic JavaScript Websites With Python

A page is dynamic, for a scraper, when the data you want is not in the HTML the server sends but is added by JavaScript in the browser. Test it in a minute:

import requests
html = requests.get("https://example.com/products", headers={"User-Agent": "Mozilla/5.0"}, timeout=30).text
print("Blue Widget" in html)     # a value you can see on the page

True means the value is in the raw HTML: Requests and Beautiful Soup are enough. False means a script loads it. The same check in the browser: View Source (Ctrl+U) and search for the value, or disable JavaScript in DevTools and reload. The background, for any language, is in how to scrape dynamic web pages.

Open DevTools, go to Network, filter by Fetch/XHR, and reload or scroll. Click each request and look at its Response tab: one of them returns the products, prices or comments as JSON. Right-click it and Copy as cURL to see the exact URL, parameters and headers.

import requests

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
s = requests.Session()
s.proxies.update({"http": PROXY, "https": PROXY})
s.headers.update({"User-Agent": "Mozilla/5.0", "Accept": "application/json",
                  "Referer": "https://example.com/products"})

items, page = [], 1
while True:
    data = s.get("https://example.com/api/products", params={"page": page, "per_page": 48},
                 timeout=30).json()
    items += data["items"]
    if page >= data["total_pages"]:
        break
    page += 1
print(len(items))

Keep the headers the browser sent that the API checks (often Referer, Accept and sometimes an X-Requested-With or a token the page fetched first). If the request needs cookies from a first page load, load that page with the same Session before calling the API. Call only public endpoints the page itself uses, at the page's own pace.

Many sites built with frameworks ship their data inside the HTML as a script, so the page can render without a second request. Search the page source for the value; if it sits inside <script id="__NEXT_DATA__">, <script type="application/ld+json"> or window.__INITIAL_STATE__ = {...}, you can read it without a browser.

import json
from bs4 import BeautifulSoup

soup = BeautifulSoup(s.get("https://example.com/product/123", timeout=30).text, "html.parser")

next_data = soup.select_one("script#__NEXT_DATA__")
if next_data:
    props = json.loads(next_data.string)["props"]["pageProps"]

for tag in soup.select('script[type="application/ld+json"]'):
    data = json.loads(tag.string)
    if isinstance(data, dict) and data.get("@type") == "Product":
        print(data.get("name"), data.get("offers", {}).get("price"))

Structured data in application/ld+json follows schema.org types, so product name, price and rating fields have stable names across sites.

When there is no API and no embedded JSON, render the page. Playwright for Python (1.64.1, Apache-2.0, Python 3.10+ per PyPI on 2026-10-23) drives Chromium, Firefox and WebKit, waits for elements automatically, and takes a proxy with a username and password.

pip install playwright beautifulsoup4
playwright install chromium

from playwright.sync_api import sync_playwright

BLOCK = {"image", "media", "font"}

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={"server": "http://gate.proxyfield.io:7000",
                                       "username": "<login>-country-us-city-new_york",
                                       "password": "<password>"})
    page = browser.new_page()
    page.route("**/*", lambda route: route.abort() if route.request.resource_type in BLOCK
               else route.continue_())                       # save time and traffic
    page.goto("https://example.com/products", wait_until="domcontentloaded")
    page.wait_for_selector("div.product-card")
    rows = page.locator("div.product-card").evaluate_all(
        "cards => cards.map(c => ({name: c.querySelector('h2')?.innerText,"
        " price: c.querySelector('.price')?.innerText}))")
    browser.close()
print(len(rows), rows[:2])

What makes it reliable:

• Wait for the element, not a fixed time. wait_for_selector (or any locator action) waits until the element exists; time.sleep(5) is either too long or too short.
• Block what you don't need. Images, media and fonts are most of a page's weight; aborting them can cut a page's traffic several times over. Keep CSS and scripts if the page needs them to build the content.
• Catch the data in flight. page.expect_response("**/api/products*") captures the JSON the page fetches, which is cleaner than parsing the rendered HTML.

Infinite scroll and "Load more"

page.goto("https://example.com/feed")
seen = 0
while True:
    page.mouse.wheel(0, 5000)
    page.wait_for_timeout(1200)
    count = page.locator("article").count()
    if count == seen:          # nothing new arrived
        break
    seen = count
# or click a button until it disappears:
# while page.locator("button:has-text('Load more')").is_visible(): page.click("button:has-text('Load more')")

On infinite scroll, Method 1 is usually better: the scroll triggers an API call with a page or cursor parameter, and you can call it directly.

Full Playwright proxy setup, including one location per browser context: Playwright proxy setup.

Selenium 4.51.0 (Apache-2.0, PyPI 2026-10-23) works the same way: load the page, wait explicitly, read the elements. Use WebDriverWait with an expected condition instead of sleep.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_argument("--proxy-server=http://gate.proxyfield.io:<port>")   # IP allowlist auth
options.page_load_strategy = "eager"          # don't wait for every image
driver = webdriver.Chrome(options=options)
try:
    driver.get("https://example.com/products")
    cards = WebDriverWait(driver, 20).until(
        EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.product-card")))
    print([c.find_element(By.TAG_NAME, "h2").text for c in cards])
finally:
    driver.quit()

Chrome's --proxy-server flag takes no username or password, so this version uses an IP allowlist: add your machine's IP to your account's allowlist first, and replace <port> with the port you set up in the dashboard for it. For username and password authentication: Selenium proxy setup.

For a crawl where only some pages need a browser, the scrapy-playwright plugin lets a Scrapy spider render just those requests (meta={"playwright": True}) while the rest stay plain HTTP. For Method 1 at volume, httpx.AsyncClient with asyncio.gather and a semaphore fetches many API pages at once. Both patterns, with limits that keep sites happy, are in how to make web scraping faster; Scrapy setup is in Scrapy proxy setup.

Dynamic sites are often the ones with stricter bot rules, and a rendered page weighs much more than its JSON. Practical settings:

• Rotation: a new IP per request for API pages; a sticky session (30 seconds to 36 hours) for a browser that loads a page and then its API calls, so they come from one visitor.
• Location: country, region and city targeting when the site shows local prices or content. The gateway is gate.proxyfield.io:7000; <login>-country-us-city-new_york asks for an IP in New York. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.
• Traffic: compare one page's size with blocking on and off in DevTools' Network panel before you size a job; ProxyField residential proxies cost $2.95/GB for 1-9 GB down to $1.95/GB from 1,000 GB, and unused traffic never expires.

If you would rather not run browsers, the ProxyField Web Scraping API fetches any URL with plain HTTP or with JavaScript rendering, at $2.00 per 1,000 successful results with rendering, down to $1.00 at volume, and you pay only for successful results; ask for it through Contact sales.

Scraping JavaScript-heavy sites from Python? ProxyField residential proxies: 412M+ residential IPs in 195+ countries, per-request rotation or sticky sessions, unlimited concurrent threads. Pay per GB, no subscription; start with the Easy Trial (1 GB). Traffic never expires, and all purchases are final.