How to Scrape Hidden APIs and Embedded JSON

Most modern pages are a shell that JavaScript fills with data after it loads. That data comes from somewhere: a request to the site's own backend, usually returning JSON, or a block of JSON the server already put into the HTML. Neither is secret. These are the site's internal APIs, undocumented but visible to anyone who opens the browser's developer tools, and they are what "hidden API" means in scraping.

Scraping them has three advantages over parsing the rendered page:

• Clean data. You get fields with names (price, stock, author) instead of text you must cut out of HTML.
• Stability. A redesign changes the HTML far more often than the JSON behind it.
• Less traffic and no browser. One JSON response replaces a page plus its scripts, styles and images, and a plain HTTP client replaces a headless browser.

Scrapy's documentation recommends the same order: find the data source first, use the browser's network tool to find the request, and reproduce it; a headless browser is the last resort (Scrapy: selecting dynamically-loaded content).

1. Open the page in Chrome or Edge, press F12, and open the Network panel.
2. Click the Fetch/XHR filter, so only data requests are listed (Chrome DevTools: Network reference).
3. Reload the page, scroll, click "next" or "load more": whatever makes new data appear.
4. Click each new request and open Preview or Response. You are looking for JSON that contains the values you see on the page. Searching the panel (Ctrl+F) for a product name or price finds it fast.
5. Note the request's URL, method (GET or POST), query parameters, and the headers on the Headers tab.

On the practice page quotes.toscrape.com/scroll, scrolling down fires GET /api/quotes?page=2, then page=3, and so on. Each response is JSON with a quotes list and a has_next flag.

Right-click the request, choose Copy, then Copy as cURL. DevTools copies the full request, with headers and cookies, as a command you can paste into a terminal to check that it works outside the browser. Then remove headers one by one until you know which ones the server needs; usually that is a few of Accept, Referer, User-Agent, a cookie, or a token header.

Two tools turn the cURL command into code:

• Scrapy: scrapy.Request.from_curl("curl 'https://...' -H '...'") builds the same request in a spider. We checked it on October 23, 2026 with Scrapy 2.19.0; it kept the URL, method and headers.
• Python Requests: copy the URL, params and the headers you kept into a requests.Session, as below.

More on cURL itself: cURL with proxies.

# pip install requests
import time
import requests

session = requests.Session()
session.headers.update({
    "Accept": "application/json",
    "Referer": "https://quotes.toscrape.com/scroll",
    "User-Agent": "Mozilla/5.0 (compatible; my-scraper/1.0)",
})

quotes, page = [], 1
while True:
    r = session.get("https://quotes.toscrape.com/api/quotes", params={"page": page}, timeout=30)
    r.raise_for_status()
    data = r.json()                      # already a Python dict: no HTML parsing
    for q in data["quotes"]:
        quotes.append({"text": q["text"], "author": q["author"]["name"], "tags": q["tags"]})
    if not data["has_next"]:
        break
    page += 1
    time.sleep(0.5)

print(len(quotes), quotes[0]["author"])

We ran it on October 23, 2026 (Requests 2.34.2): 10 requests, 100 quotes, no HTML parsed. r.json() returns Python dicts and lists, so the fields are ready to write to CSV, a database or a pandas DataFrame (pandas.json_normalize(quotes) flattens nested fields).

Sometimes there is no data request, because the server puts the data into the HTML for the page's JavaScript to read. View the page source (not the Elements panel) and search for a value you see on the page. Common places:

• A JavaScript variable such as var data = [...] or window.__INITIAL_STATE__ = {...}.
• A framework's data tag, such as <script id="__NEXT_DATA__" type="application/json"> on sites built with Next.js's Pages Router, or similar state blocks in other frameworks.
• JSON-LD in <script type="application/ld+json">: structured data (product, price, rating, article) that sites add for search engines; see HTML scraping for an example.

The JavaScript version of Quotes to Scrape keeps its quotes in a variable:

import json, re
import requests

html = requests.get("https://quotes.toscrape.com/js/", timeout=30).text
match = re.search(r"var data = (\[.*?\]);", html, re.S)
data = json.loads(match.group(1))
print(len(data), data[0]["author"]["name"], data[0]["tags"])

It printed 10 quotes when we ran it. A regular expression is fine when the JSON sits in a clear var x = ...; statement, as Scrapy's docs also show. When the object is JavaScript rather than strict JSON (unquoted keys, trailing commas), json.loads() fails; the chompjs package parses JavaScript objects into Python dicts. For a <script type="application/json"> tag, select it with Beautiful Soup or parsel and pass its text straight to json.loads().

Real endpoints add details the practice site does not:

Keep the request rate close to what a person browsing would cause, read Retry-After on 429 responses, and stop if the endpoint starts to fail. Web scraping best practices covers retries and backoff.

Some sites build requests with signatures computed in JavaScript, or only return data after a browser check. Then let a browser do the requesting and capture the JSON as it arrives. In Playwright for Python, page.on("response", handler) hands you each response, and response.json() gives the data without touching the DOM. You still avoid parsing HTML; you only borrow the browser to make the call. More on this in Python dynamic web scraping and how to scrape dynamic web pages.

An internal API returns the same public data as the page, so the same rules apply as for scraping the page: collect public data, respect the site's terms and robots.txt, avoid personal data without a lawful basis, and keep the load low. Do not use credentials, keys or sessions that are not yours, and do not get around a login or a paywall. Official, documented APIs are the better route when the site offers one; web scraping vs API compares the two, and is web scraping legal covers the law.