Scraper API vs Proxies vs Datasets

The options differ in what you pay for and in who keeps the scraper working when the target site changes.

A proxy is a server that forwards your scraper's requests, so the target site sees the proxy's IP instead of yours. With a rotating pool, each request can leave from a different IP, which keeps any one address under the site's limits. Everything else stays with you: the code, the headless browsers for JavaScript pages, the retries and the parsing.

Rotating proxies are billed per GB, so the cost follows page size, not page count. ProxyField residential proxies cost $2.95/GB for 1 GB, $2.79 from 10 GB, $2.65 from 50 GB, $2.49 from 100 GB, $2.29 from 250 GB, $2.09 from 500 GB and $1.95 from 1,000 GB; the tier rate applies to the whole purchase, there is no subscription and the traffic never expires. Mobile proxies cost $3.45/GB down to $2.45, and rotating datacenter proxies $0.29/GB down to $0.22.

A proxy only changes the IP your request comes from. With proxies, you handle yourself:

• blocks: rate limits, challenges and empty responses, and when to slow down or switch IPs;
• browser fingerprints and headers, which a new IP does not change;
• JavaScript: running a headless browser where the data is not in the HTML;
• retries, with a cap, because failed requests still use traffic;
• monitoring: success rate and bytes per target, so a silent failure does not run for a week;
• site changes: selectors that break when the layout changes.

Proxies win when:

• you scrape a few sites at high volume and can maintain the code;
• pages are light HTML or JSON, so each GB covers many pages;
• you need control over sessions, timing, headers and location, down to the city.

A scraper API takes a URL, or a query such as a search term, and returns the page or structured data. The provider runs the IP pool, the browsers and the retries, and many APIs parse common pages into JSON fields. You pay per request or per successful result, so the cost follows page count, not page size.

ProxyField's scraper APIs are in development and priced already: billed on successful results only, per 1,000 successful results, entry price → volume price:

The volume price applies from 10,000 successful results in one purchase. The APIs are paid from the same crypto balance as proxies, with no subscription; the planned design is in the API documentation. Contact sales.

A scraper API wins when:

• the targets block hard and you would rather not tune browsers and sessions;
• pages are heavy, so per-result billing costs less than per-GB traffic;
• nobody on the team can maintain scrapers, or you need results this week.

A dataset is a file or a feed of data already scraped and cleaned, such as a catalog of products or a list of companies. It is the fastest route when the data is standard and you do not need it fresher than the seller's schedule. You give up control over what is collected, how and when, and you still have to check that the fields and coverage answer your question. ProxyField sells proxies and is building scraper APIs; it does not sell datasets.

Take 100,000 product pages a month. The page sizes below are examples; measure a sample of your real targets before you buy. For scale, the HTTP Archive's Web Almanac 2025 puts the median home page at 2.56 MB on mobile and 2.86 MB on desktop, with the median HTML response at 33 KB on mobile and 35 KB on desktop (third-party transfer sizes, not ProxyField figures).

Where the lines cross. Per 1,000 pages, proxies cost $2.79 × the GB those pages use:

At the API's volume prices the crossover sits near 70 KB per page for plain HTML and near 360 KB per page for rendered pages, before you count servers and the time someone spends fixing scrapers, which is often the largest line.

Answer: skip the browser. At an example 10 KB per JSON call, 100,000 calls use about 1 GB of proxy traffic, $2.95, against about 200 GB, $498, for the same pages in a full browser at an example 2 MB per page. Many JavaScript-heavy pages load their data from a JSON request you can call directly; render a page only when no such request exists, and keep the browser light. The Scrapy documentation on dynamic content gives the same order: reproduce the request first, use a headless browser last.

1. Find the data request. Open the page in Chrome, press F12, choose Network, filter by Fetch/XHR and reload. Click requests until one returns your data as JSON (Chrome DevTools: Network panel).

2. Copy it. Right-click the request, Copy > Copy as cURL. In Scrapy, Request.from_curl() turns that line into a request. To test whether the data is already in the raw HTML, run scrapy fetch --nolog https://www.example.com/product/1 > response.html and search the file for a value you can see on the page.

3. Call it directly, through the proxy, and change the page or ID parameters to walk through the data:

   PythonCopy Python code

   import requests
   
   proxy = "http://<login>-country-us:<password>@gate.proxyfield.io:7000"
   r = requests.get("https://www.example.com/api/products?page=2",
                    headers={"Accept": "application/json"},
                    proxies={"http": proxy, "https": proxy}, timeout=30)
   items = r.json()

4. No separate request? Look inside the HTML. Many sites ship the data as JSON in a <script> tag, such as __NEXT_DATA__ on Next.js sites; parse it from the plain HTML, still without a browser. Valid JSON goes through json.loads(); a JavaScript object that is not valid JSON (unquoted keys, single quotes) goes through chompjs.parse_js_object() from the chompjs library:

   PythonCopy Python code

   import json, re, chompjs, requests
   
   html = requests.get("https://www.example.com/product/1",
                       proxies={"http": proxy, "https": proxy}, timeout=30).text
   m = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.S)
   if m:
       data = json.loads(m.group(1))                                # valid JSON
   else:
       js = re.search(r"window\.__INITIAL_STATE__\s*=\s*(\{.*?\});", html, re.S).group(1)
       data = chompjs.parse_js_object(js)                           # a JavaScript object literal

   Still in a browser for some other reason? Let the page make its own data request and keep the JSON: Playwright's page.expect_response() (or page.on("response", ...)) returns the response, and response.json() gives the data without parsing the DOM (Playwright: network).

5. Render only what is left, in a headless browser such as Playwright, and keep it light: reuse one browser for many pages, block images, fonts and media, and wait for the one element you need instead of a fixed delay.

   PythonCopy Python code

   from playwright.sync_api import sync_playwright
   
   BLOCK = {"image", "font", "media"}
   with sync_playwright() as p:
       browser = p.chromium.launch(proxy={"server": "http://gate.proxyfield.io:7000",
                                          "username": "<login>-country-us", "password": "<password>"})
       page = browser.new_page()
       page.route("**/*", lambda route: route.abort()
                  if route.request.resource_type in BLOCK else route.continue_())
       for url in ["https://www.example.com/product/1", "https://www.example.com/product/2"]:
           page.goto(url, wait_until="domcontentloaded")
           page.wait_for_selector(".price")
           print(page.inner_text(".price"))
       browser.close()

   In Scrapy, the scrapy-playwright plugin renders only the requests you mark with meta={"playwright": True}; it needs the asyncio reactor (TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor") and its download handlers in settings.py (Scrapy for JavaScript pages).

6. Or hand the rendering to an API billed per result: JS rendering on the web scraping API is priced at $2.00 per 1,000 successful results, $1.00 from 10,000 results (in development; Contact sales).

What each step saves on 100,000 pages a month, with example sizes:

More: scraping dynamic web pages, scraping hidden APIs and headless browser scraping.

Collecting web data costs more than the bandwidth suggests, because:

• Sites limit requests per IP, so volume needs a pool of IPs, and protected sites need residential or mobile IPs, which cost more than datacenter ones.
• JavaScript pages need a browser, which uses more CPU, memory and traffic than a plain request.
• Failures repeat work: blocked or empty responses must be retried, and you pay for the traffic either way with proxies.
• Layouts change, and every change means engineering time to fix selectors.
• Data needs cleaning, deduplication and storage before anyone can use it.

A scraper API moves the first three to the provider and charges for successful results; proxies keep them with you and charge for traffic.

Many teams use both. The scraper API handles the hardest targets, such as search engines, and the pages that need rendering; their own scraper on rotating proxies handles the high-volume, simple pages where per-GB billing is lowest. For AI agents the split is similar: an API returns clean results in one call, while a proxy lets the agent's own browser act like one visitor across several pages.

Proxies by job and what each job costs are worked through on proxies for web scraping. All purchases are final; in return, proxy traffic never expires.

• The site's terms. Public pages only, and follow the site's terms; the route you pick does not change them.
• robots.txt. It says which paths the site asks crawlers to leave alone and sometimes a crawl delay; read it before you size a job.
• Personal data. Names, profiles and reviews are personal data under the GDPR in the EU and the UK; collect only the fields your purpose needs.
• Logins and access controls. Data behind a login you are not entitled to is off limits on either route, under ProxyField's acceptable use policy.
• Rate. Keep a modest rate per IP and per site; a pool of IPs is for spreading load, not multiplying it.
• The provider's billing. What counts as a billed result (ProxyField's APIs bill successful results only; a target's 404 or 410 counts as a result), whether traffic expires, and the smallest purchase. ProxyField's proxy traffic never expires and the smallest purchase is 1 GB.

The law differs by country and by what you collect: Is Web Scraping Legal? US, EU and UK Rules (2026). This is not legal advice.