What Is a Scraper API? Scrape, Crawl and Map Endpoints

A scraper API wraps the whole fetching side of a scraper behind one endpoint. A typical call goes through six steps:

1. You send a request with the target URL and options: the country to fetch from, whether to render JavaScript, the device type, sometimes the fields you want.
2. The API picks an IP from its proxy pool, in the country you asked for.
3. It fetches the page, either as a plain HTTP request or in a headless browser that runs the page's scripts.
4. It checks the response. In a typical scraper API, a block page, a CAPTCHA page, a 403 or a 429 counts as a failure, and the API retries with another IP.
5. It parses the result when the API offers parsing: it returns the HTML as is, or turns it into JSON or Markdown.
6. It returns the result to you and records whether the request counts for billing.

From your side, scraping a page becomes one HTTP call. This is the request format of ProxyField's Web Scraping API as designed in the API documentation; the API is in development, so access is through Contact sales:

curl https://api.proxyfield.io/v1/scrape \
  -H "Authorization: Bearer <api_key>" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.example-shop.com/product/42", "render": false, "country": "us", "output": "json"}'

{
  "metadata": {"id": "req_41ac7e93d05f2b68", "api": "web_scrape", "status": "success", "billed": true, "cached": false, "tag": null},
  "status_code": 200,
  "final_url": "https://www.example-shop.com/product/42",
  "content_type": "text/html; charset=utf-8",
  "content": "<!DOCTYPE html><html>…</html>"
}

The parameters that decide the result: url; render (false is plain HTTP, true loads the page in a browser and sets the price); country, a two-letter code from the pool's 195+ countries; output (html, markdown, text, json or screenshot); and, with rendering, wait_for (a CSS selector) and up to 20 browser actions such as click and scroll. metadata.billed and the X-Billed header say whether the call counted. In this design, blocked pages are retried on new IPs within the request's timeout (120,000 ms by default), and the retries are not billed.

Scraper APIs split the work by scope: one page, a whole section of a site, or only the list of its pages.

In one line each: scrape is "give me this page", crawl is "explore this site and give me the pages' content", map is "tell me which pages exist".

The three chain together. Map finds the URLs, you filter them (for example, keep only /product/ pages), and scrape fetches each one. Crawl does map and scrape in one job, which is simpler but gives you less control over which pages you pay for. Two other endpoint types are common: search or SERP endpoints take a query and return parsed results, and batch endpoints take a list of known URLs and return a job.

What a crawl and a map call look like

The shape below is typical of scraper APIs that have crawl and map endpoints. It is a generic example on a placeholder host, not ProxyField's API: ProxyField's design has no crawl or map endpoint (see the next sections). Field names vary by provider; the flow does not.

1. Start a crawl. The call returns a job ID at once, not the pages:

curl -X POST https://api.scraper.example/v1/crawl \
  -H "Authorization: Bearer <api_key>" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.example-shop.com/category/shoes/", "limit": 500, "max_depth": 2,
       "include_paths": ["/product/"], "exclude_paths": ["/cart", "/login"],
       "webhook": "https://your-app.example.com/hooks/crawl"}'

{"id": "crawl_9f2c41d7", "status": "queued", "url": "https://api.scraper.example/v1/crawl/crawl_9f2c41d7"}

2. Poll the job (or wait for the webhook). While it runs you see progress; finished pages come back in pages of results, with a link to the next page:

curl https://api.scraper.example/v1/crawl/crawl_9f2c41d7 -H "Authorization: Bearer <api_key>"

{
  "id": "crawl_9f2c41d7",
  "status": "running",
  "completed": 120,
  "total": 500,
  "next": "https://api.scraper.example/v1/crawl/crawl_9f2c41d7?skip=100",
  "data": [
    {"url": "https://www.example-shop.com/product/42", "status_code": 200, "markdown": "# Trail Runner 3\n\nPrice: $129.95 …"}
  ]
}

The status moves from queued to running to completed (or failed). limit is the hard page cap and max_depth the link depth; without them a crawl stops only at the provider's default cap.

3. A map call is a single synchronous request that returns URLs only:

curl -X POST https://api.scraper.example/v1/map \
  -H "Authorization: Bearer <api_key>" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.example-shop.com", "search": "product", "limit": 1000}'

{"links": ["https://www.example-shop.com/product/42", "https://www.example-shop.com/product/43", "…"]}

How to configure a crawl job

A crawl job needs limits, or it will fetch far more pages than you meant to pay for. Set these before you start one:

• Start URL and scope: the section to crawl (/blog/ or /category/shoes/), and whether subdomains count.
• Depth: how many links away from the start URL to follow. Depth 2 or 3 covers most category trees.
• Page limit: a hard cap on pages, so a calendar or a filter combination cannot loop forever.
• Include and exclude patterns: keep /product/, skip /cart, /login, ?sort= and ?page= duplicates.
• Sitemap: use the site's sitemap to find pages faster, skip it, or crawl only the URLs it lists.
• Rendering: turn JavaScript rendering on only if the pages need it, because it usually costs more per page.
• Delivery: poll the job status, or receive a webhook when it finishes, and verify the webhook's signature.

Check permission before you set the limit. A crawl follows links on its own, so it reaches pages you never looked at. Read the site's robots.txt first: RFC 9309 defines its User-agent, Allow and Disallow rules, and a crawl should stay out of disallowed paths. Read the site's terms as well, because robots.txt is not the whole answer: an API makes fetching easier, but it does not give you permission to collect or republish a site's content. Scrape public pages only, and follow the site's terms.

Map then scrape, with ProxyField's API design

ProxyField's API design has a scrape endpoint and batch jobs rather than separate crawl and map endpoints, so you build map-then-scrape from those two. Most sites list their pages in an XML sitemap, which the sitemaps protocol caps at 50,000 URLs per file; robots.txt, defined in RFC 9309, usually names it in a Sitemap: line.

{
  "api": "web_scrape",
  "params": {"render": false, "country": "us", "output": "json"},
  "items": [{"url": "https://www.example-shop.com/product/42"}, {"url": "https://www.example-shop.com/product/43"}],
  "callback_url": "https://your-app.example.com/hooks/proxyfield",
  "tag": "shop-products"
}

4. Collect: each finished job is POSTed to your callback_url, signed with an X-ProxyField-Signature header (an HMAC-SHA256 of the body with your webhook secret), or you poll GET /v1/batches/{id}, which returns counts per status and one job ID per URL, in the order you sent them:

{
  "id": "bat_7e19c4a2d0f5b683",
  "api": "web_scrape",
  "created_at": "2026-10-26T15:20:00Z",
  "counts": {"total": 1000, "queued": 0, "running": 12, "done": 975, "failed": 13},
  "jobs": [
    {"index": 0, "id": "job_a1b2c3d4e5f60718", "status": "done"},
    {"index": 1, "id": "job_a1b2c3d4e5f60719", "status": "done"}
  ]
}

GET /v1/jobs/{id} returns one job's result once its status is done. Results are kept 48 hours, and the status calls are not billed. An account runs 20 direct requests at once; higher limits go through sales.

What this map-then-scrape run costs: 1 sitemap fetch plus 1,000 product pages is 1,001 billed results, about $0.50 at the plain-HTTP entry price of $0.50 per 1,000, or $0.20 when the results come from a purchase of 10,000 or more. With JavaScript rendering on every product page, the same run costs about $2.00, or $1.00 at the volume price. A crawl job that follows every link instead also pays for the category, filter and pagination pages it passes through.

All three exist. Per request or credit is the common model for general scraper APIs, per successful result bills only the calls the provider counts as a success, and per GB belongs to proxies, unblockers and scraping browsers, where you pay for the bytes that pass through.

They often sit side by side at one provider: per request or credit (each call, or a credit cost per call that rises with rendering and hard targets), per successful result (only the calls the provider counts as a success) or per GB (the bytes that pass through). ProxyField's scraper APIs are designed to bill per successful result: $0.50 per 1,000 plain-HTTP pages and $2.00 per 1,000 rendered pages, or $0.20 and $1.00 from 10,000 results in one purchase. The same comparison in brief, with its FAQ, is on the pricing page.

Credits make a bill hard to forecast. As an illustration with no provider named: if a plain page costs 1 credit and a rendered page 10, then 100,000 rendered pages use 1,000,000 credits, ten times the plain-page plan you might have sized for. A per-result price for the same job is a single figure: 100 × $1.00 = $100 in ProxyField's design (the worked example below).

Before you choose any provider, check four things:

1. Are retries billed, or only the final result?
2. Are 404s and sign-in pages billed?
3. What multiplier do rendering and hard targets carry?
4. Does every response say what it cost, in a header or a field?

ProxyField's scraper APIs are designed to bill on successful results only, priced per 1,000 results, with no credits and no per-target multiplier: the price depends on the API and, on the Web Scraping API, on render. That is the only multiplier: a rendered page costs 4 times a plain one at the entry price ($2.00 vs $0.50 per 1,000) and 5 times from 10,000 results ($1.00 vs $0.20).

Which responses are billed in ProxyField's design:

Every response carries metadata.billed and an X-Billed header. There is no subscription: the APIs are paid from the same crypto balance as the proxies, and the balance never expires.

The lower price applies from 10,000 successful results in one purchase. Details: web scraping API, pricing.

Worked example: 100,000 pages, three ways

Per-GB pricing rewards light pages and punishes heavy ones; per-result pricing costs the same whatever a page weighs.

Break-even: per result or per GB?

A plain-HTTP result at $0.20 per 1,000 costs $0.0002 a page. At $2.79/GB, that buys about 72 KB of proxy traffic, so pages lighter than 72 KB cost less through your own proxies and heavier ones cost less through the API (before your servers and your time). The break-even page weight for each API price:

Measure your own pages first: the HTML of a product page alone is often far lighter than the full page with images and scripts that a browser loads, and only the bytes your scraper fetches count. (1 KB = 1,000 bytes and 1 GB = 1,000,000 KB here.) ProxyField residential proxies cost $2.95/GB for 1 GB, $2.79 from 10 GB, $2.65 from 50 GB, $2.49 from 100 GB, $2.29 from 250 GB, $2.09 from 500 GB and $1.95 from 1,000 GB, with no subscription; the traffic never expires and all purchases are final. The full comparison: scraper API vs proxies.

The reason to pay for a scraper API is the work it takes off your hands. With your own scraper, each of these is your job:

You still decide what to scrape, how often, and what to do with the data. You also stay responsible for scraping lawfully: an API does not change what is allowed. See is web scraping legal.

Using a scraper API takes five steps:

# pip install requests beautifulsoup4
import requests
from bs4 import BeautifulSoup

resp = requests.post(
    "https://api.proxyfield.io/v1/scrape",
    json={"url": "https://www.example-shop.com/product/42", "render": False, "country": "us", "output": "json"},
    headers={"Authorization": "Bearer <api_key>"},
    timeout=130,
)
resp.raise_for_status()
data = resp.json()
print(data["metadata"]["billed"], data["status_code"])
print(BeautifulSoup(data["content"], "html.parser").title.get_text(strip=True))

5. Scale it. Send requests concurrently up to your account's limit, or use batch jobs for long URL lists. Log failures and the cost per page, so you can see which targets need rendering.

Rendered requests take longer, so give them a generous timeout. AI agents can call a scraper API the same way, or through an MCP server: see web scraping MCP servers.

A scraper API sits between two other ways to get web data:

If the site has an official API that covers your data, use it first: web scraping vs API compares the two.

A scraper API pays off for:

• Teams without a scraping engineer: analysts, marketers and founders who can make an HTTP call but do not want to run browsers and proxy pools.
• Many different targets: when you scrape hundreds of sites a little each, writing block handling per site costs more than the API.
• Projects that need data this week, not after a month of building a crawler.
• AI agents and pipelines that need one dependable "fetch this URL" tool.

It pays off less for a team that scrapes a few sites at high volume and can maintain its own code. There, residential proxies billed per GB can cost less per page, as the worked example shows. ProxyField residential proxies rotate the IP on every request or hold a sticky session from 30 s to 36 h, with unlimited concurrent threads: see rotating proxies.

How reliable a scraper API is depends on the target, not on the API alone. The same API can return nearly every page of a simple site and fail often on a heavily protected one. Three things tell you what to expect:

• Test on your own targets. Run a few hundred real URLs through the API and count the successes and the cost per successful page.
• Check what counts as success. A 200 response can still be a block page or an empty template, so validate the fields you need, not only the status code.
• Prefer billing on successful results, and read how the provider defines a successful result.

Keep a fallback for the targets that matter most: a second API, or your own scraper with proxies for the pages the API cannot fetch.