Ecommerce Scraper: Collect Product Pages
The best e-commerce scraper API for many stores depends on one choice: fields parsed for you on a list of named stores, or one price for every store. ProxyField's Web Scraping API is the one-price option, designed to fetch any public store's product, category and search pages for $0.50 per 1,000 over plain HTTP, or $0.20 from 10,000 in one purchase ($2.00 and $1.00 with JS rendering); Amazon's 8 marketplaces are designed to come back as parsed JSON on the Amazon scraper API at $0.60 per 1,000 ($0.30 from 10,000). Both are in development: contact sales, and see named providers' prices on the web scraping API price comparison. To run your own scraper today, start with ProxyField residential proxies and the Easy Trial, 1 GB for $2.95.
Through the sales team on our 24/7 live chat
| Fetch | Entry | Volume, from |
|---|---|---|
| HTTP | $0.50 | $0.20 |
| JS rendering | $2.00 | $1.00 |
Billed on successful results only.
The e-commerce scraper API for product data from many stores
For product data from many different stores, ProxyField's Web Scraping API is designed to take any public store URL at one price per successful result, $0.50 per 1,000 or $0.20 from 10,000. That is a fetcher, not a list of store parsers: outside Amazon you read the fields yourself, from the places the next table shows. It is in development, through Contact sales. What comes back depends on the store:
| Store | What the API is designed to return | Price per 1,000 successful results | A whole job |
|---|---|---|---|
| Amazon: amazon.com, .co.uk, .de, .fr, .it, .es, .ca, .co.jp | Parsed JSON: title, brand, price, price_strikethrough, currency, stock, rating, reviews_count, the buybox seller, variations | $0.60, or $0.30 from 10,000 | 100,000 product pages: $30 |
| Shopify, WooCommerce and brand stores, marketplaces with the price in the HTML | The page as HTML, Markdown, text or a JSON envelope, with the schema.org/Product block (name, SKU or GTIN, price, currency, availability) | $0.50, or $0.20 from 10,000 | 1,000,000 catalog pages: $200 |
| Stores that load price, stock or variants with scripts | The rendered page: render: true with wait_for | $2.00, or $1.00 from 10,000 | 1,000,000 catalog pages: $1,000 |
Read moreShow less
Outside Amazon there are no prebuilt store parsers: you get the page and parse it. In return, one price covers every store, and a store's redesign changes your parser, not your plan.
Where the product fields sit, by platform
Most stores publish their product fields in one of these documented places, so one fetcher and a small parser cover many stores:
| Platform | Where the fields are | What you get | ProxyField tier |
|---|---|---|---|
| Shopify stores | GET /products/<handle>.js, the product's JSON (Shopify Ajax Product API) | title, vendor, price, compare_at_price, available, and every variant with its sku, barcode, price and available; money in the store's presentment currency | plain HTTP: $0.50 per 1,000, $0.20 from 10,000 |
| WooCommerce stores | GET /wp-json/wc/store/v1/products, the public Store API (WooCommerce Store API: products) | a prices object (price, regular_price, sale_price, currency_code, currency_minor_unit) and is_in_stock | plain HTTP: $0.50 per 1,000, $0.20 from 10,000 |
| Any store with structured data | the schema.org/Product JSON-LD block in the page (schema.org/Product) | name, SKU or GTIN, and an offers price, currency and availability | plain HTTP, with the parser below |
| Stores that load price or stock with scripts | the rendered page | the fields once they appear: render: true with wait_for | JS rendering: $2.00 per 1,000, $1.00 from 10,000 |
| Amazon (8 marketplaces) | the Amazon scraper API, parsed | title, brand, price, price_strikethrough, currency, stock, rating, reviews_count, buybox, variations | $0.60 per 1,000, $0.30 from 10,000 |
Cost anchors: a 1,000,000-page catalog costs $200 over plain HTTP or $1,000 rendered, bought in one purchase; 100,000 parsed Amazon products cost $30 at the $0.30 volume price. Public pages only, follow each store's terms: a public JSON route is not a permission.
Which kind of API fits you. If you need prebuilt, parsed fields for many named stores today, choose a vendor with prebuilt store parsers. If you need any store URL at one fixed price per page and will read the fields yourself (often from the schema.org/Product block, test below), this API is built for that. Named providers' prices per 1,000: web scraping API price comparison.
A one-minute test on your own stores. Fetch one product page and count its structured-data blocks:
curl -s -A "price-research/1.0 (+mailto:[email protected])" \
https://store.example.com/products/blue-kettle | grep -c 'application/ld+json'
# 1 or more: the page carries JSON-LD; check that one block has "@type": "Product"
curl -s -A "price-research/1.0 (+mailto:[email protected])" \
https://store.example.com/products/blue-kettle | grep -o '"@type": *"Product"' | head -1| What you see | What it means | The tier you need |
|---|---|---|
A count of 1 or more and "@type": "Product" | name, price, currency and availability are in the HTML | plain HTTP: $0.50 per 1,000, $0.20 from 10,000 |
| A count of 0, but the price is in the page source | the fields are in the HTML without structured data | plain HTTP, with CSS selectors for the price and title |
| No price in the page source | the store loads it with scripts | JS rendering: $2.00 per 1,000, $1.00 from 10,000 |
These are the other points to compare between APIs, with ProxyField's answer for each:
| What to check | Why it matters | ProxyField |
|---|---|---|
| What counts as a billable result | You pay for these, so blocks and errors should not count | Successful results only. A store's 404 or 410, or a sign-in page served with a 200 status, is billed. Blocked and timed-out requests and our internal retries are not |
| JS rendering | Some stores load price, stock or variants with scripts | render: true, with wait_for and browser actions (click, scroll, infinite scroll) |
| Country of the request | Stores show prices, currency and stock by country | country: any of the proxy pool's 195+ countries |
| The same IP across pages | Cart, store selection and pagination expect one visitor | session_id: the same exit IP for up to 10 minutes |
| Concurrency | How fast a big catalog finishes | 20 requests at once per account, higher limits through sales |
| Batches and delivery | Large jobs without holding connections open | Up to 1,000 requests per batch, a signed callback for each finished job, results kept 48 hours. No SDK, MCP server or no-code export in v1 |
| Payment | Plans and expiring credits cost money you don't use | The same crypto balance as proxies. It never expires, and there is no subscription, plan or monthly fee |
Test your own stores before you commit. Send sample product URLs from each store through Contact sales and check that variants, pagination, country prices and stock come back the way you need them. The APIs are paid from the shared balance, topped up from $2.95; there is no separate test volume. Full parameter reference: API documentation.
How the Web Scraping API fetches a product page
You send the URL of a public product or listing page and choose plain HTTP or JS rendering. The API is designed to return the page and bill only successful results.
- Request: a
POSTtohttps://api.proxyfield.io/v1/scrapewith your API key in anAuthorization: Bearer <api_key>header. - JSON body:
url,render(falsefor plain HTTP,truefor JS rendering),countryandoutput(html,markdown,textorjson). - Script-loaded stores: where price, stock or variants load with scripts,
wait_fortakes a CSS selector to wait for (with rendering).
Read moreShow less
| Choice | Price per 1,000 successful results | Use it for |
|---|---|---|
| Plain HTTP | $0.50, or $0.20 from 10,000 in one purchase | Stores whose price and stock are in the HTML or the structured data |
| JS rendering | $2.00, or $1.00 from 10,000 in one purchase | Stores that load price, stock or variants with scripts after the page opens |
Test a sample of each store's pages over plain HTTP first, and render only the stores where the price is missing from the HTML.
# The request the Web Scraping API takes (access through Contact sales)
curl https://api.proxyfield.io/v1/scrape \
-H "Authorization: Bearer <api_key>" \
-H "Content-Type: application/json" \
-d '{"url": "https://store.example.com/products/blue-kettle", "render": false, "country": "us", "output": "json"}'Example response (shortened; content is the page's HTML, with the schema.org/Product block inside):
{
"metadata": {"id": "req_9345a95e51dacbbf", "api": "web_scrape", "status": "success", "billed": true},
"status_code": 200,
"final_url": "https://store.example.com/products/blue-kettle",
"content_type": "text/html; charset=utf-8",
"content": "<!DOCTYPE html><html lang=\"en\">…<script type=\"application/ld+json\">…</html>"
}Read the product fields from any store
The API returns the page, not parsed store fields, so the parsing is yours. This function reads name, SKU, GTIN, price, currency and availability from the schema.org/Product block. It works on the API's content field and on a page you fetch yourself:
import json
import requests
from bs4 import BeautifulSoup
def products(html):
"""Yield the schema.org/Product items embedded in a store page."""
soup = BeautifulSoup(html, "html.parser")
for tag in soup.select('script[type="application/ld+json"]'):
try:
data = json.loads(tag.string or "")
except ValueError:
continue
items = data if isinstance(data, list) else data.get("@graph", [data])
for item in items:
kind = item.get("@type") if isinstance(item, dict) else None
if kind == "Product" or (isinstance(kind, list) and "Product" in kind):
offer = item.get("offers") or {}
offer = offer[0] if isinstance(offer, list) else offer
yield {"name": item.get("name"), "sku": item.get("sku"),
"gtin": item.get("gtin13") or item.get("gtin"),
"price": offer.get("price") or offer.get("lowPrice"),
"currency": offer.get("priceCurrency"),
"availability": offer.get("availability")}
resp = requests.post("https://api.proxyfield.io/v1/scrape",
headers={"Authorization": "Bearer <api_key>"},
json={"url": "https://store.example.com/products/blue-kettle",
"render": False, "country": "us", "output": "json"},
timeout=130)
for product in products(resp.json()["content"]):
print(product)A store with no schema.org/Product block needs CSS selectors for its price and title instead. Amazon pages need no parser: the Amazon scraper API is designed to return title, brand, price, price_strikethrough, currency, stock, rating, reviews_count, the buybox seller and variations as JSON.
Pricing: $0.50 per 1,000 product pages, $0.20 from 10,000
Product pages cost the Web Scraping API's price. Plain HTTP is $0.50 per 1,000 successful results, or $0.20 from 10,000 results in one purchase. JS rendering is $2.00 per 1,000, or $1.00 from 10,000. Billing rules:
- Billed: a page the store answered with a 2xx status. A store's 404 or 410, and a sign-in page the site serves with a 200 status, also count as billed results.
- Not billed: error responses, blocked and timed-out requests, and our internal retries.
- Every response says whether it was billed.
A mixed job: 100,000 pages with 80,000 over plain HTTP and 20,000 rendered cost $16 + $20 = $36 at the volume prices, or $40 + $40 = $80 at the entry prices. Below 10,000 results in a purchase, the entry prices apply: 5,000 pages over plain HTTP cost $2.50.
Payment: from the same crypto balance as proxies, which never expires. There is no subscription or monthly fee. All prices: pricing.
Through the sales team on our 24/7 live chat
Sales contact: the sales team on our 24/7 live chat.
| Workload (30-day month, bought in one purchase) | Results | Plain HTTP at $0.20 | JS rendering at $1.00 |
|---|---|---|---|
| 50,000 product pages, once a month | 50,000 | $10 | $50 |
| 2,000 SKUs on 5 competitor stores, daily | 300,000 | $60 | $300 |
| 1 million catalog pages a month | 1,000,000 | $200 | $1,000 |
How do I scrape product data from an online store without getting blocked?
Stores block scrapers mostly for volume and speed, so the answer is a slow, honest crawl that stops when a store says stop: one request at a time, every 5 seconds or slower, which is about 83 minutes per 1,000 pages. In order:
-
Check permission first
Read the store's terms and its
robots.txt(the Robots Exclusion Protocol, RFC 9309). Many stores' terms restrict robots and data mining. Amazon's Conditions of Use & Sale (last updated 28 November 2025) exclude from its licence "any use of data mining, robots, or similar data gathering and extraction tools", and its Agents section requires an agent to identify itself by "including the following in the request’s user agent string" (Agent/[agent name]), and forbids "mimicking the speed or pattern of human keystrokes, page navigation, or other interactions" and "completing or circumventing CAPTCHAs or other measures intended to distinguish computers from humans." Steps 3 and 5 follow the same logic. On ProxyField the rule is public pages only, follow the site's terms. How to read both: how to check if scraping is allowed. A ten-second first check:bash curl -s https://store.example.com/robots.txt | grep -iE 'sitemap|disallow'Sitemap:lines are the URLs you can crawl instead of search pages;Disallow:lines are the paths to skip. -
Use the official route where there is one
A product feed, an affiliate or partner API, or the sitemaps listed in
robots.txtare cheaper than crawling search pages. -
Identify your client
Send a User-Agent that names your project and a contact address, plus normal
AcceptandAccept-Languageheaders. -
Crawl slowly and once
Make one request at a time per store, several seconds apart: at one request every 5 seconds, 1,000 pages take about 83 minutes, and a full day is at most 17,280 pages from one store. Cache pages, de-duplicate URLs and skip pages that have not changed. Store limits are not only per IP: Cloudflare's rate-limiting rules can count requests by IP, cookie or header, and a cookie rule "will group together requests from different IP addresses" in one session (Cloudflare: rate limiting best practices), so rotating IPs does not reset a limit.
-
Respect the signals
Status 429 means "the client has sent too many requests in a given amount of time" (MDN: 429 Too Many Requests); wait as long as its
Retry-Afterheader says, in seconds or as a date (RFC 9110, section 10.2.3), and resume slower. The header is optional, so with noRetry-After, stop and wait minutes, not seconds. On 403 or a CAPTCHA page, stop the run and review the job rather than retrying at once. -
Collect product facts, not people
Leave out reviewer names, accounts, checkout and anything behind a login.
Read moreShow less
| What gets a scraper blocked | What to do | The ProxyField setting |
|---|---|---|
| Many requests in a short time | Slow down and spread the job over hours | Rotation (<login>-country-us) spreads the load over many IPs; it does not replace a slow rate, and it is not a way past a block |
| Server IP ranges on large stores (our assessment) | Use IPs of home connections | 412M+ residential IPs in 195+ countries |
| A new IP on every page of a cart, store selection or pagination | Keep one IP for the whole flow | A sticky session: <login>-country-us-session-cart01-ttl-600 (30 s to 36 h) |
| Prices that load with scripts, read before they appear | Render and wait for the element | render: true with wait_for ($2.00 per 1,000) |
| Retrying at once after 429 | Wait for Retry-After, then resume slower | The code below stops and waits |
| The wrong country | The store shows other prices and stock | -country-de in the username, or "country": "de" on the API |
More on rate limits and fingerprints: anti-bot systems and web scraping without getting blocked in Python.
The proxy setup for German e-commerce catalogs such as Otto and MediaMarkt
For German stores, ProxyField's setup is rotating residential IPs in Germany for catalog pages, one sticky German IP for a cart or a store selection, and an Accept-Language: de-DE header. Start from the sitemaps without any proxy, and add a German IP for the pages whose price, stock or delivery terms depend on the visitor's country.
Read moreShow less
| Germany on ProxyField | Figure |
|---|---|
| Residential IPs | 20.2M+ |
| Mobile IPs | 190K+, on the Telekom, Vodafone and O2 networks |
| Cities | 980+ |
| Residential price | $2.95/GB for 1 GB down to $1.95/GB from 1,000 GB, the same as every country |
| Catalog pages | <login>-country-de: a new German IP on every request |
| A cart or a chosen store | <login>-country-de-session-de01-ttl-600: one German IP for 10 minutes |
Which proxy type for a German catalog:
| Type | Price | When it fits a German catalog |
|---|---|---|
| Residential (above) | $2.95/GB for 1 GB down to $1.95/GB from 1,000 GB | stores that treat server IP ranges differently from home connections (our assessment) |
| Rotating datacenter | $0.29/GB down to $0.22/GB, in the same countries as residential | stores that serve their catalog to any IP: test a sample first |
| Dedicated datacenter IPv4 | $1.39 per IP for 30 days | a small catalog from a few fixed addresses: 5 IPs for 30 days cost $6.95, in the countries the order form lists; test a sample first |
| Dedicated ISP | from $2.29 per IP for 30 days | a fixed address of a home internet provider, in the countries the order form lists |
The first test decides the type. Fetch 200 catalog pages of each store through rotating German datacenter IPs ($0.29/GB; 200 pages at 150 KB are about 30 MB, a small part of 1 GB). If all 200 come back as the product page, keep that store on datacenter. If any return 403, 429 or a challenge page, move that store to rotating German residential IPs and run the same 200 again.
Datacenter details: datacenter proxies.
-
Read the terms first, plus each store's robots.txt (otto.de/robots.txt, mediamarkt.de/robots.txt). What we found on October 28, 2026: - Otto (Nutzungsbedingungen, "Stand: Juli 2025"): using information from Otto's services for commercial purposes "ist ohne ausdrückliche Zustimmung von OTTO nicht gestattet" (needs Otto's express consent), and copying or reusing content, which the terms define to include data, information and images, needs Otto's express approval. For commercial price work, ask Otto first. - MediaMarkt (Nutzungsbedingungen, "Stand: September 2025"): the terms of use say nothing about automated access or reuse of prices; their copyright rules cover what users and marketplace sellers publish. Its robots.txt and its other legal pages still apply. Collect public pages only, follow the site's terms, and ask for a feed or permission before commercial use.
-
Start from the sitemaps each store lists in its robots.txt, not from search pages. When we read them, Otto's listed 19 sitemaps: a category sitemap index, 9 product sitemap indexes by department (
/products-sitemap/…) and 9 for product variations (/variations-sitemap/…). MediaMarkt's listed 3: a main sitemap index (/sitemaps/sitemap-index.xml) and two for deals and seasonal pages. -
Set the proxy lines in the
host:port:username:passwordformat:
# catalog pages: a new German IP on every request
gate.proxyfield.io:7000:<login>-country-de:<password>
# a cart or a chosen store: one German IP for 10 minutes
gate.proxyfield.io:7000:<login>-country-de-session-de01-ttl-600:<password>-
Keep the rate low, cache pages, and stop on 403, 429 or a CAPTCHA page.
-
Budget it (example inputs; measure your own page weight): - Proxies: 40,000 product pages at 150 KB each is 6 GB, or $17.70 at $2.95/GB. - API: the same pages with
"country": "de"cost $8 at $0.20 per 1,000, bought in one purchase.
All German figures: proxies in Germany.
Which stores the API fetches
The Web Scraping API fetches any public URL you send. It has no list of supported stores, and each store's terms decide what you may collect: public pages only, follow the site's terms. That covers Shopify and WooCommerce shops, brand stores and marketplaces alike. Amazon has its own API, the Amazon scraper API, at $0.60 per 1,000 successful results, from $0.30. Walmart and eBay pages are fetched as pages by the Web Scraping API: see the Walmart scraper and the eBay scraper.
Location for your own scraper: ProxyField residential proxies offer country, region (state) and city targeting, with 412M+ residential IPs in 195+ countries.
Location for the API: country sets the country the request comes from (the exit IP), as a two-letter code such as de. There is no city setting, and it accepts the proxy pool's 195+ countries.
Some site categories are restricted to protect the network, including financial and government services.
Scraper API or your own scraper on proxies?
Use the API when you would rather pay per page than maintain the fetching; use proxies when you already run a scraper and want to control every request.
| Item | Web Scraping API | Your scraper on ProxyField proxies |
|---|---|---|
| You pay for | Successful results: $0.50 per 1,000, or $0.20 from 10,000 in one purchase | Traffic: $2.95/GB for 1 GB, from $1.95/GB |
| Billing | Successful results only | Every request uses traffic |
| JS rendering | $2.00 per 1,000 ($1.00 from 10,000), per request | Your own headless browser |
| You maintain | The parser | The fetcher, the browser, the retries and the parser |
| Good for | Many stores, mixed page types, fixed cost per page | Large single-store jobs, custom logic, logged-in seller tools you own |
How many GB a product-page job uses depends on the page weight; the worked example is on web scraping proxies. Proxies or a scraper API in general: Scraper API vs Proxies.
What teams use ecommerce data for
- Price monitoring daily competitor prices and promotions, covered on the price scraping page.
- Assortment and stock tracking which products a competitor adds, drops or runs out of.
- Catalog enrichment titles, specifications and GTINs for your own listings, within the store's terms.
- Review analysis ratings and review counts by product, to spot quality problems early.
- Brand protection where your products appear and at what price on marketplaces and resellers.
Not allowed on ProxyField: spam, fraud including fake accounts or engagement, and accessing data behind logins you are not entitled to.
Scrape an online store with Python and ProxyField proxies
For a few thousand pages, a Python script through rotating residential IPs is enough. It fetches each product page, reads it with the products() function above, waits between requests and stops on 403 or 429:
import time
import requests
proxy = "http://<login>-country-us:<password>@gate.proxyfield.io:7000"
headers = {"User-Agent": "price-research/1.0 (+mailto:[email protected])",
"Accept-Language": "en-US"}
urls = ["https://store.example.com/products/blue-kettle",
"https://store.example.com/products/red-kettle"] # public product pages
for url in urls:
resp = requests.get(url, proxies={"http": proxy, "https": proxy},
headers=headers, timeout=30)
if resp.status_code in (403, 429):
wait = resp.headers.get("Retry-After", "")
print(f"{resp.status_code} on {url}: stopping; Retry-After={wait or 'none'}")
break # review before the next run
resp.raise_for_status()
for product in products(resp.text): # the parser above
print(product)
time.sleep(5) # one request at a time per storeProxyField residential proxies cost $2.95/GB for 1 GB and fall with volume. The tier rate applies to the whole purchase:
| Volume in one purchase | Residential, per GB |
|---|---|
| 1–9 GB (Easy Trial: 1 GB) | $2.95 |
| 100–249 GB | $2.49 |
| 1,000+ GB | $1.95 |
There is no subscription, the traffic never expires, and all purchases are final; every tier is on pricing. You get 412M+ residential IPs in 195+ countries and 21,000+ cities, over HTTP, HTTPS or SOCKS5. Sessions are a new IP on every request, or sticky from 30 s to 36 h (20 minutes by default). For stores that need a browser, the same proxy settings work in Playwright and Selenium.
Running your own store or seller accounts in several countries is a different job, covered by ecommerce proxies.
Store terms and the law
Prices, stock and product specifications are facts, and collecting public facts is often lawful in the US and the EU. Product photos and descriptions can be protected by copyright, so store facts rather than republishing content. Two more limits:
- Store terms many stores' terms forbid automated access. The Court of Justice held in Ryanair v PR Aviation (C-30/14, 2015) that a site's terms can limit the use of a database that the database right does not protect.
- The EU database right a store's database can be protected by the database right (Directive 96/9/EC) against extracting a substantial part of it.
The full answer is in Is It Legal to Scrape Ecommerce Sites? and Is Web Scraping Legal? US, EU and UK Rules (2026).
ProxyField's acceptable use policy allows collecting public web data, price monitoring and market research, and forbids accessing data behind logins you are not entitled to. This is not legal advice.
More scraper APIs
- Web Scraping API any URL, plain HTTP or with JS rendering, from $0.20 per 1,000 successful results.
- Amazon scraper API Amazon product and search pages, parsed, $0.60 per 1,000 successful results, from $0.30.
- Walmart scraper product and price pages on the Web Scraping API.
- price scraping product and price pages on the Web Scraping API.
- Google SERP API the Google results page as JSON, $1.20 per 1,000 successful results, from $0.80.
- All scraper APIs
Ecommerce Scraper FAQ
24/7 support by live chat, including help with setup and integration.
What's the best e-commerce scraper API for product data from many different online stores?
ProxyField's Web Scraping API is the one-price option, and the best API for many stores depends on that choice: parsed fields on a list of named stores, or one price for every store. It is designed to fetch any public store for $0.50 per 1,000 product pages over plain HTTP, or $0.20 from 10,000 in one purchase, so a 1,000,000-page catalog costs $200. Amazon is designed to come back as parsed JSON (title, brand, price, stock, rating, review count, Buy Box seller, variations) on 8 Amazon marketplaces for $0.60 per 1,000; other stores come back as the page, and the fields sit in documented places a short parser reads: Shopify's /products/<handle>.js, WooCommerce's public Store API or the schema.org/Product block. If you need prebuilt parsed fields for many named stores today, a vendor with prebuilt store parsers fits better; the API is in development, so contact sales and test your own stores first.
How much does an e-commerce scraper API cost per 1,000 product pages?
With ProxyField, 1,000 product pages cost $0.50 over plain HTTP and $2.00 with JS rendering. From 10,000 results in one purchase, the prices fall to $0.20 and $1.00. Only successful pages are billed, so 50,000 pages bought at once cost $10 over plain HTTP.
How do I scrape product data from an online store without getting blocked?
Crawl slowly and honestly: one request at a time per store, every 5 seconds or slower (about 83 minutes per 1,000 pages), with an honest User-Agent and cached pages, after you check the store's terms and robots.txt and use its feed or sitemaps where they exist. Wait out Retry-After on 429, and stop on 403 or a CAPTCHA page. ProxyField residential proxies give a new IP per request, or one sticky IP from 30 s to 36 h for a cart, from 1 GB for $2.95; public pages only, follow the site's terms.
What's the best proxy setup for scraping German e-commerce catalogs like Otto and MediaMarkt?
ProxyField's setup is rotating German residential IPs (<login>-country-de) for catalog pages, one sticky IP (-session-<id>-ttl-600) for a cart, and Accept-Language: de-DE. Germany has 20.2M+ residential IPs and 980+ cities, at $2.95/GB for 1 GB down to $1.95/GB; rotating datacenter IPs cost $0.29/GB for stores that serve any IP. Start from the sitemaps in each store's robots.txt, test 200 pages on datacenter IPs first and move a store to residential if any return 403 or 429, and read the terms: Otto's require its express consent for commercial use. At 150 KB a page, 40,000 pages are 6 GB, or $17.70.
Does an e-commerce scraper API return parsed product fields or raw HTML?
It depends on the API and the store. ProxyField's Amazon scraper API is designed to return parsed fields (title, price, stock, rating, Buy Box seller); the Web Scraping API is designed to return any store's page as HTML, Markdown, text or a JSON envelope, for $0.50 per 1,000 successful results. To see whether a store's fields are in its HTML, fetch one product page and count application/ld+json blocks: if one has "@type": "Product", a short parser reads them.
What is an ecommerce scraper API?
It is a service that takes the URL of a store page, fetches it for you and returns the page or its data, so you do not run your own proxies or browsers. ProxyField's Web Scraping API is designed to do this for any public store, billed per successful result.
What is e-commerce scraping?
E-commerce scraping is collecting public data from online stores automatically: product names, prices, stock, ratings and listings. Retailers and brands use it to track competitors' prices and assortment.
What data can you scrape from ecommerce websites?
Public product data: title, brand, SKU or GTIN, price and sale price, currency, stock status, variants, images, ratings and review counts, plus category listings and marketplace offers. Prices behind a customer login and B2B price lists are out of scope.
How do I scrape data from an ecommerce website?
Collect the product URLs from the store's sitemaps or category pages, fetch each product page and read the fields from the HTML or from its schema.org/Product block. Do it at a low rate through rotating IPs, or send the URLs to a scraping API and parse what comes back.
How do I scrape Shopify stores?
Shopify documents a JSON view of each product in its Ajax API: GET /products/{product-handle}.js returns "the JSON of a product using the product handle" (Shopify Ajax API: Product). Fetch product pages or that JSON over plain HTTP, at $0.50 per 1,000 successful results on ProxyField's Web Scraping API, and respect each store's terms.
Show 11 more questionsShow fewer questions
How do I handle JavaScript when scraping Shopify?
Run the one-minute test above on one product page: if the price is in the HTML or the product JSON, plain HTTP is enough. Render only when a theme loads prices or stock with scripts; on ProxyField that costs $2.00 per 1,000 successful results instead of $0.50.
Which ecommerce sites can you scrape?
ProxyField's Web Scraping API fetches any public URL you send, so it has no list of supported stores; Amazon has its own parsed API at $0.60 per 1,000 successful results. What you may collect from a store is set by that store's terms: public pages only, follow the site's terms. Some site categories are restricted to protect the network, including financial and government services.
Should I use an ecommerce scraper API or proxies?
Use an API if you want a fixed price per page and no fetching to maintain: $0.50 per 1,000 successful results with ProxyField, or $0.20 from 10,000 in one purchase. Use proxies, from $1.95/GB, if you already run a scraper and want full control of each request.
What features does an ecommerce scraper API include?
ProxyField's Web Scraping API is designed to fetch any public URL over plain HTTP or with JS rendering and bill successful results only: $0.50 per 1,000 over plain HTTP and $2.00 with JS rendering. It is designed to return the page as HTML, Markdown or plain text, or as a JSON envelope with the status code, final URL, headers, cookies and content. It also takes a country for the request, 20 requests at once per account and batches of up to 1,000 requests with a signed callback.
How fast is an ecommerce scraper API?
ProxyField publishes no response-time figure for the Web Scraping API. Speed depends mostly on the store and on rendering: plain HTTP pages return faster than pages that need a browser, and 20 requests can run at once per account.
How many proxies do I need for ecommerce scraping?
With rotating residential proxies you do not count IPs: one gateway gives a new IP on every request, and you pay per GB, from $1.95/GB with ProxyField. What you plan is the request rate per store and the traffic your pages use.
Can I scrape localized e-commerce data?
Yes, by fetching each store from the country whose prices you want. ProxyField residential proxies offer country, region (state) and city targeting, including all 50 US states, with 412M+ residential IPs in 195+ countries; on the API, country sets the country of the request.
Why do marketplaces block my scraper?
Mostly for volume: fast, repetitive requests from one IP, server IP ranges and clients that ignore 429 responses. Slow down, cache, wait out Retry-After, and use residential IPs at a low rate per store.
Is real-time scraping better than buying an e-commerce dataset?
Scraping gives you today's prices for exactly the products you choose; a dataset is quicker to start but is a snapshot someone else picked. ProxyField does not sell datasets; it sells proxies, and the Web Scraping API is in development, for collecting your own.
Do you need coding skills to use an e-commerce scraper?
For a scraping API, a little: you send page URLs and process the pages that come back, usually from a short script or an automation tool. Browser extensions need no code but stop at a few pages.
Can I scrape Zara or Shein product pages?
Their public product pages can be fetched like any other store page on the Web Scraping API, at $0.50 per 1,000 successful results, or $2.00 where a page needs JS rendering. ProxyField has no dedicated Zara or Shein API, and both stores' terms apply.
Related pages
Start collecting public pages with the Web Scraping API
Billed on successful results only. Contact sales for the Web Scraping API and the volume you need, or run your own scraper on residential proxies. 24/7 live chat helps with setup and integration.