Large-Scale Web Scraping

There is no official threshold, but a scraper becomes "large-scale" when one machine and one loop stop being enough. Typical signs:

• Volume: hundreds of thousands of pages per day, or millions per month.
• Breadth: many sites, each with its own layout, rate limits and failure modes.
• Freshness: prices, stock or rankings that must be re-collected every hour or day, sometimes close to real time.
• Reliability: other systems (dashboards, pricing engines, models) depend on the data arriving complete and on time.

At that point the questions change from "how do I parse this page?" to "how do I keep 50 workers busy, polite and unblocked, and know when the data goes wrong?"

Most large scrapers, whatever the language, have the same six parts.

Two design choices pay off more than any library choice:

Throughput is concurrency divided by the time per request. With 50 requests in flight and 1 s per request, a fleet completes about 50 requests per second, or 4,320,000 per day (50 × 86,400). Retries, pauses and per-site limits lower that, so plan for a fraction of it on any single site.

Traffic is pages times page weight. Residential proxies are billed per GB, so this is your main running cost. As an example with an assumed 100 KB of HTML per page, 1,000,000 pages is 100 GB; at ProxyField's 100-249 GB rate of $2.49/GB, that is $249, and the traffic never expires if you buy ahead. Render the same pages in a browser that pulls an assumed 2 MB each, and the job becomes 2,000 GB, $3,900 at the 1,000+ GB rate of $1.95/GB. Your own page weights will differ: measure 100 pages before you size the job. The web scraping use case page works through more traffic and cost examples.

Retries and status codes in depth are in web scraping best practices; the error codes themselves in 429 Too Many Requests and 403 Forbidden.

Scrapy runs many requests concurrently in one process, which is enough for a surprising amount of work. Beyond one machine, its docs say plainly that it "doesn't provide any built-in facility for running crawls in a distributed (multi-server) manner" and suggest two patterns: run many spiders on several Scrapyd servers, or partition one big URL list so each server crawls its part (Scrapy common practices).

The third common pattern is a shared queue. scrapy-redis (0.9.1, MIT, on PyPI) replaces Scrapy's scheduler and duplicate filter with Redis-backed ones, so any number of spider processes on any number of machines pull from one queue and skip URLs another worker has seen:

# settings.py, with: pip install scrapy-redis   (and a Redis server)
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
SCHEDULER_PERSIST = True          # keep the queue between runs
REDIS_URL = "redis://redis-host:6379/0"
CONCURRENT_REQUESTS_PER_DOMAIN = 8
AUTOTHROTTLE_ENABLED = True

A caution from our check of the package on October 23, 2026: scrapy-redis's last release dates from July 2024, and its RedisSpider base class still defines the old start_requests() hook, which Scrapy replaced with async def start() in 2.13.0 (the old hook was removed in Scrapy 2.16.0, May 2026). RedisSpider also pulls URLs when the spider goes idle, so it may still work, but we did not run it against a Redis server: test it with your Scrapy version, or use only the shared scheduler and duplicate filter above. The scrapy-redis readme itself points to Frontera for features such as URL expiration and advanced prioritization.

In Node.js, Crawlee (3.18.2) offers request queues, autoscaling of concurrency and both HTTP and browser crawlers; outside both frameworks, many teams write small httpx or aiohttp workers that read URLs from a queue. For the crawl logic itself (frontier, traps, deduplication), see how to build a web crawler.

Run a browser only for the page types that need one. Before you scale Playwright or Puppeteer, check the page's Network tab: most JavaScript sites load their data from JSON requests that a plain HTTP worker can call at a fraction of the traffic (how to scrape hidden APIs and embedded JSON). Where a browser is unavoidable:

• reuse browsers and open new contexts instead of launching per URL;
• block images, fonts and media;
• cap pages per browser and restart it to release memory;
• keep one sticky IP per context when a session must survive several pages.

At millions of requests, the proxy layer decides how much of your traffic arrives as data rather than as blocks. What matters:

• Rotation per request for independent pages (listings, product pages, search results), and sticky sessions for flows that must keep one IP (logins, carts, paginated sessions). ProxyField offers both: a new IP on every request, or sticky sessions from 30 s to 36 h.
• No concurrency ceiling at the proxy. Your per-site limits should set the pace, not the provider. ProxyField has unlimited concurrent threads on one gateway, gate.proxyfield.io:7000.
• Location. Prices, stock and rankings differ by country and city. ProxyField has 412M+ residential IPs in 195+ countries, with targeting by country, region (state) and city; in the US, all 50 states.
• Success rate and failover. ProxyField has a 99.6% success rate, according to our service data, and if a node slows down, traffic moves to the next one automatically.

How proxies, scraper APIs and in-house fetching compare is covered in scraper API vs proxies, and how to pace a fleet in how to make web scraping faster.

Owning the whole stack gives control and the lowest cost per page once it runs, but you maintain parsers, queues and browsers. A scraping API moves fetching, rendering and retries to the provider and bills per result. ProxyField's Web Scraping API fetches any URL as plain HTTP ($0.50 down to $0.20 per 1,000 successful results) or with JavaScript rendering ($2.00 down to $1.00 per 1,000), billed on successful results only, through Contact sales. Many teams mix the two: their own Scrapy fleet for easy sites, an API for the hardest pages.

Scale multiplies the effect of every request on the target site. Collect public data, read robots.txt and the terms, keep per-site concurrency modest, and avoid personal data unless you have a lawful basis. Is web scraping legal? covers the main cases and laws.