Why Is My Web Scraper Slow? How to Make It Faster

Start with the numbers for your own job: total time is about pages × seconds per page ÷ requests in flight.

Measure before you change anything. Time each stage for 20 pages: waiting for the response, downloading, parsing, saving. One of them is almost always most of the total.

import time, requests
s = requests.Session()
t0 = time.perf_counter(); r = s.get("https://example.com/page/1", timeout=30); t1 = time.perf_counter()
soup_t0 = time.perf_counter(); n = r.text.count("<tr"); soup_t1 = time.perf_counter()
print(f"network {t1 - t0:.2f}s, parse {soup_t1 - soup_t0:.3f}s, {len(r.content) / 1024:.0f} KB")

If network time is most of the total, concurrency is your fix. If parse or save time is, more requests in flight will not help.

Concurrency means keeping several requests in flight at once, so the scraper works while it waits. Throughput is about concurrency divided by latency: with 1 request in flight and 0.5 s per response you get 2 pages per second; with 20 in flight, about 40, until the site, your bandwidth or your CPU becomes the limit.

Concurrency is not the same as parallelism. Concurrency interleaves many waiting tasks on one core (async I/O, threads that mostly wait); parallelism runs work on several cores at the same time (processes). Scrapers are I/O-bound, so concurrency gives most of the speed; parallel processes help only when parsing is the bottleneck.

Async HTTP in Python (httpx)

# pip install httpx beautifulsoup4
import asyncio, httpx
from bs4 import BeautifulSoup

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
URLS = [f"https://www.scrapethissite.com/pages/forms/?page_num={n}" for n in range(1, 11)]

async def fetch(client, sem, url):
    async with sem:                                   # at most N requests in flight
        for attempt in range(3):
            try:
                r = await client.get(url)
                if r.status_code in (429, 503):
                    await asyncio.sleep(float(r.headers.get("Retry-After", 2 ** attempt)))
                    continue
                r.raise_for_status()
                return len(BeautifulSoup(r.text, "html.parser").select("tr.team"))
            except httpx.TransportError:
                await asyncio.sleep(2 ** attempt)
        return None

async def main():
    sem = asyncio.Semaphore(5)                        # tune per site
    async with httpx.AsyncClient(proxy=PROXY, timeout=30,
                                 headers={"User-Agent": "Mozilla/5.0"}) as client:
        results = await asyncio.gather(*(fetch(client, sem, u) for u in URLS))
    print(results)

asyncio.run(main())

The Semaphore is the throttle: raise it while responses stay fast and clean, lower it as soon as 429s or timeouts rise. The proxy= argument needs httpx 0.26 or later. The Python async setup, aiohttp included, is in Python proxy setup.

Threads for code you don't want to rewrite

If your scraper uses Requests, a thread pool gives you concurrency without async:

from concurrent.futures import ThreadPoolExecutor
import requests, threading

local = threading.local()
def session():
    if not hasattr(local, "s"):
        local.s = requests.Session()                 # one Session per thread
    return local.s

def fetch(url):
    return session().get(url, timeout=30).status_code

with ThreadPoolExecutor(max_workers=10) as pool:
    print(list(pool.map(fetch, URLS)))

Scrapy: the settings that decide its speed

Scrapy is concurrent by design, but a new project is polite by default: the startproject template sets DOWNLOAD_DELAY = 1, one request per second per domain, which alone makes 3,000 pages on one site take 50 minutes. These are the settings to change in settings.py (Scrapy: optimizing performance, settings reference):

# settings.py: a faster starting point for a site that tolerates it
CONCURRENT_REQUESTS = 32             # total requests in flight
CONCURRENT_REQUESTS_PER_DOMAIN = 16  # per site; the real limit on a single-site crawl
DOWNLOAD_DELAY = 0                   # drop the template's 1 s; let AutoThrottle pace you
AUTOTHROTTLE_ENABLED = True          # adjusts the delay to the site's observed latency
AUTOTHROTTLE_TARGET_CONCURRENCY = 8  # average requests in flight per site
COOKIES_ENABLED = False              # unless the site needs a session
RETRY_TIMES = 1                      # fewer retries on hopeless URLs
DOWNLOAD_TIMEOUT = 30                # do not wait minutes on a dead request
LOG_LEVEL = "INFO"

For a broad crawl over many sites, set CONCURRENT_REQUESTS close to CONCURRENT_REQUESTS_PER_DOMAIN × the number of sites: 8 per site over 10 sites is 80 in flight.

Read the stats line Scrapy logs every minute to see whether a change helped:

INFO: Crawled 1200 pages (at 60 pages/min), scraped 1150 items (at 58 items/min)

• Pages per minute stays flat after you raise concurrency: the limit is elsewhere. Check the delay, the site's 429 or 503 responses, or your callbacks.
• The retry/count stat climbs and 429s appear: you are above what the site allows; lower CONCURRENT_REQUESTS_PER_DOMAIN or spread requests over more IPs.
• Requests in flight stay at your CONCURRENT_REQUESTS value (len(engine.downloader.active) in Scrapy's telnet console est()): the downloader is the limit, so more concurrency or more IPs can help.
• The scheduler queue (len(engine.scheduler.mqs)) keeps growing: the spider finds links faster than it downloads them, which is normal for a crawl; it is not a slowdown by itself.

Proxy setup for Scrapy is in Scrapy proxy setup; the spider itself is in the Scrapy tutorial. Crawlee is concurrent by design in the same way (maxConcurrency).

Robots.txt. A site's robots.txt may set Crawl-delay (seconds between requests). Scrapy applies a site's robots.txt rules when ROBOTSTXT_OBEY = True, the template's setting; to respect a Crawl-delay, set DOWNLOAD_DELAY to that value for the site, and keep your concurrency for the other sites. More on pacing a crawl: what is a web crawler.

The biggest single gain after concurrency is often skipping the browser altogether. Many pages load their data from a JSON request after the HTML arrives; call that request directly and you get the data without scripts, images or layout.

1. Open the page in Chrome, press F12 and open the Network panel (Chrome DevTools: Network).
2. Select the Fetch/XHR filter and reload the page.
3. Scroll, page or click to load more data; watch which request returns it.
4. Click the request and open Response: you are looking for JSON with the fields you scrape, such as {"items": [{"title": "...", "price": 19.99}], "next_page": 2}.
5. Right-click the request, Copy as cURL, and replay it from your code with the same query parameters and the headers it needs.
6. Change the page or limit parameter to step through results, often at 50-100 items per call instead of 20 per HTML page.

The full method, including JSON embedded in the HTML (__NEXT_DATA__ and similar), is in how to scrape hidden APIs.

Opening a new TCP and TLS connection for every request can cost more than the request itself. A requests.Session or one httpx.Client keeps connections alive between requests to the same host; httpx.Client(http2=True) (with pip install httpx[http2]) multiplexes many requests over one connection where the server supports HTTP/2. Through a rotating proxy, each new exit IP needs its own connection to the site, so reuse helps most with sticky sessions and with the connection to the gateway itself.

A headless browser is the slowest way to get a page: it downloads and runs every script, then lays out the page. Before you render, check whether the data is in the HTML or in a JSON request (section 2, and how to scrape dynamic web pages). When you must render:

• block images, media and fonts with request interception (in Playwright, page.route("**/*", ...) and abort by request.resource_type);
• wait for the element you need (wait_for_selector) instead of networkidle or a fixed sleep;
• reuse one browser and open several contexts or pages in it, rather than launching a browser per URL;
• close pages you are done with, and restart the browser if its memory keeps growing over a long run.

Selenium follows the same rules: run it headless, set page_load_strategy = "eager" so it stops waiting for every image, reuse one driver and move pages that need no browser to plain HTTP. The full list of Selenium speed-ups is in Selenium web scraping, and the proxy setup in Selenium proxy setup.

• Use the lxml parser (BeautifulSoup(html, "lxml")) or lxml directly; selectolax is faster still for CSS-selector work.
• Parse only the fragment you need: find the container first, then select inside it.
• Batch database writes (executemany, or a bulk insert every few hundred rows), and move writing to its own thread or task so fetching never waits on disk.

The fastest request is the one you skip:

• start from sitemaps or listing APIs instead of crawling every link;
• skip URLs you already have unless they changed: send If-Modified-Since or If-None-Match and accept 304 Not Modified (MDN: conditional requests);
• request bigger pages where the site allows it (per_page=100 instead of 20);
• deduplicate URLs before they reach the queue;
• in development, cache responses (Scrapy's HTTPCACHE_ENABLED = True) so reruns do not refetch.

ProxyField does not limit concurrent threads, so the real limits are the target site, your machine and your network. A practical way to find the number:

1. Start at 2-5 concurrent requests per site.
2. Double it while the error rate stays low and response times stay flat.
3. Stop and step back at the first rise in 429, 503, 403 or timeouts.
4. Spread across sites: 50 requests in flight over 10 sites is gentler on each than 50 on one.

Higher concurrency against one site usually needs more IPs as well, because sites count requests per IP. With per-request rotation, each request leaves from a new IP out of 412M+ residential IPs in 195+ countries; sticky sessions (30 seconds to 36 hours) keep one IP when a flow needs it, and reuse its connection.

A full line: gate.proxyfield.io:7000:<login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600:<password>; every part after the login is optional. Typical response times, according to our service data, are under 1 s for residential and under 1.5 s for mobile.

Proxy providers and scraping APIs often sell or limit "concurrency": the number of requests or connections you may have open at the same time. It is not a requests-per-second limit: 10 concurrent requests that each take 2 s give you 5 requests per second, and the same 10 at 0.5 s give 20. When you compare offers, check both the concurrency limit and what is billed (GB of traffic, requests or successful results). ProxyField residential proxies are billed per GB with unlimited concurrent threads: $2.95/GB for 1 GB down to $1.95/GB from 1,000 GB. The ProxyField Web Scraping API is in development and will bill per 1,000 successful results.

• The basic loop this page speeds up: how to build a web scraper.
• Scaling a crawl across many hosts: how to build a web crawler.
• Shorter runs on a schedule: how to schedule web scrapers.
• Picking the IP type and sizing traffic: proxies for web scraping.

Scaling up concurrency? ProxyField residential proxies: unlimited concurrent threads, a new IP per request, from $1.95/GB with no subscription. Start with the Easy Trial (1 GB for $2.95); traffic never expires, and all purchases are final.