Web Scraping Without Getting Blocked in Python

A Python scraper gets blocked because it is easy to tell apart from a person in a browser, and because it sends more requests from one IP than a person would. Sites and their bot-protection services check a handful of signals, and a plain requests.get() fails most of them:

Each row is a separate reason to block you, so fixing one rarely fixes everything. The order that works: rate first, then headers, then IPs, then the TLS client, and a real browser last. The anti-bot hub explains each signal in general.

Start with a requests.Session: it reuses connections and keeps cookies between requests, as a browser does. Give it the full header set of one real browser, not just a new User-Agent:

# pip install requests
import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,"
              "image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
})
r = session.get("https://books.toscrape.com/", timeout=(5, 30))
print(r.status_code, r.request.headers["User-Agent"][:40])

• Copy headers from your own browser. Open the page in Chrome, then DevTools, Network, the document request, and copy the request headers. Keep the User-Agent and Accept-Language consistent with each other and with the IP's country.
• Rotate whole header sets, not single fields. A Chrome User-Agent with Firefox's Accept header is a contradiction that is easy to score. The user agent list and HTTP headers for web scraping go deeper.
• Always set a timeout. The Requests documentation warns that without one "your program [can] hang indefinitely" (Requests: Timeouts). timeout=(5, 30) gives 5 seconds to connect and 30 to read.

Most blocks come from rate, not from anything clever. One IP that requests a page every 200 ms looks like a bot no matter what headers it sends.

• Start at one request every 2 to 5 seconds per site, chosen at random with random.uniform(2, 5), and raise the rate only while responses stay clean.
• Keep concurrency per site low: 1 or 2 parallel requests to one host is a good start. Run other sites in parallel instead.
• Do not refetch what you already have. Cache pages and skip images, fonts and scripts you do not need.
• Prefer the site's JSON endpoints where the page loads its data from them: one request instead of a full page, and often fewer checks. See how to scrape hidden APIs.

There is no universal safe rate; it depends on the site. If you watch your logs, the site will tell you: a rise in 429s, 403s or tiny pages means you are too fast.

Retry the responses that mean "try later" and stop on the ones that mean "no". urllib3, which Requests uses underneath, has a Retry class that does the waiting for you:

import random, time
import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retry = Retry(
    total=4,
    backoff_factor=1,                         # sleeps 0, 2, 4, 8 s between tries
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],          # never blindly retry POSTs
    respect_retry_after_header=True,          # wait as long as the site asks
)
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry))
session.mount("http://", HTTPAdapter(max_retries=retry))

def fetch(url):
    r = session.get(url, timeout=(5, 30))
    # crude check: a word match also hits ordinary pages that mention "captcha";
    # match the target's real block page (its title or a marker) instead
    if r.status_code == 403 or "captcha" in r.text[:20000].lower():
        raise RuntimeError(f"blocked: {r.status_code} on {url}")  # stop, don't hammer
    r.raise_for_status()
    return r.text

for url in ["https://books.toscrape.com/catalogue/page-1.html",
            "https://books.toscrape.com/catalogue/page-2.html"]:
    html = fetch(url)
    print(url, len(html))
    time.sleep(random.uniform(2, 5))          # jitter between pages

The urllib3 documentation gives the backoff formula: it sleeps backoff_factor * 2 ** (previous retries) seconds with no wait before the first retry, so a factor of 1 gives 0, 2, 4 and 8 seconds; every wait is capped at backoff_max (120 seconds by default), and respect_retry_after_header is on by default (urllib3: Retry). When the retries run out, Requests raises an error that reads "Max retries exceeded with url"; treat it as a signal to slow down, not as a bug.

Two rules keep retries from making things worse:

Rotation spreads your requests over many IPs, so no single address crosses the site's rate limit, and residential IPs belong to home internet providers, so sites refuse them as a group less often than datacenter ranges. With ProxyField, a request through the gateway gets a new IP unless you use a sticky session:

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"

session.proxies = {"http": PROXY, "https": PROXY}
print(session.get("https://api.ipify.org", timeout=(5, 30)).text)  # a new IP each call

• Rotate per request for independent pages: search results, product pages, listings.
• Keep one IP for a sticky session when several requests must look like one visitor: a login, a cart, paginated results. ProxyField sticky sessions last from 30 seconds to 36 hours. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.
• Match the country of the content: the username above targets New York; ProxyField has 412M+ residential IPs in 195+ countries, with targeting by country, region (state) and city.
• Use datacenter IPs where they work. On sites without strict protection they are faster and cost less per GB ($0.29 down to $0.22 at ProxyField); move to residential when they are refused. Compare the types in residential vs datacenter proxies.

The proxy setup itself, including SOCKS5, async clients and aiohttp, is in our Python proxy guide and the Python requests guide.

If a site blocks you even with good headers and a residential IP, it may be reading your TLS handshake. Requests uses Python's OpenSSL handshake, which looks nothing like Chrome's, and bot-protection services fingerprint the handshake before your headers are even read. curl_cffi is "a http client that can impersonate browser tls/ja3/http2 fingerprints", with an API close to Requests:

# pip install curl_cffi
from curl_cffi import requests as creq

r = creq.get(
    "https://books.toscrape.com/",
    impersonate="chrome",                     # Chrome's TLS and HTTP/2 fingerprint
    proxies={"http": PROXY, "https": PROXY},
    timeout=30,
)
print(r.status_code)

Keep the User-Agent consistent with the browser you impersonate: a Chrome handshake under a Firefox user agent is another contradiction. The mechanics of JA3 and JA4 are in TLS and TCP fingerprinting.

Some pages build their content with JavaScript, and some sites send a JavaScript check before the content. There, an HTTP client gets an empty shell or a challenge page, and you need a browser. Playwright runs Chromium, Firefox or WebKit from Python and takes the proxy at launch:

# pip install playwright && playwright install chromium
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={
        "server": "http://gate.proxyfield.io:7000",
        "username": "<login>-country-us-city-new_york",
        "password": "<password>",
    })
    page = browser.new_page(locale="en-US", timezone_id="America/New_York")
    page.goto("https://books.toscrape.com/", wait_until="domcontentloaded")
    print(page.title())
    browser.close()

Set the locale and time zone to match the IP's location, as above; a New York IP with a browser set to another time zone is an easy flag. A browser costs far more CPU, memory and traffic per page than an HTTP request, because it also loads scripts, styles and images, so use it for the pages that need it and plain HTTP for the rest. More: scraping dynamic pages with Python, and what gives automated browsers away in browser fingerprinting and scraping.

Undetected ChromeDriver is a patched Selenium ChromeDriver that removes the best-known automation traces. Its own README is clear about the limit: "THIS PACKAGE DOES NOT, and i repeat DOES NOT hide your IP address, so when running from a datacenter (even smaller ones), chances are large you will not pass" (undetected-chromedriver on GitHub). Its last PyPI release, 3.5.5, dates from February 2024, and the same author's nodriver project calls itself its official successor.

If you use Scrapy, most of the steps above are settings rather than code:

# settings.py
ROBOTSTXT_OBEY = True
CONCURRENT_REQUESTS_PER_DOMAIN = 2
DOWNLOAD_DELAY = 2                  # seconds between requests to one site
AUTOTHROTTLE_ENABLED = True         # adapts the delay to the site's latency
AUTOTHROTTLE_START_DELAY = 2
AUTOTHROTTLE_MAX_DELAY = 60
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
RETRY_TIMES = 3
DEFAULT_REQUEST_HEADERS = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}
USER_AGENT = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
              "(KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36")

AutoThrottle adjusts the delay so that the spider sends about AUTOTHROTTLE_TARGET_CONCURRENCY requests at a time to each site, based on how fast the site answers (Scrapy: AutoThrottle). Scrapy already retries 429 and the common 5xx codes by default (RETRY_HTTP_CODES), and since version 2.19 it varies DOWNLOAD_DELAY at random by 50% either way (DOWNLOAD_DELAY_JITTER, default 0.5; Scrapy settings). Proxy rotation in Scrapy is in the Scrapy proxy guide.

When a site still blocks you after all of this, the remaining signals are usually the browser fingerprint and behavior, which proxies do not change. What proxies can and cannot fix separates the two. If you would rather not run browsers and retries yourself, the ProxyField Web Scraping API fetches any URL, with JavaScript rendering when the page needs it, billed on successful results only; the trade-offs are in scraper API vs proxies.

Start small: the ProxyField Easy Trial is 1 GB of residential traffic for $2.95, with no subscription, on the same pool as larger orders. Traffic never expires, and all purchases are final.

Checked on October 23, 2026:

• Requests: Advanced usage and Quickstart: Timeouts (Requests 2.34.2 on PyPI).
• urllib3: Retry.
• curl_cffi on GitHub (0.16.3 on PyPI).
• undetected-chromedriver (3.5.5 on PyPI, February 2024) and nodriver.
• Scrapy 2.19: AutoThrottle, settings and downloader middleware.