How to Avoid Getting Blocked While Scraping

Websites block scrapers to protect their servers, their data and their users' accounts, and bot-protection services score every request on a handful of signals. Knowing the signal tells you the fix.

Most blocks are made by an anti-bot service sitting in front of the site, not by the site's own code. The service scores each request and decides to allow it, slow it, challenge it with a CAPTCHA or block it. That is why a scraper can work for days and then fail everywhere at once: the scoring model changed, not your code.

Blocks rarely come from one signal alone. A datacenter IP with perfect headers may pass a news site and fail a retailer; a residential IP with a script's TLS handshake can still meet a challenge page. Fix the signals that are easiest to change first: rate, headers and IP type.

Work through the list in order: the first five items are the easiest to change, and the later ones matter on sites with a dedicated anti-bot service.

1. Read the rules. Check robots.txt and the terms, and collect only public data. robots.txt rules "are not a form of access authorization" (RFC 9309), but they tell you which paths the site wants left alone: skip them. More: how to check if a website allows scraping.
2. Look for an API or embedded JSON first. Many pages load their data from a JSON endpoint you can call directly, with fewer requests and fewer blocks.
3. Don't download what hasn't changed. Cache pages and send conditional requests: If-None-Match with the stored ETag, or If-Modified-Since with the stored date. An unchanged page comes back as 304 Not Modified with no body (MDN: conditional requests), so a re-crawl costs a fraction of the requests and traffic.
4. Slow down. Start with one request every few seconds per site, add random jitter, and raise the rate only while responses stay clean.
5. Rotate IPs. Spread requests over many residential IPs; use a sticky session only when several requests must come from one visitor.
6. Send headers that fit the site. On sites that welcome crawlers, a descriptive User-Agent with a contact URL is the lowest-friction choice, for example PriceWatchBot/1.2 (+https://example.com/bot; [email protected]): the operator can see who you are and reach you instead of blocking you. On sites that serve only browsers, send a current browser User-Agent with the matching Accept, Accept-Language and Accept-Encoding, in the browser's order, and rotate whole header sets, not single fields. User agent list.
7. Keep sessions consistent. One IP, one user agent and one cookie jar per session; a session that changes IP or browser in the middle looks like a takeover.
8. Match the location. Use IPs in the country, and if needed the city, of the content you need; set the matching language and time zone in a browser.
9. Use a browser only where needed. Plain HTTP for server-rendered pages; a headless browser for pages built by JavaScript: scrape dynamic web pages.
10. Act like a visitor. Follow links rather than jumping to deep URLs in a perfect sequence, load the pages a visitor would load, and vary the timing.
11. Fix your browser's fingerprint. Remove automation flags and keep fingerprints consistent with the user agent: fingerprint spoofing.
12. Avoid traps. Never follow hidden links, fill hidden form fields or wander into endless URL spaces: honeypots and crawler traps.
13. Back off by the numbers. On a 429 or 503, wait the Retry-After time if the response sends one: it holds either a number of seconds (Retry-After: 120) or an HTTP date (Retry-After: Wed, 28 Oct 2026 07:28:00 GMT), so parse both (MDN: Retry-After); otherwise wait 2, 4, 8, then 16 seconds with random jitter, and give up on the URL after 5 tries. After a 403 or a CAPTCHA, wait longer (20, 40, then 80 seconds, at most 3 tries) and continue on a new session. Never retry in a tight loop. Details: 429 errors, CAPTCHAs.
14. Monitor. Log status codes and response sizes per site; a rise in 403s or in tiny pages is the early warning.

The IP is the first thing a site checks and the easiest one to change.

• Residential IPs belong to home internet providers and look like ordinary visitors; they are the default for protected sites. Mobile IPs are shared by many phone users behind carrier NAT, so blocking one can lock out many real visitors; that makes them the next step for the strictest sites. Datacenter IPs cost less per GB ($0.29 down to $0.22 at ProxyField) but sit in ranges that sites can block by network. The types compared: types of proxies.
• Rotation gives each request a new IP, so no single address crosses a rate limit. ProxyField can give you a new IP on every request.
• Sticky sessions keep one IP for a set time, for logins, carts and paginated searches; ProxyField's last from 30 seconds to 36 hours.
• Location matters: many sites serve or block by country. ProxyField targets country, region and city, in 195+ countries and all 50 US states.
• IP ranges: sites and anti-bot services often block a whole subnet, or every range that belongs to a hosting provider, which is why datacenter proxies on one subnet can fail together. Residential IPs spread across many home internet providers and many ranges, so a range block hits far fewer of them.
• Google: the pool is clean for Google, checked and segmented. If you want Google results as data rather than pages, the Google SERP API returns the results page as JSON, at $1.20 down to $0.80 per 1,000 successful results.

A proxy changes the IP and nothing else. If headers, fingerprints or behavior give you away, a new IP only moves the block to the next request; the rest of this checklist matters as much.

A headless browser renders JavaScript like a real browser, but out of the box it leaves automation traces such as navigator.webdriver set to true. Stealth plugins patch the best-known traces.

A minimal Playwright example in Node.js, with a ProxyField residential IP in New York:

// npm install playwright playwright-extra puppeteer-extra-plugin-stealth
const { chromium } = require('playwright-extra');
const stealth = require('puppeteer-extra-plugin-stealth')();

chromium.use(stealth);

(async () => {
  const browser = await chromium.launch({
    proxy: {
      server: 'http://gate.proxyfield.io:7000',
      username: '<login>-country-us-city-new_york',
      password: '<password>',
    },
  });
  const page = await browser.newPage();
  await page.goto('https://api.ipify.org');
  console.log(await page.textContent('body')); // the proxy's IP
  await browser.close();
})();

Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.

Does playwright-stealth or puppeteer-extra-plugin-stealth still work?

Not reliably. The Node.js puppeteer-extra-plugin-stealth has had no release since 2.11.2 on March 1, 2023, and stealth plugins pass simple checks and public bot-test pages but, on their own, not a dedicated anti-bot service such as Cloudflare or DataDome. The Python playwright-stealth is maintained (2.0.3, April 4, 2026), and its own README warns it handles only "the simplest of bot detection methods". The Puppeteer plugin patches named traces of headless Chrome, with evasion modules such as navigator.webdriver, navigator.plugins, chrome.runtime, iframe.contentWindow, webgl.vendor, user-agent-override and window.outerdimensions. Its README says it passes "all public bot tests" (it links bot.sannysoft.com) and names the fpscanner, Intoli and areyouheadless tests. The same README calls the work "a rather interesting cat and mouse game", says "it's probably impossible to prevent all ways to detect headless chromium", and sets a narrower goal: to make detection so difficult "that it becomes cost-prohibitive or triggers too many false-positives". On reCAPTCHA v3 it says stealth "seems to help with maintaining a normal reCAPTCHA v3 score", calls the evidence "anecdotal observations", and warns that the score "is calculated individually per site". The Python playwright-stealth README is blunter: "Don't expect this to bypass anything but the simplest of bot detection methods."

Passing a public test page is not passing Cloudflare or DataDome, which also score the TLS handshake, the IP's reputation, the request rate and behavior. Cloudflare's bot detection engines, for example, run heuristics on every request, a machine-learning model that gives each request a bot score from 1 to 99, and optional JavaScript detections that, in Cloudflare's words, identify "headless browsers and other malicious fingerprints".

The same setup in Python with playwright-stealth 2.x (its Stealth().use_async wrapper applies the evasions to every page) and a ProxyField IP in New York:

# pip install playwright playwright-stealth && playwright install chromium
import asyncio
from playwright.async_api import async_playwright
from playwright_stealth import Stealth

PROXY = {"server": "http://gate.proxyfield.io:7000",
         "username": "<login>-country-us-city-new_york", "password": "<password>"}

async def main():
    async with Stealth().use_async(async_playwright()) as p:
        browser = await p.chromium.launch(proxy=PROXY)
        page = await browser.new_page()
        print(await page.evaluate("navigator.webdriver"))  # no longer True once stealth is applied
        await page.goto("https://api.ipify.org")
        print(await page.text_content("body"))              # the proxy's IP
        await browser.close()

asyncio.run(main())

Maintenance, from the npm and PyPI registries on October 27, 2026 (the release dates are the registries', not the READMEs'):

The Node.js plugins and selenium-stealth have had no release since 2023 or earlier, while browsers and detection keep changing; playwright-stealth for Python had a release in 2026. Use any of them together with residential IPs, real headers and calm pacing, not instead of them. More: can Playwright, Puppeteer and Selenium be detected, and the setup guides for Playwright, Puppeteer and Selenium.

A CAPTCHA or "checking your browser" page means the site's protection has already scored your request as suspicious. The better answer is to avoid the score that triggers it, through the checklist above, rather than to solve challenges at scale. The guides:

• How to avoid and solve CAPTCHAs when scraping and CAPTCHA solver services compared.
• CAPTCHA proxies: how the IP type changes how often challenges appear.
• Why Cloudflare thinks you are a bot, scraping sites behind Cloudflare and Cloudflare error codes.
• DataDome, PerimeterX and hCaptcha challenges.

How often a site challenges you depends on your whole request, not only the IP.

Follow only links a person could see, normalize and deduplicate URLs, cap depth and pages per site, skip what robots.txt disallows, and never follow links that change something. That covers the two kinds of trap a crawler meets.

Honeypots are links or form fields that real visitors never see, placed so that only a bot follows them. Typical forms: links hidden with CSS (display: none, visibility: hidden, zero size, off-screen), links in the same color as the background, and hidden form inputs. A scraper that follows every href on a page walks into them and gets its IP flagged.

Crawler traps are URL spaces with no end, often not built as traps at all: calendars with a "next month" link forever, search results that link to more searches, filter combinations that multiply, and session IDs or timestamps in the query string that make every visit a "new" URL.

How a trap shows up in your logs, so you can stop it while the crawl runs:

• one path keeps growing: thousands of queued URLs under /calendar/ or /search/ while the rest of the site has hundreds;
• URLs that differ only in a query parameter (?page=, ?sort=, ?date=, a session ID) return pages of the same size and content;
• the URL gets longer with each hop, with repeated path segments such as /a/b/a/b/a/;
• the queue grows faster than pages are fetched, long after the sitemap's URLs are done;
• one IP, or every IP at once, starts getting 403s right after the crawler fetched a URL no visible link points to: a honeypot.

Stop a path when it passes your cap, for example 500 pages per path prefix, and review it by hand.

This Playwright (Python) check keeps only the links a visitor can see, then applies the URL rules before a link enters the queue:

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

VISIBLE_LINKS = """els => els.filter(a => {
  const s = getComputedStyle(a), r = a.getBoundingClientRect();
  return s.display !== 'none' && s.visibility !== 'hidden' && s.opacity !== '0'
         && r.width > 0 && r.height > 0 && r.right > 0 && r.bottom > 0;
}).map(a => a.href)"""
VISIBLE_INPUTS = """els => els.filter(i => {
  const s = getComputedStyle(i), r = i.getBoundingClientRect();
  return i.type !== 'hidden' && s.display !== 'none' && s.visibility !== 'hidden'
         && s.opacity !== '0' && r.width > 0 && r.height > 0;
}).map(i => i.name)"""
DROP_PARAMS = {"sessionid", "sid", "phpsessid", "utm_source", "utm_medium", "utm_campaign"}
ACTION_SEGMENTS = {"logout", "signout", "delete", "remove", "cart", "add-to-cart", "checkout", "buy"}
MAX_DEPTH, MAX_PARAMS = 5, 4   # example defaults: tune them per site

def normalize(url):
    p = urlsplit(url)
    query = sorted((k, v) for k, v in parse_qsl(p.query) if k.lower() not in DROP_PARAMS)
    return urlunsplit((p.scheme, p.netloc.lower(), p.path or "/", urlencode(query), ""))

def next_links(page, depth, seen, allowed_by_robots):
    if depth >= MAX_DEPTH:
        return []
    out = []
    for href in page.eval_on_selector_all("a[href]", VISIBLE_LINKS):
        url = normalize(href)
        segments = set(urlsplit(url).path.lower().strip("/").split("/"))
        if (url in seen or len(parse_qsl(urlsplit(url).query)) > MAX_PARAMS
                or segments & ACTION_SEGMENTS          # whole path segments only: /cart, not /carter
                or not allowed_by_robots(url)):
            continue
        seen.add(url)
        out.append(url)
    return out

def fields_to_fill(page, form_selector="form"):
    # the names of the inputs a visitor can see; leave every other field empty, as a person would
    return page.eval_on_selector_all(f"{form_selector} input[name], {form_selector} textarea[name]",
                                     VISIBLE_INPUTS)

Pass allowed_by_robots from Python's urllib.robotparser (RobotFileParser.can_fetch), and keep a page cap per host in the loop that calls it. When the crawler submits a search form, fill only the fields fields_to_fill returns; a hidden field that comes back filled is how many honeypot forms spot a bot. A parser such as BeautifulSoup sees only inline styles and the hidden attribute, not the site's CSS files, so use a browser where visibility matters.

When a site starts blocking you, step back before you push harder.

Change one thing at a time and measure the success rate after each step. If a site still refuses after all of this, stop: it is telling you it does not want automated visitors, and its terms may say the same.

Avoiding blocks is about not overloading sites and not looking like an attacker; it is not a license to take data you are not allowed to take. Scrape public pages, respect robots.txt and rate limits, never use logins you are not entitled to, and read what data is legal and safe to scrape. ProxyField's acceptable use policy forbids attacks, fraud, spam and access to data behind logins you are not entitled to.

• IPs: why is my IP address blocked, can websites detect proxies, CAPTCHA proxies.
• Headers: HTTP headers for web scraping, what is a user agent, user agents for scraping.
• Fingerprints: what is a browser fingerprint, browser fingerprinting in scraping, TLS and TCP fingerprinting, fingerprint spoofing.
• CAPTCHAs: avoid and solve CAPTCHAs, CAPTCHA solver services, CAPTCHA solver extensions, CAPTCHA providers compared, DataDome, PerimeterX and hCaptcha.
• Cloudflare: why Cloudflare thinks you are a bot, scraping behind Cloudflare, human verification extensions.

Checked on October 27, 2026:

• puppeteer-extra-plugin-stealth README and playwright-extra README; versions and dates from the npm registry.
• playwright-stealth (Python) README; version and date from PyPI.
• selenium-stealth README; version and date from PyPI.
• RFC 9309: Robots Exclusion Protocol (September 2022).
• MDN: HTTP conditional requests.
• MDN: Retry-After.
• Cloudflare: bot detection engines.