How to Avoid and Solve CAPTCHAs When Scraping

A CAPTCHA appears when the site's risk check scores your request as likely automated. The score comes from five groups of signals, and a scraper usually trips more than one:

The CAPTCHA is the last step of that check, not the first. That is why the fixes below work on the signals and leave the CAPTCHA itself for last.

Work through the list in order; each step says when it applies.

1. Check the rules and set a budget before you scale. Read the site's /robots.txt and its terms, and skip the paths it disallows. The robots.txt standard, RFC 9309 (September 2022), says its rules "are not a form of access authorization", so they are the site's request to crawlers, not a lock; follow them anyway. Set a request budget per site before the job starts: how to check if a website allows scraping.

2. Slow down. If CAPTCHAs start after a burst, the rate is the trigger. Start at one request every few seconds per site, with random jitter, lower concurrency per site, and raise the rate only while no challenges appear.

3. Cache and skip what you have. Every page you don't fetch again is a request that cannot be challenged: store pages, deduplicate URLs, and re-fetch only what changed.

4. Spread requests over clean IPs. If one IP gets challenged while a fresh one passes, the IP's reputation or request count is the trigger. Rotate residential proxies or mobile proxies so no single IP carries much traffic; ProxyField gives a new IP on every request, or a sticky IP for 30 s to 36 h.

5. Keep one IP per session. If you are challenged in the middle of a session, an IP change may be the trigger. Keep a sticky IP and one set of cookies for the length of a session, and rotate between sessions, not inside one.

6. Make the client consistent. If a plain HTTP client is challenged at once while a browser passes, the fingerprint is the trigger. Send headers that match the client, use an HTTP client that matches a browser's TLS fingerprint, or use a real browser (TLS and TCP fingerprinting, HTTP headers for web scraping).

7. Run JavaScript and keep cookies. If the page sets cookies with scripts before showing content, a client without JavaScript looks new on every request. Use Playwright, Puppeteer or Selenium for those pages and reuse the browser context.

8. Use the data source the site offers. An official API, a data feed or a sitemap often gives the same data without any challenge.

9. Retry later, not harder. A challenge or a 429 Too Many Requests is information. MDN's reference says a 429 "may" carry a Retry-After header that tells you how long to wait (MDN: 429). The rule:

   • wait the Retry-After time when the response sends one;
   • otherwise wait 20 s, then 40 s, then 80 s, with random jitter;
   • try at most 3 times per URL, then log it and move on;
   • continue on a new session, never by repeating the same request at once.

   More on rate limits: 429 Too Many Requests.

Proxies are one step of nine. What they can and cannot change is covered in CAPTCHA proxies.

When a CAPTCHA appears anyway on a public page, you have three ways to handle it:

• A solver service through its API. You send the CAPTCHA's details (the type, the site key and the page URL); the service returns an answer or a token; your script submits it the way the page would. Human-powered and machine-based services exist; CAPTCHA solver services compared explains how they differ.
• A person in the loop. For low volumes, pause the script, show the challenge to an operator, and continue once it is solved. This suits monitoring jobs that hit a CAPTCHA a few times a day.
• Skip the page. If one page in a thousand is challenged, log it and retry it later on a new session.

Tokens are short-lived and tied to the page. hCaptcha's documentation, for example, says a token can be redeemed once and expires after 120 seconds by default (hCaptcha docs). Solve just before you submit, on the same page and session, and through the same IP where the service supports it.

Three causes, in the order to check them:

The test: run the same job once in headed Chrome (headless=False) through the same IP. If the CAPTCHAs stop, headless signals are the trigger; if they continue, look at the IP and the rate. Also check whether your script logs in on every run: a fresh login is a common trigger, and reusing a saved session avoids it (below).

Browser automation does not solve CAPTCHAs by itself; it lowers how often you see them and lets you handle the rest. The pattern is the same in all three tools: detect the challenge, stop that session, back off, and continue on a new one.

Playwright. This script loads a page through the ProxyField gateway, checks for a CAPTCHA frame, and starts a new browser context (new cookies, new IP on rotation) when it finds one, with the back-off rule from step 9. Replace <login> and <password>.

import random
import time
from playwright.sync_api import sync_playwright

PROXY = {"server": "http://gate.proxyfield.io:7000",
         "username": "<login>-country-us-city-new_york", "password": "<password>"}
CAPTCHA_FRAMES = "iframe[src*='recaptcha'], iframe[src*='hcaptcha'], iframe[src*='challenges.cloudflare.com']"

def fetch(url, attempts=3):
    with sync_playwright() as p:
        browser = p.chromium.launch(proxy=PROXY)
        try:
            for attempt in range(attempts):
                context = browser.new_context()      # new cookies for each try
                page = context.new_page()
                page.goto(url, wait_until="domcontentloaded")
                page.wait_for_timeout(3000)           # CAPTCHA frames load after the page
                if page.locator(CAPTCHA_FRAMES).count() == 0:
                    html = page.content()
                    context.close()
                    return html
                context.close()                       # challenged: drop this session
                time.sleep(20 * 2 ** attempt * random.uniform(1, 1.5))  # 20 s, 40 s, 80 s + jitter
            return None                               # 3 tries: log the URL and move on
        finally:
            browser.close()

print(fetch("https://example.com/") is not None)

Reuse a login instead of logging in again. If your job uses an account you are entitled to, log in once and save the browser state; later runs load it, so the site does not see a new login, and the login CAPTCHA, on every run. Playwright's documentation describes this (Playwright: authentication):

# once, after logging in:
context.storage_state(path="playwright/.auth/state.json")

# every later run:
context = browser.new_context(storage_state="playwright/.auth/state.json")

Playwright warns that the state file "may contain sensitive cookies and headers that could be used to impersonate you or your test account", so keep playwright/.auth out of git. Load the saved state through the same sticky session, such as <login>-country-us-city-new_york-session-he9f2qga-ttl-3600, because the same cookies arriving from a new IP can look like a session takeover. If the login itself shows a CAPTCHA every time, treat it as a stop (see below).

Selenium. Selenium does not handle CAPTCHAs itself; detect the frame, quit the driver and start a new one. Set the proxy in your driver as in the Selenium guide.

import random
import time
from selenium.webdriver.common.by import By

CAPTCHA_FRAMES = "iframe[src*='recaptcha'], iframe[src*='hcaptcha'], iframe[src*='challenges.cloudflare.com']"

def fetch(url, make_driver, attempts=3):
    for attempt in range(attempts):
        driver = make_driver()                    # a new driver: new cookies, new IP on rotation
        try:
            driver.get(url)
            time.sleep(3)                         # CAPTCHA frames load after the page
            if not driver.find_elements(By.CSS_SELECTOR, CAPTCHA_FRAMES):
                return driver.page_source
        finally:
            driver.quit()                         # challenged or done: close this session
        time.sleep(20 * 2 ** attempt * random.uniform(1, 1.5))
    return None

With per-request rotation the new context or driver gets a new IP; with a sticky session, change the session id instead: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default) after the location, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600. Proxy setup for each tool: Playwright, Puppeteer, Selenium. Why automated browsers get flagged in the first place: browser fingerprinting in web scraping.

The common CAPTCHAs differ in what they show and in what a scraper can do about them.

Turnstile runs "non-interactive JavaScript challenges" and aims to avoid showing a puzzle at all; it can be embedded on any site, not only sites behind Cloudflare (Cloudflare: Turnstile). The hCaptcha row follows hCaptcha's documentation; reCAPTCHA v3's score model is described in Google's reCAPTCHA developer guide. The vendors themselves are compared in CAPTCHA providers compared.

A headless browser runs JavaScript and keeps cookies, so it passes checks that stop plain HTTP clients, but it is also easier to detect than a person's browser: default headless builds and automation flags differ from a normal Chrome in ways detection scripts test. Expect fewer CAPTCHAs than with a plain client, not none. Image and puzzle challenges still need a solver or a person.

Google shows its "unusual traffic" page and a CAPTCHA when a network sends too many automated searches. For rank tracking and SERP data, ProxyField's Google SERP API returns the results page as JSON, from $1.20 down to $0.80 per 1,000 successful results, billed on successful results only; contact sales. If you run your own scraper on ProxyField proxies, the rate and client rules above still apply, and Google may still show its checks.

If you see CAPTCHAs while browsing, the site's check scored your connection or browser as risky:

• Turn off your VPN or proxy and reload, or switch to another network; shared VPN IPs are often flagged.
• Allow cookies and JavaScript for the site, and pause ad-blocking or privacy extensions on it.
• Update your browser.
• Restart your router if your home IP may have been flagged; many home connections get a new IP that way.

If the CAPTCHA appears on every visit, only the site can change its rule. Extensions that promise to solve CAPTCHAs for you carry risks of their own: see CAPTCHA solver extensions for Chrome.

A CAPTCHA in front of a login, a checkout or an account sign-up protects something the site does not want automated, and getting around it on purpose can break the site's terms and raise legal risk. Scrape public pages, follow the site's terms, and ask the owner for access or an API when the data sits behind a barrier: see is web scraping legal. ProxyField's acceptable use policy forbids fake accounts and fake engagement.

The IP side of the checklist. ProxyField residential proxies: 412M+ IPs in 195+ countries, a new IP per request or sticky sessions up to 36 h, from $1.95/GB with no subscription. Start with the Easy Trial (1 GB); traffic never expires, and all purchases are final.