How to Bypass Cloudflare When Scraping

Cloudflare checks a request in layers, and a scraper can fail at any of them:

The bot score row follows Cloudflare's bot score documentation, which lists heuristics, machine learning and JavaScript detections ("identifies headless browsers and other automation tools") among the engines; the TLS row follows its JA3/JA4 page. How each signal works is in why Cloudflare thinks you are a bot. If you already have an error code, start from Cloudflare 403 Forbidden or Cloudflare Access Denied errors.

No single trick gets past Cloudflare; each method fixes some layers and not others.

1. Use an API or ask for access

If the site offers an API, a data export or a partner feed, use it. Many sites behind Cloudflare also load their data from JSON endpoints you can see in the browser's network tab; calling those directly is lighter than rendering pages, though the same Cloudflare rules can protect them.

2. Drive a real browser, on one sticky IP per session

A real browser runs the challenge scripts, sends a genuine TLS handshake and HTTP/2 settings, and keeps the cf_clearance cookie that records a passed challenge (Cloudflare cookies). Keep one IP and one browser context for the whole session, so the clearance stays valid, and rotate between sessions. Setup: Playwright, Puppeteer, Selenium.

3. Patch the automation traces, and expect upkeep

Headless browsers and WebDriver leave traces that Cloudflare's JavaScript detections look for. Stealth plugins and patched drivers hide some of them, but detection changes faster than the patches, so a setup that works this month can fail the next. Headed mode on a desktop-like machine leaves fewer traces than default headless mode, at a higher cost per page.

4. Match the TLS fingerprint for plain HTTP requests

For pages that show no JavaScript challenge, an HTTP client that impersonates a browser's TLS and HTTP/2 fingerprint avoids the cost of a browser. curl_cffi does this in Python (lexiforest/curl_cffi); see TLS and TCP fingerprinting for what the fingerprint contains.

5. Know what cloudscraper and FlareSolverr can do today

• cloudscraper is a Python module, built on Requests, that solves Cloudflare's JavaScript challenges with a JavaScript interpreter. Its README claims support for all challenge types, but its Turnstile path "requires external configuration (proxies/API keys)" and its example uses a paid CAPTCHA solver (VeNoMouS/cloudscraper, MIT).
• FlareSolverr is a proxy server you run yourself: for each request it starts Chrome through Selenium and undetected-chromedriver, waits for the challenge to clear, and returns the HTML and cookies. Its README says "at this time none of the captcha solvers work", and each request starts a browser, so memory use is high (FlareSolverr, MIT).

Both can help on sites with light protection. Neither is a durable answer, and both depend on how quickly their maintainers follow Cloudflare's changes.

6. Change the IP type

If one IP passes where another fails, or you see blocks on data center IPs only, the rule is about IP reputation or the network. Move to residential proxies, mobile proxies or static residential (ISP) proxies, and spread requests so no single IP hits a rate limit. A new IP does not change your client, so methods 2 to 4 still apply.

7. Consider a scraping API

Some scraping APIs run the browser and the IPs for you, so you send a URL and get the page back. None is a guaranteed way past Cloudflare: test one on your own target before you rely on it, and the same rules on access and terms apply.

ProxyField's Web Scraping API fetches any URL as plain HTTP or with JavaScript rendering, billed on successful results only; contact sales.

Plain Python requests fails on most Cloudflare-protected pages: its TLS handshake identifies it as Python whatever User-Agent you send, and it runs no JavaScript. Use curl_cffi where there is no JavaScript challenge, and a browser where there is. This script tries the light client first and reports when a browser is needed. Replace <login> and <password>.

from curl_cffi import requests

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"

def fetch(url):
    r = requests.get(url, impersonate="chrome",
                     proxies={"http": PROXY, "https": PROXY}, timeout=30)
    if r.headers.get("cf-mitigated") == "challenge":
        return None, "challenge page: load this URL in a real browser"
    if r.status_code in (403, 429):
        return None, f"blocked ({r.status_code}): slow down, check the IP and the client"
    return r.text, "ok"

html, status = fetch("https://example.com/")
print(status)

Cloudflare marks challenge responses with the cf-mitigated: challenge header (Detect a Challenge Page response), so the check above is reliable. For a session that must keep one IP, use a sticky session (30 s to 36 h): -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default) after the location, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600. More on the gateway in Python: Python requests with proxies.

• Scrapy. Scrapy's default downloader has its own TLS fingerprint and runs no JavaScript. For Cloudflare pages, render them through a browser with the scrapy-playwright download handler, or send them through a scraping API, and keep Scrapy for the parsing and the crawl logic.
• Puppeteer. Launch Chrome with --proxy-server=http://gate.proxyfield.io:7000, authenticate with page.authenticate(), keep one browser context per session, and wait for the challenge to clear before you read the page. See Puppeteer with proxies.
• Selenium. Plain ChromeDriver exposes WebDriver traces that challenges look for; expect more challenges than with Playwright or Puppeteer in headed mode. See Selenium with proxies.

Turnstile is Cloudflare's CAPTCHA alternative: a widget that runs "non-interactive JavaScript challenges" such as proof-of-work and browser checks, with managed, non-interactive and invisible modes, and it can sit on any site, not only sites behind Cloudflare (Cloudflare: Turnstile). It returns a token the site checks on its server. That leaves two ways through: a real, consistent browser that passes the checks, or a solver service that returns a valid token, which your script then submits. HTTP-only tools cannot run the widget. Solver options and their limits are in CAPTCHA solver services compared.

Cloudflare updates its detections continuously, and the tools that get past them follow with a delay. When a setup that worked starts failing, check in this order: a new browser or library version that changed your fingerprint, a stealth plugin that no longer hides what it used to, a rotation setting that now changes the IP inside a session, a higher request rate, and a change on the site itself, such as a new WAF rule. Keep a small test that loads one protected page per run, so you notice the day it breaks.

If every IP and client you try is blocked, the site does not want automated access, and getting past a barrier the owner set up on purpose can raise legal risk. Read the terms, check robots.txt, and ask the owner for access: is web scraping legal covers the rules.

Residential IPs for the IP layer. ProxyField residential proxies: 412M+ IPs in 195+ countries, per-request rotation or sticky sessions up to 36 h, a 99.6% success rate according to our service data. From $1.95/GB, no subscription; start with the Easy Trial (1 GB). Traffic never expires, and all purchases are final.