403 Forbidden When Scraping Through a Proxy

A 403 is a refusal, not a failure. RFC 9110, section 15.5.4 says the server "understood the request but refuses to fulfill it", and that a server which wants to hide why may answer 404 instead. For scrapers, the refusal usually comes from one of three places:

The last row is rare but worth ruling out. ProxyField restricts some site categories to protect the network, including financial and government services, but its gateway has no 403 or other response of its own for a restricted target. A 403 through ProxyField comes from the site and is passed through.

Your browser and your scraper send very different requests, even from the same IP:

• Headers. A browser sends a full, consistent set: User-Agent, Accept, Accept-Language, Accept-Encoding, Sec-Fetch-* and, in Chrome, client hints. An HTTP library sends a few defaults, and a default User-Agent such as python-requests/2.x is easy to refuse.
• Cookies and JavaScript. Many sites set a cookie through JavaScript on the first visit and expect it on every later request. A scraper that doesn't run JavaScript never gets it.
• TLS and HTTP/2 fingerprint. The way a client opens the encrypted connection identifies its library, whatever User-Agent it claims.
• Rate and pattern. A browser loads one page every few seconds; a scraper loads hundreds a minute, in order, with no assets.

If the page loads in a normal browser over the same proxy IP, the IP is fine and your client is the problem. If it fails in the browser too, the IP or the network it belongs to is the problem.

Change one thing at a time and re-test, simplest first.

1. Send complete, consistent browser headers

Copy the headers a real browser sends to that site (the browser's developer tools show them) and send the same set, in a consistent combination: a Chrome User-Agent with Chrome's Accept values, not a mix. Web scraping headers lists what a real browser sends and what to copy. Add the Referer the site would see when a user clicks through.

2. Keep cookies across requests

Use a session object (requests.Session(), a Scrapy cookie jar, one browser context) so the cookies the site sets on the first page go back on the next ones.

3. Use a real browser for pages that need JavaScript

If the HTML you get is a near-empty shell or a challenge page, load it with Playwright, Puppeteer or Selenium and keep one browser context per session. Setup with proxies: Playwright, Puppeteer, Selenium.

4. Slow down

A 403 that starts after a burst, or after a run of 429s, means a rate rule escalated to a block. Lower concurrency per domain, add random delays and back off on errors; the 429 Too Many Requests doc has a retry loop.

5. Change the IP type, and use sticky sessions for flows

If a browser on your current IP is refused too, the site distrusts the IP or its network. Datacenter ranges are the usual target; move to residential proxies, mobile proxies or static residential (ISP) proxies. Use a new IP per request for independent pages, and one sticky IP (30 s to 36 h) for logins, carts and multi-page flows, so cookies and IP stay together. Choose the country the site serves: country and city targeting are part of the username, as in <login>-country-us-city-new_york. Region and session syntax: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default) after the location, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600.

6. Use an API where the site offers one

An official API or data export is the most stable route. Otherwise, ProxyField's Web Scraping API fetches any URL as plain HTTP or with JavaScript rendering, billed on successful results only; contact sales for access.

7. If the block is deliberate, stop

If every client and IP you try gets a 403, the site has decided not to serve automated traffic. Read its terms and robots.txt, and ask for access. Getting past a barrier set up on purpose can raise legal risk: see is web scraping legal.

This Python snippet sends the same URL twice through the ProxyField gateway, once with library defaults and once with browser-like headers in a session, and prints both status codes and the page title. Replace <login>, <password> and the URL.

import re

import requests

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
PROXIES = {"http": PROXY, "https": PROXY}
URL = "https://example.com/"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate",
}


def title(html):
    m = re.search(r"<title>(.*?)</title>", html, re.I | re.S)
    return m.group(1).strip() if m else None


plain = requests.get(URL, proxies=PROXIES, timeout=30)
print("defaults:", plain.status_code, title(plain.text))

with requests.Session() as s:
    s.headers.update(HEADERS)
    s.proxies.update(PROXIES)
    browserlike = s.get(URL, timeout=30)
    print("browser-like:", browserlike.status_code, title(browserlike.text))

If only the second request passes, headers were the cause. If both fail, try a real browser on the same proxy IP (fix 3), then another IP type (fix 5).

Scrapy's HttpErrorMiddleware drops non-2xx responses before your callback sees them, so a 403 shows up only as a log line. To read the block page, let 403 through for that spider (Scrapy docs: HttpErrorMiddleware):

import scrapy


class ProductSpider(scrapy.Spider):
    name = "products"
    handle_httpstatus_list = [403]
    custom_settings = {
        "USER_AGENT": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                      "(KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
        "DEFAULT_REQUEST_HEADERS": {
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
            "Accept-Language": "en-US,en;q=0.9",
        },
        "AUTOTHROTTLE_ENABLED": True,
    }

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com/",
            meta={"proxy": "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"},
        )

    def parse(self, response):
        if response.status == 403:
            self.logger.warning("403 on %s: %s", response.url, response.css("title::text").get())
            return
        yield {"title": response.css("title::text").get()}

Scrapy's default User-Agent names Scrapy, so setting a browser one is the first fix. Full proxy setup: Scrapy.

A 403 that persists for one IP is also covered in why is my IP address blocked.