Proxy Error 429 Too Many Requests

A 429 tells you that you hit a limit the site set on purpose. RFC 6585, section 4 defines it as "the user has sent too many requests in a given amount of time" and lets the server add a Retry-After header that says how long to wait. MDN's 429 page gives the same meaning.

What the limit counts depends on the site:

Who sent the 429 matters. If the response comes from the target site (its branding, its API error format, or a Cloudflare page with error 1015), the site rate-limited you. A 429 in your proxy provider's own format would mean a limit on the proxy side; ProxyField allows unlimited concurrent threads, and its gateway's own responses are 407 (wrong login or an IP not on the allowlist), 402 (no traffic left) and 502 (no exit could be connected).

Work through these in order; most 429s stop after the first two.

1. Read the Retry-After header and wait

If the response carries Retry-After, wait at least that long before the next request to that site. The value is either a number of seconds or an HTTP date (RFC 9110, section 10.2.3). Some APIs add their own headers, such as X-RateLimit-Remaining or RateLimit-Reset; these are not standardized, so read the API's documentation for what they mean.

2. Slow down and back off

Lower the number of requests per second per domain, add a random delay between requests, and double the wait after each 429 (exponential back-off) up to a cap. Retrying at once makes it worse: Cloudflare's Error 1015 page warns that repeated attempts within a short period "may extend the block".

This Python requests loop honors Retry-After, backs off on every 429 and sends each attempt through the ProxyField gateway, so a retry leaves from a new IP:

import random
import time
from email.utils import parsedate_to_datetime

import requests

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
PROXIES = {"http": PROXY, "https": PROXY}


def retry_after_seconds(value):
    if not value:
        return None
    if value.isdigit():
        return int(value)
    try:
        return max(0, parsedate_to_datetime(value).timestamp() - time.time())
    except (TypeError, ValueError):
        return None


def get(url, attempts=5):
    delay = 2.0
    for _ in range(attempts):
        r = requests.get(url, proxies=PROXIES, timeout=30)
        if r.status_code != 429:
            return r
        wait = retry_after_seconds(r.headers.get("Retry-After")) or delay
        time.sleep(wait + random.uniform(0, 1))
        delay = min(delay * 2, 120)
    return r


print(get("https://example.com/").status_code)

3. Spread requests over rotating IPs

If the site counts requests per IP, rotation divides your load: with a new IP on every request, each address sends a fraction of the total. ProxyField's rotating proxies can give a new IP on every request, with unlimited concurrent threads and country, region and city targeting. Rotation does not raise a limit that counts your account, API key or cookies, and it does not make an aggressive crawl polite: keep the per-domain pace reasonable.

4. Keep sessions where the site expects them

Some sites tie a session cookie to one IP and rate-limit sessions that jump between addresses. For logged-in or multi-step flows, use a sticky session (30 s to 36 h; presets 1, 5, 10, 30 min, 1, 3 and 6 h) and rotate between sessions, not inside one. The session part of the username is -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default) after the location, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600. See sticky vs rotating proxies for when to use which.

5. Send fewer, lighter requests

Cache pages you already have, skip images, fonts and scripts you don't parse, and request only the pages that changed. If the site offers an official API or a data export, use it within its quota. For Google results, ProxyField's Google SERP API returns the results page as JSON at $1.20 per 1,000 successful results, falling to $0.80 at volume.

6. Stop if the limit is the site's answer

If the 429s continue at a slow, polite pace from many IPs, the site does not want this volume of automated traffic. Read its terms and robots.txt and ask the owner for access or an API key; see how to check if scraping is allowed.

Scrapy retries 429 by default: 429 is in the default RETRY_HTTP_CODES list of the retry middleware. Add pacing so the retries don't hit the same limit:

# settings.py
AUTOTHROTTLE_ENABLED = True          # adapts the delay to the server's response times
AUTOTHROTTLE_START_DELAY = 2
AUTOTHROTTLE_MAX_DELAY = 60
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
DOWNLOAD_DELAY = 1                   # base delay per domain, randomized by default
CONCURRENT_REQUESTS_PER_DOMAIN = 4
RETRY_TIMES = 5

Set the proxy per request with request.meta["proxy"] = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"; the full setup is on the Scrapy proxy page. AutoThrottle and DOWNLOAD_DELAY are documented in the Scrapy docs.

A 403 that appears right after a run of 429s often means the site escalated from slowing you down to blocking you. The 403 Forbidden when scraping doc and the Cloudflare access denied errors doc cover those cases.

A new IP fixes a 429 only when the limit counts IPs. It does not help when:

• the limit counts your API key, account or session cookie;
• the site's quota is global (for example, a fixed daily quota on an API plan);
• you send so many requests that every IP in your rotation hits the limit.

In those cases, slow down, buy a higher quota from the site, or ask for access. Retries that keep failing also move data through the proxy, so back-off saves bandwidth as well as the target's patience.