Scrapy Proxy Setup: Step by Step

import os
import scrapy

PROXY = f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}@gate.proxyfield.io:7000"


class IpSpider(scrapy.Spider):
    name = "ip"

    def start_requests(self):
        for _ in range(3):
            yield scrapy.Request("https://api.ipify.org?format=json", meta={"proxy": PROXY}, dont_filter=True)

    def parse(self, response):
        yield response.json()  # a different IP on each line

Run it with scrapy runspider ip_spider.py -o ips.json. HttpProxyMiddleware, on by default, reads meta["proxy"], turns the username and password in the URL into a Proxy-Authorization header, and opens an HTTPS tunnel for https:// URLs, so TLS runs from Scrapy to the site.

Rather than adding meta in every spider, set it once in a downloader middleware. In middlewares.py:

class ProxyFieldMiddleware:
    def __init__(self, proxy):
        self.proxy = proxy

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.get("PROXYFIELD_URL"))

    def process_request(self, request, spider):
        request.meta.setdefault("proxy", self.proxy)  # a spider can still override it per request

In settings.py:

import os

PROXYFIELD_URL = f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}@gate.proxyfield.io:7000"

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyFieldMiddleware": 350,  # runs before HttpProxyMiddleware (750)
}

The number matters: your middleware must run before HttpProxyMiddleware, which handles the credentials. Replace myproject with your project's package name. A quicker option for a test: HttpProxyMiddleware also reads the http_proxy and https_proxy environment variables when the crawl starts.

Scrapy has no rotation of its own, and with ProxyField you don't need a rotation package or a list: the gateway gives a new IP on every request, so one proxy URL rotates on its own.

• A new IP per request: the rotating username. Scrapy may reuse an open connection for several requests to the same site; if each request must get its own IP, check the IPs your spider logs, as in section 1. An open HTTPS connection keeps one exit IP until it closes.
• One IP for a sequence (a login and the pages behind it): a sticky session in the username, 30 s to 36 h, set per request in meta. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional. Residential is the default; add -mobile after the login for mobile IPs: <login>-mobile-country-us.
• Several countries: a proxy URL per country, picked per request:

import os, random

def proxy_for(country):
    return f"http://{os.environ['PROXY_USER']}-country-{country}:{os.environ['PROXY_PASS']}@gate.proxyfield.io:7000"

# in a spider or middleware
request.meta["proxy"] = proxy_for(random.choice(["us", "gb", "de"]))

The -country- key follows the example <login>-country-us-city-new_york. Country keys outside the US: the two-letter country code in lowercase, as in -country-de.

When you change the proxy of a request, keep the credentials in the URL. Since Scrapy 2.6.2, HttpProxyMiddleware drops a stale Proxy-Authorization header when the proxy URL changes; on older versions, or if you set that header by hand, a rotated request can carry the previous login and fail with 407. Update Scrapy rather than managing the header yourself.

The gateway accepts unlimited concurrent threads, so the limits to set are for the target site's sake:

CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_DELAY = 0.5
AUTOTHROTTLE_ENABLED = True
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]
DOWNLOAD_TIMEOUT = 60

Leave 403 out of RETRY_HTTP_CODES: it usually means the site refused the request, and repeating it unchanged rarely helps. A retried request on a rotating username goes out from a new IP. More on refusals: 403 Forbidden When Scraping Through a Proxy.

scrapy-playwright renders pages in a real browser, which takes the proxy in its launch options (or per browser context):

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "proxy": {
        "server": "http://gate.proxyfield.io:7000",
        "username": os.environ["PROXY_USER"],
        "password": os.environ["PROXY_PASS"],
    }
}

Requests with meta={"playwright": True} then go through the proxy. For one IP per browser context, define contexts in PLAYWRIGHT_CONTEXTS, each with its own proxy, and pick one with meta["playwright_context"]. The same settings are explained for plain Playwright on Playwright proxy setup.

Splash takes the proxy as an argument of each render request:

from scrapy_splash import SplashRequest

yield SplashRequest(url, self.parse, args={"wait": 1, "proxy": PROXY})

Scrapy's downloader does not support SOCKS proxies. Use the HTTP proxy URL above: ProxyField's gateway speaks HTTP, HTTPS and SOCKS5 on the same account, and HTTP covers everything Scrapy fetches. If another part of your stack needs SOCKS5, see SOCKS5 proxies.

1. Run the spider in section 1: each line should show an IP that is not your server's.
2. Scrapy's stats at the end of a crawl count responses by status: watch downloader/response_status_count/407 (the login), /403 and /429 (the site) and retry/count.
3. A 407 means the proxy did not accept the login: check the credentials, encode special characters with urllib.parse.quote, and update Scrapy if you rotate credentials.
4. Timeouts point to the port or a firewall: allow outgoing port 7000.
5. To test lines without a spider, use the proxy checker.

Gateway error messages: 407 Proxy Authentication Required for a wrong login or password, an IP not on the allowlist or a malformed location, 402 Payment Required when no traffic is left and 502 Bad Gateway when no exit could be connected; a restricted site gets no separate response, and a 403 comes from the site itself. Status codes are explained in proxy errors and HTTP Error 407 Proxy Authentication Required. Some site categories are restricted to protect the network, including financial and government services. Our 24/7 live chat helps with setup and integration.

If you would rather receive pages than run proxies, the Web Scraping API fetches any URL for $0.50 down to $0.20 per 1,000 successful results (plain HTTP) or $2.00 down to $1.00 with JS rendering, billed on successful results only, through Contact sales.

New to Scrapy? Start with Scrapy Tutorial: Build Your First Spider. For single requests outside a crawl, see Python requests. Use ProxyField for lawful work such as public web data collection, price and SEO monitoring and testing, within our acceptable use policy.

Start with the Easy Trial: 1 GB for $2.95, with no minimum purchase and no subscription. The traffic never expires and all purchases are final. See pricing per GB.