How to Use Proxies for Web Scraping

For the proxy type to buy for scraping and the cost per page, see web scraping proxies. For the code, you need:

• Proxy access: the gateway gate.proxyfield.io:7000 with a username and password, or your server's IP on the allowlist and the port you set up in the dashboard. Access opens once the payment is confirmed; both are in your account.
• A location, if the site varies by country: written into the username, as in <login>-country-us-city-new_york. ProxyField has 412M+ residential IPs in 195+ countries, with targeting by country, region (state) and city (proxy location targeting).
• A scraper: any HTTP client or browser automation tool that accepts a proxy. The gateway speaks HTTP, HTTPS and SOCKS5; the examples below use HTTP with a username and password.

Send one request through the proxy before you touch the scraper:

curl -x gate.proxyfield.io:7000 -U "<login>-country-us-city-new_york:<password>" https://api.ipify.org

The IP it prints should not be yours. Run it twice: on a rotating gateway the 2 IPs differ. A 407 means the username or password is wrong (407 proxy authentication required).

The proxy goes wherever your code opens connections.

Python requests

import requests

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
PROXIES = {"http": PROXY, "https": PROXY}

r = requests.get("https://example.com/products?page=1", proxies=PROXIES, timeout=30)
print(r.status_code, len(r.text))

Scrapy: set the proxy per request; Scrapy's built-in proxy middleware reads the credentials from the URL.

import scrapy

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"

class ProductsSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/products?page=1"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": PROXY})

    def parse(self, response):
        for title in response.css(".product-card h2::text").getall():
            yield {"title": title.strip()}

Node.js (fetch with undici)

import { fetch, ProxyAgent } from "undici";

const proxy = new ProxyAgent({
  uri: "http://gate.proxyfield.io:7000",
  token: "Basic " + Buffer.from("<login>-country-us-city-new_york:<password>").toString("base64"),
});

const res = await fetch("https://example.com/products?page=1", { dispatcher: proxy });
console.log(res.status, (await res.text()).length);

Headless browsers take the proxy at launch: see Playwright, Puppeteer and Selenium. More libraries: Python, Scrapy, Node.js, cURL.

Use logins only for your own account and for data you are allowed to access; the acceptable use policy does not allow access to data behind logins you are not entitled to. Scraping behind a login is allowed only for data you are entitled to and the site's terms allow.

ProxyField holds a sticky session from 30 seconds to 36 hours (presets 1, 5, 10 and 30 minutes, 1, 3 and 6 hours). The username syntax for a session: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default) after the location, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600.

One detail matters for per-request rotation: an HTTPS request travels inside a tunnel through the proxy, and requests that reuse one open tunnel may leave through the exit that opened it. If every request must get a new IP, open a new connection for each one (as requests.get without a Session does); reuse a Session when speed matters more than a new IP on every request. An open HTTPS connection keeps one exit IP until it closes.

A block does not always come as an error. Count a response as failed when:

• the status is 403, 429 or 503;
• the page is a challenge or "access denied" page, even with status 200;
• the content you parse is missing (no title, no price, an empty list).

Then wait and try again; on a rotating gateway the next attempt leaves from a new IP. Stop after 2 or 3 attempts and log the URL, so a hard block does not eat your traffic in a loop. Honor Retry-After when the site sends it.

import random
import re
import time
import requests

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
PROXIES = {"http": PROXY, "https": PROXY}
BLOCK_MARKERS = ("captcha", "access denied", "unusual traffic", "verify you are human")

def looks_blocked(r):
    if r.status_code in (403, 429, 503):
        return True
    m = re.search(r"<title[^>]*>(.*?)</title>", r.text[:5000], re.I | re.S)
    title = m.group(1).lower() if m else ""
    return any(marker in title for marker in BLOCK_MARKERS)  # check the title, not the whole page

def fetch(url, attempts=3):
    for attempt in range(1, attempts + 1):
        wait = 2 ** attempt
        try:
            r = requests.get(url, proxies=PROXIES, timeout=30)
            if r.status_code == 200 and not looks_blocked(r):
                return r.text
            retry_after = r.headers.get("Retry-After", "")
            if retry_after.isdigit():
                wait = int(retry_after)
        except requests.RequestException:
            pass
        if attempt < attempts:
            time.sleep(wait + random.random())  # the next attempt gets a new IP
    return None  # log it and move on

What each status code means and how to fix it: 403 forbidden, 429 too many requests, and the full proxy errors guide.

Rotation spreads requests over IPs, but sites also judge the total rate against them. ProxyField allows unlimited concurrent threads, so the limit is yours to set:

1. Start with 3-5 parallel requests per website.
2. Raise the number while the success rate holds and response times stay flat.
3. Back off at the first 429 responses or when response times climb.
4. Add a small random delay between requests to the same site.

from concurrent.futures import ThreadPoolExecutor

urls = [f"https://example.com/products?page={n}" for n in range(1, 51)]
with ThreadPoolExecutor(max_workers=4) as pool:  # 4 parallel requests to this site
    pages = list(pool.map(fetch, urls))
print(sum(p is not None for p in pages), "of", len(urls), "pages fetched")

An IP is one signal of several. Send a current browser User-Agent with the headers that browser sends (Accept, Accept-Language), keep cookies within a sticky session and drop them when the IP changes, and match the Accept-Language to the exit country where it matters. A proxy cannot fix a scraper that announces itself in its headers: what proxies can and cannot fix.

Log, per site: requests sent, successes (by content, not status), retries and response times. Watch traffic in the dashboard, which shows usage in real time and the gigabytes left. If traffic runs ahead of your estimate, the usual causes are a browser loading whole pages and retries in a loop: block images, fonts and media you do not parse, and cap retries at 2 or 3.

Scrape public data, read each site's terms and robots.txt, and keep within our acceptable use policy; the legal side is in is web scraping legal.