Python Proxy Setup: requests, httpx, aiohttp and Rotation

Point every request at the gateway with a username that has no session: each request then leaves from a new residential IP. You need no proxy list and no rotator.

import os
import requests

PROXY = f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}@gate.proxyfield.io:7000"
proxies = {"http": PROXY, "https": PROXY}

for i in range(5):
    print(requests.get("https://api.ipify.org", proxies=proxies, timeout=30).text)

203.0.113.24
198.51.100.71
192.0.2.145
203.0.113.199
198.51.100.8

curl -x gate.proxyfield.io:7000 -U "<login>:<password>" https://api.ipify.org

Start with requests, or httpx if you want async code or HTTP/2; use Scrapy for crawls of many pages, and Playwright when the page needs JavaScript. Parse the HTML with Beautiful Soup in every case. All of them take the same ProxyField URL.

The libraries without proxies, compared for beginners: Python web scraping libraries; async patterns in depth: async web scraping in Python.

import os
import requests

PROXY = f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}@gate.proxyfield.io:7000"
proxies = {"http": PROXY, "https": PROXY}

r = requests.get("https://api.ipify.org", proxies=proxies, timeout=30)
print(r.text)  # the proxy's IP

Set PROXY_USER=<login>-country-us to get a US IP. The "https" key is the one used for https:// URLs; the scheme of its value (http://) is how Python talks to the proxy. For an https:// URL, the client asks the proxy to open a tunnel with CONNECT and then runs TLS through it end to end, so the proxy forwards encrypted bytes; ProxyField's gateway carries HTTPS this way and does not decrypt it. For a plain http:// URL, the proxy forwards the request itself. Encode special characters in the password with urllib.parse.quote(password, safe="").

For many requests, put the proxy on a session:

s = requests.Session()
s.proxies.update(proxies)
print(s.get("https://ipinfo.io/json", timeout=30).json())

More on requests (environment variables, retries, TLS) is on our Python requests guide.

import os
import urllib.request

PROXY = f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}@gate.proxyfield.io:7000"
opener = urllib.request.build_opener(urllib.request.ProxyHandler({"http": PROXY, "https": PROXY}))

with opener.open("https://api.ipify.org", timeout=30) as resp:
    print(resp.read().decode())

ProxyHandler sends the username and password from the URL to the proxy. Without a handler, urllib uses the http_proxy and https_proxy environment variables.

• httpx: httpx.AsyncClient(proxy=PROXY), then await client.get(url).
• aiohttp: session.get(url, proxy="http://gate.proxyfield.io:7000", proxy_auth=aiohttp.BasicAuth(user, password)).

httpx, sync or async: the proxy argument arrived in httpx 0.26, which deprecated proxies, and httpx 0.28 removed proxies, so old code that passes proxies= fails on 0.28 (httpx changelog):

import asyncio, os
import httpx

PROXY = f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}@gate.proxyfield.io:7000"

async def main():
    async with httpx.AsyncClient(proxy=PROXY, timeout=30) as client:
        r = await client.get("https://api.ipify.org")
        print(r.text)

asyncio.run(main())

mounts = {
    "http://": httpx.AsyncHTTPTransport(proxy=PROXY),
    "https://": httpx.AsyncHTTPTransport(proxy=PROXY),
}
async with httpx.AsyncClient(mounts=mounts, timeout=30) as client:
    r = await client.get("https://api.ipify.org")

import asyncio, os
import aiohttp

PROXY = "http://gate.proxyfield.io:7000"
AUTH = aiohttp.BasicAuth(os.environ["PROXY_USER"], os.environ["PROXY_PASS"])

async def fetch(session, sem, url):
    async with sem:
        async with session.get(url, proxy=PROXY, proxy_auth=AUTH) as r:
            return await r.text()

async def main(urls):
    sem = asyncio.Semaphore(20)
    timeout = aiohttp.ClientTimeout(total=30)
    connector = aiohttp.TCPConnector(force_close=True)  # a new connection per request
    async with aiohttp.ClientSession(timeout=timeout, connector=connector) as session:
        return await asyncio.gather(*(fetch(session, sem, u) for u in urls))

print(asyncio.run(main(["https://api.ipify.org"] * 5)))

async def fetch_retry(session, sem, url, attempts=3, pause=2):
    for i in range(attempts):
        try:
            return await fetch(session, sem, url)
        except (aiohttp.ClientError, asyncio.TimeoutError):
            if i == attempts - 1:
                raise
            await asyncio.sleep(pause * 2 ** i)  # 2 s, 4 s

async def main_retry(urls):
    sem = asyncio.Semaphore(20)
    async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=30),
                                     connector=aiohttp.TCPConnector(force_close=True)) as session:
        results = await asyncio.gather(*(fetch_retry(session, sem, u) for u in urls), return_exceptions=True)
    failed = [u for u, r in zip(urls, results) if isinstance(r, Exception)]
    return results, failed

pip install "requests[socks]"

PROXY = "socks5h://<login>:<password>@gate.proxyfield.io:7000"
r = requests.get("https://api.ipify.org", proxies={"http": PROXY, "https": PROXY}, timeout=30)

socks5h lets the proxy resolve hostnames, so DNS lookups happen at the proxy's location; socks5 resolves them on your machine. SOCKS5 takes the same username and password. httpx takes SOCKS5 after pip install "httpx[socks]", aiohttp through the aiohttp-socks connector. For raw sockets, PySocks sets a default proxy for the whole program:

import socket, socks  # pip install PySocks

socks.set_default_proxy(socks.SOCKS5, "gate.proxyfield.io", 7000, rdns=True, username="<login>", password="<password>")
socket.socket = socks.socksocket

When SOCKS5 is worth it over HTTP: SOCKS5 proxies.

You don't need a proxy list or a rotator. Send every request to the gateway: with a username without a session, each request gets a new IP.

• A new IP per request: one gateway URL for all requests. A client that keeps one HTTPS connection open for several requests to the same site sends them through one tunnel; open a new connection per request (plain requests.get, or force_close=True in aiohttp) when each request needs its own IP. An open HTTPS connection keeps one exit IP until it closes.
• One IP for a task: add a session id to the username and keep it for 30 s to 36 h, for example for a login and the pages behind it. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional. Residential is the default; add -mobile after the login for mobile IPs: <login>-mobile-country-us.
• Several countries: build the proxy URL per task from the username: <login>-country-us, <login>-country-gb, <login>-country-de.

import os
import requests

def proxy_for(country: str) -> dict:
    url = f"http://{os.environ['PROXY_USER']}-country-{country}:{os.environ['PROXY_PASS']}@gate.proxyfield.io:7000"
    return {"http": url, "https": url}

for c in ("us", "gb", "de"):
    print(c, requests.get("https://ipinfo.io/json", proxies=proxy_for(c), timeout=30).json()["country"])

Retry a failed request after a short wait rather than at once, and treat 407 (credentials), 429 (rate limit) and 403 (blocked) differently: they are explained in proxy errors.

BeautifulSoup parses HTML; it doesn't fetch pages. Fetch with requests or httpx through the proxy and pass the text to it:

from bs4 import BeautifulSoup

html = requests.get("https://example.com", proxies=proxies, timeout=30).text
print(BeautifulSoup(html, "html.parser").title.string)

In Scrapy, set the proxy per request with meta={"proxy": "http://<login>:<password>@gate.proxyfield.io:7000"}; the built-in HttpProxyMiddleware sends the credentials (Scrapy proxy setup). For pages that need JavaScript, use a real browser. Playwright for Python takes the proxy at launch:

import os
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={
        "server": "http://gate.proxyfield.io:7000",
        "username": os.environ["PROXY_USER"],
        "password": os.environ["PROXY_PASS"],
    })
    page = browser.new_page()
    page.goto("https://api.ipify.org")
    print(page.inner_text("body"))
    browser.close()

Use the HTTP endpoint in Playwright: Chromium takes no login for SOCKS5. More on Playwright, Selenium and Puppeteer. Browser Use drives a Playwright browser, so it takes the same server, username and password in its browser settings.

To install packages through the proxy:

pip install --proxy "http://<login>:<password>@gate.proxyfield.io:7000" requests

Or set it once in pip.conf (pip.ini on Windows) under [global] as proxy = http://<login>:<password>@gate.proxyfield.io:7000. requests, urllib, httpx and pip also read HTTP_PROXY, HTTPS_PROXY and NO_PROXY from the environment, which is how to proxy a script you don't want to edit; see Linux or Windows for setting them.

If you would rather receive the pages than run the proxies, the Web Scraping API fetches any URL, plain HTTP or with JS rendering: $0.50 down to $0.20 per 1,000 successful results for HTTP and $2.00 down to $1.00 with JS rendering, billed on successful results only, through Contact sales.

1. requests.get("https://api.ipify.org", proxies=proxies).text prints the exit IP; it should not be your server's.
2. requests.get("https://ipinfo.io/json", proxies=proxies).json() shows the city and country.
3. requests.exceptions.ProxyError ending in Tunnel connection failed: 407 Proxy Authentication Required (HTTP's generic status) means the proxy did not accept your login: check the credentials and encode special characters. More Python error strings are in proxy errors.
4. A timeout points to the port or a firewall: allow outgoing port 7000.
5. To test lines without code, use the proxy checker.

Gateway error messages: 407 Proxy Authentication Required for a wrong login or password, an IP not on the allowlist or a malformed location, 402 Payment Required when no traffic is left and 502 Bad Gateway when no exit could be connected; a restricted site gets no separate response, and a 403 comes from the site itself. Some site categories are restricted to protect the network, including financial and government services. Our 24/7 live chat helps with setup and integration.

The same proxy URL works from Python, Node.js, Go or cURL. Use ProxyField for lawful work such as public web data collection, price and SEO monitoring and testing, on public pages, within each site's terms and our acceptable use policy.

Start with the Easy Trial: 1 GB for $2.95, with no minimum purchase and no subscription. The traffic never expires and all purchases are final. See pricing per GB.