Python Requests Proxy Setup

import os
import requests

PROXY = f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}@gate.proxyfield.io:7000"
proxies = {"http": PROXY, "https": PROXY}

r = requests.get("https://api.ipify.org", proxies=proxies, timeout=30)
print(r.text)  # the proxy's IP, not your server's

The keys of proxies are the schemes of the URLs you request: "https" is used for every https:// URL and "http" for every http:// URL. The value is how requests reaches the proxy, so it starts with http:// in both keys. That does not weaken HTTPS: requests opens a tunnel through the proxy with CONNECT, and TLS runs from your code to the site inside it. A dictionary with only an "http" key leaves every https:// URL on your own IP.

POST, PUT and every other method take the same argument:

r = requests.post("https://httpbin.org/post", json={"q": "test"}, proxies=proxies, timeout=30)

Put the login in the proxy URL, as above. Encode special characters in the password first:

from urllib.parse import quote

PROXY = f"http://{quote(user, safe='')}:{quote(password, safe='')}@gate.proxyfield.io:7000"

requests.auth.HTTPProxyAuth is not a substitute for HTTPS URLs: it adds the Proxy-Authorization header to the request itself, and for https:// URLs that request travels inside the encrypted tunnel, where the proxy never sees it. The result is a 407 Proxy Authentication Required error. Credentials in the URL are sent when the tunnel is opened, so they work for both schemes. More on the error: HTTP Error 407 Proxy Authentication Required.

With your server's IP on the ProxyField allowlist, drop the login entirely and use the port you set up in the dashboard: PROXY = "http://gate.proxyfield.io:<port>", where <port> is that port. Proxy Authentication compares the two methods.

A Session keeps cookies and reuses connections. Give it the proxy once:

s = requests.Session()
s.proxies.update(proxies)
r = s.get("https://ipinfo.io/json", timeout=30)
print(r.json()["country"])

One catch: if HTTP_PROXY or HTTPS_PROXY is set in the environment, requests uses it ahead of s.proxies. Either pass proxies= on each call, or switch the environment off for the session with s.trust_env = False.

The environment is also the way to proxy a script you don't want to edit:

export HTTPS_PROXY="http://<login>:<password>@gate.proxyfield.io:7000"
export HTTP_PROXY="$HTTPS_PROXY"
export NO_PROXY="localhost,127.0.0.1"
python3 script.py

requests reads HTTP_PROXY, HTTPS_PROXY, ALL_PROXY and NO_PROXY (upper or lower case); without them it falls back to the operating system's proxy settings. Setting these variables on each system: Set a Proxy From the Command Line.

You don't need a list of IPs or a rotator. ProxyField's gateway gives a new IP on every request, so the same proxy URL rotates on its own:

import os
import requests

PROXY = f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}@gate.proxyfield.io:7000"
proxies = {"http": PROXY, "https": PROXY}

for _ in range(3):
    print(requests.get("https://api.ipify.org", proxies=proxies, timeout=30).text)  # a new IP each time

requests.get opens a new connection each time. A Session keeps one HTTPS tunnel open for repeated requests to the same site, which can keep the same IP for those requests; use plain requests.get when every request needs its own IP, and a session when it should stay put. An open HTTPS connection keeps one exit IP until it closes.

For one IP across a login and the pages behind it, use a sticky session in the username, which holds the IP for 30 s to 36 h. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional. Residential is the default; add -mobile after the login for mobile IPs: <login>-mobile-country-us.

To spread work across countries, build the proxy per country:

def proxies_for(country: str) -> dict:
    url = f"http://{os.environ['PROXY_USER']}-country-{country}:{os.environ['PROXY_PASS']}@gate.proxyfield.io:7000"
    return {"http": url, "https": url}

for c in ("us", "gb", "de"):
    print(c, requests.get("https://ipinfo.io/json", proxies=proxies_for(c), timeout=30).json()["country"])

The -country- key follows the example <login>-country-us-city-new_york. Country keys outside the US: the two-letter country code in lowercase, as in -country-de.

A key can name a scheme and a host, so one dictionary can route sites differently:

proxies = {
    "https://example.com": proxies_for("us")["https"],  # this site through a US IP
    "https": proxies_for("gb")["https"],                  # every other https:// URL through a UK IP
}

The more specific key wins. To keep a host off the proxy, list it in NO_PROXY.

pip install "requests[socks]"

PROXY = "socks5h://<login>:<password>@gate.proxyfield.io:7000"
proxies = {"http": PROXY, "https": PROXY}

Without the extra package, requests stops with InvalidSchema: Missing dependencies for SOCKS support. socks5h resolves hostnames at the proxy; socks5 resolves them on your machine. SOCKS5 takes the same username and password. HTTP is enough for web pages; SOCKS5 proxies explains when SOCKS5 is worth it.

Mount an adapter with a retry policy, and always set a timeout:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504])
s = requests.Session()
s.mount("https://", HTTPAdapter(max_retries=retry))
s.mount("http://", HTTPAdapter(max_retries=retry))
r = s.get("https://api.ipify.org", proxies=proxies, timeout=(10, 30))

In rotating mode each retry is a new request, so it goes out from a new IP. Treat a 403 as a block rather than a glitch: slow down, check the request headers, and don't retry the same request in a loop; 403 Forbidden When Scraping Through a Proxy lists the causes. Keep verify=True (the default): the site's certificate is checked end to end through the tunnel, so there is no need to switch verification off for the proxy.

1. requests.get("https://api.ipify.org", proxies=proxies).text prints the exit IP; it should not be your server's.
2. Still your own IP? The "https" key is missing or misspelled, NO_PROXY matches the host, or a session's environment overrides it.
3. A ProxyError with an authentication error (407 in most proxies) means the login was not accepted: check the credentials and encode special characters.
4. ConnectTimeout points to the port or a firewall: allow outgoing port 7000.
5. Test a proxy line without code with the proxy checker.

Gateway error messages: 407 Proxy Authentication Required for a wrong login or password, an IP not on the allowlist or a malformed location, 402 Payment Required when no traffic is left and 502 Bad Gateway when no exit could be connected; a restricted site gets no separate response, and a 403 comes from the site itself. Common error codes are explained in proxy errors. Some site categories are restricted to protect the network, including financial and government services. Our 24/7 live chat helps with setup and integration.

requests is one of several Python clients; httpx, aiohttp and urllib are covered on Python proxy setup, and crawls of many pages on Scrapy proxy setup. Use ProxyField for lawful work such as public web data collection, price and SEO monitoring and testing, within our acceptable use policy.

Start with the Easy Trial: 1 GB for $2.95, with no minimum purchase and no subscription. The traffic never expires and all purchases are final. See pricing per GB.