Slow Proxies and Changing Sticky IPs

A proxy adds one more hop to every request, so some extra time is normal; a proxy that is much slower than that has a cause you can find. The table puts the usual causes in the order to check them:

ProxyField's figures for comparison: an average response of about 400 ms, according to our service data. By type, a typical response is under 1 s for residential and under 1.5 s for mobile, according to our service data. Datacenter proxies cost less per GB and suit sites that accept datacenter ranges; residential vs datacenter proxies compares them.

One cURL command splits a request into its stages. Run it 5 times through the proxy, then without -x and -U, and compare:

curl -s -o /dev/null -x gate.proxyfield.io:7000 -U "<login>-country-us-city-new_york:<password>" \
  -w "dns %{time_namelookup}s  connect %{time_connect}s  tls %{time_appconnect}s  first byte %{time_starttransfer}s  total %{time_total}s\n" \
  https://example.com/

Each value counts from the start of the request (cURL --write-out variables):

• dns: the proxy's host name was resolved. A high value points to your DNS resolver.
• connect: the TCP connection to the proxy was open. A high value points to your network or a distant gateway.
• tls: the TLS handshake with the website, through the proxy, was done. The gap from "connect" is the tunnel plus the handshake, and it repeats on every new connection.
• first byte: the website started to answer. The gap from "tls" is mostly the website's own time plus the distance from the exit to the site.
• total: the whole response arrived. A big gap from "first byte" means a large page.

If "first byte" is as slow without the proxy, the website is the bottleneck. If it drops when you target a location closer to the site, distance was the cause. The proxy checker gives a quick reading for one endpoint; each check uses a small amount of proxy traffic.

1. Pick an exit near the website. ProxyField targets country, region (state) and city, with all 50 US states. A US retailer answers faster to a US exit than to one on another continent.
2. Reuse connections. A requests.Session in Python, or keep-alive in any HTTP client, keeps the TCP connection and the TLS handshake for the next request to the same site. If you rotate on every request, read "Why a rotating proxy can return the same IP twice" below before you reuse connections.
3. Tune concurrency. Start low (for example 10 parallel requests) and raise it in steps while the median time and the error rate stay flat. ProxyField allows unlimited concurrent threads, but your machine and the website have limits; past them, everything slows down.
4. Download less. In a browser, block images, fonts and media you don't parse. Traffic is billed per GB ($2.95/GB for 1 GB down to $1.95/GB from 1,000 GB), so a lighter page is also a cheaper one.
5. Time out and retry. Set a connect timeout and a read timeout, and retry a slow request on a new IP instead of waiting. On ProxyField, "If a node slows down, traffic automatically moves to the next one."
6. Match the IP type to the target. Use datacenter IPs where the site accepts them, residential where it doesn't, and mobile only where the site expects mobile users.

This Python example reuses connections across 10 threads and prints each request's time, so you can see the effect of a change:

import time
from concurrent.futures import ThreadPoolExecutor

import requests
from requests.adapters import HTTPAdapter

proxy = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
session = requests.Session()  # keeps connections open between requests
session.proxies = {"http": proxy, "https": proxy}
session.mount("https://", HTTPAdapter(pool_connections=10, pool_maxsize=10))


def fetch(url):
    start = time.perf_counter()
    try:
        r = session.get(url, timeout=(10, 30))  # (connect, read) in seconds
        result = r.status_code
    except requests.RequestException as e:
        result = type(e).__name__
    return url, result, round(time.perf_counter() - start, 2)


urls = ["https://example.com/"] * 20
with ThreadPoolExecutor(max_workers=10) as pool:
    for row in pool.map(fetch, urls):
        print(row)

In Playwright, one line blocks the heavy resource types before you open a page (setup: Playwright with proxies):

page.route("**/*", lambda route: route.abort()
           if route.request.resource_type in ("image", "media", "font")
           else route.continue_())

More ways to speed up a scraper beyond the proxy are in how to make web scraping faster.

A sticky session keeps one IP only for its set time and only for the same session settings. Check these causes:

• The session time ran out. ProxyField sticky sessions last from 30 s to 36 h, with presets of 1, 5, 10 and 30 min and 1, 3 and 6 h. When the time ends, the next request gets a new IP. Set a session longer than the task.
• The session settings changed. A sticky session is tied to its session settings; another location, other session settings or a typo in the proxy line start a new session with a new IP. The syntax: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default) after the location, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600
• The session ended early. If the IP changed well before the session time and nothing above applies, send the 24/7 live chat the time, your session settings and both IPs. Whether a sticky session can end early, and what you see then: the IP is not guaranteed if the exit node drops; on a connection error the gateway retries once with a new session, so the IP may change, and a connection that breaks is not restored, so your client reconnects
• Your tool rotates. Some tools pick a new proxy line for each profile, thread or retry. Check that every step of the flow uses the same proxy line.

Check the exit IP at the start of a flow and before each important step; if it changed, restart the flow instead of continuing with half a session. When to use sticky and when to rotate: sticky vs rotating proxies.

Why a rotating proxy can return the same IP twice

Per-request rotation gives each request a new IP from the pool, and a large pool can still hand out the same address again. Connection reuse may do it too: an HTTPS request travels inside a tunnel through the proxy, and requests that reuse one open tunnel may leave through the exit that opened it. If you need a new IP on every request and see repeats, close the connection after each one and compare. An open HTTPS connection keeps one exit IP until it closes.

When the same code works on your laptop and fails on a cloud server, the server usually sends something different, or sends some requests without the proxy:

This line, run on the server, shows the IP the website sees for a request through the proxy:

curl -x gate.proxyfield.io:7000 -U "<login>-country-us-city-new_york:<password>" https://api.ipify.org

If it prints a ProxyField exit IP but your scraper is still blocked, compare the scraper's own requests with it: one of them leaves without the proxy.

Send 24/7 live chat the time and time zone, your location targeting, the target URL and the cURL timing line from above. If the requests fail rather than crawl, start with why is my proxy not working; timeouts have their own guide, 502 Bad Gateway and 504 Gateway Timeout.