Proxy Management: Track Traffic and Costs per Project

Most teams start with the gateway and add static IPs only where one IP must stay tied to one account. How to set up each proxy type covers the setup side.

Treat the proxy bill like a shared utility: the dashboard gives the account total, and your own request log says which project and which person used it.

The steps in detail:

• Lines per project. Each project gets its own list with its own targeting, so a stray job shows up under its project.
• Access per person. With allowlist entries on servers, nobody needs the account password. The method does not rely on sub-users: the person is a label in your log or your gateway.
• What to log. One row per attempt: time, project, person, target domain, country, status, bytes, attempt number, seconds.
• Two numbers per project: cost per successful page = project cost ÷ successful pages, and cost per usable record = project cost ÷ records you could actually use. Retries count: 1,000 good pages plus 250 failed attempts of the same size cost 1,250 page loads, so each good page costs 25% more.
• Reconciling. Note the gigabytes left in the dashboard, compare the week's drop with your log's total, and share the dashboard's figure out by each project's share of the logged bytes. Your log is an estimate and the dashboard's figure is the one you pay by. The Python logger below counts response bodies and headers, not TLS handshakes, so expect a small gap; a large gap usually means another tool uses the same login.

What to look for in a provider's dashboard

Budgets. There is no monthly plan to cap: the balance is prepaid and never expires, so a project's budget is the gigabytes you assign to it. Your code enforces it, as in the script below, and stops the job when the project reaches its share. What the meter counts, by direction and for failed requests: both directions (upload and download), including a site's error responses; a connection the gateway refuses (407, 402, 502) costs nothing

A request logger with per-project budgets (Python)

import csv, time, requests
from urllib.parse import urlparse

BUDGET_GB = {"price-monitoring": 40, "real-estate": 25}   # GB assigned to each project
used = {}                                                 # bytes per project in this run

def fetch(url, project, user, country="us", retries=2):
    proxy = f"http://<login>-country-{country}:<password>@gate.proxyfield.io:7000"
    for attempt in range(retries + 1):
        if used.get(project, 0) / 1e9 >= BUDGET_GB[project]:
            raise RuntimeError(f"{project}: budget of {BUDGET_GB[project]} GB reached")
        start = time.time()
        try:
            r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=30)
            size = len(r.content) + sum(len(k) + len(v) + 4 for k, v in r.headers.items())
            status = r.status_code
        except requests.RequestException:
            r, size, status = None, 0, "error"
        used[project] = used.get(project, 0) + size
        with open("proxy-usage.csv", "a", newline="") as f:
            csv.writer(f).writerow([time.strftime("%Y-%m-%d %H:%M:%S"), project, user,
                                    urlparse(url).hostname, country, status, size, attempt,
                                    round(time.time() - start, 2)])
        if status == 200:
            return r
        time.sleep(2 ** attempt)   # back off, then retry on a new IP
    return None

r = fetch("https://example.com/", project="price-monitoring", user="alice")

And the monthly report, cost per project and person:

import csv
from collections import defaultdict

RATE = 2.49   # your tier's $/GB, here a 100-249 GB purchase
gb, ok = defaultdict(float), defaultdict(int)
for ts, project, user, host, country, status, size, attempt, secs in csv.reader(open("proxy-usage.csv")):
    gb[project, user] += int(size) / 1e9
    ok[project, user] += status == "200"
for key in sorted(gb):
    cost = gb[key] * RATE
    print(*key, f"{gb[key]:.3f} GB", f"${cost:.2f}",
          f"${cost / max(ok[key], 1) * 1000:.2f} per 1,000 successful pages")

It prints one row per project and person, as in the cost table at the top of this page. The logger uses the Python requests library and the standard csv module. len(r.content) is the decompressed body, so the log is an upper estimate when a site compresses its pages, and failed connections log 0 bytes; the dashboard's count is the billed one. Load the CSV at start-up if a budget must carry over between runs.

Alert before the traffic runs out

Warn the team when 15% of a purchase is left, so nobody finds out from a failed job. Two ways, both with what the dashboard shows today (real-time usage and the gigabytes left):

1. Weekly, by hand: read the gigabytes left in the dashboard and compare it with 15% of what you bought (15 GB of a 100 GB purchase).
2. Daily, from your log: gigabytes left ≈ purchased GB − logged GB. Add this to the report script:

PURCHASED_GB = 100   # the purchase the team draws on
left = PURCHASED_GB - sum(gb.values())
print(f"about {left:.1f} GB left of {PURCHASED_GB} GB")
if left < 0.15 * PURCHASED_GB:
    print("WARNING: under 15% left: check the dashboard and buy more before jobs stop")

In the month above, 100 − 69.3 = 30.7 GB is left, so no warning yet. Run the script from cron and send its output to your team's chat or email. Your log is an estimate, so trust the dashboard's figure when the two differ.

Put one metering gateway in front of your scrapers

When every scraper does its own logging, the budget logic lives in every codebase. A small local gateway does it in one place: scrapers connect to it with <project>.<person> as the proxy username, and it attaches the ProxyField credentials and targeting for that project, counts the bytes in both directions, refuses projects over budget and writes one CSV row per connection.

# meter.py: one local entry point for every scraper (Python 3.8+, standard library only)
# Scrapers use the proxy http://<project>.<person>:x@<meter host>:8899
import asyncio, base64, csv, time

HOST, PORT = "127.0.0.1", 8899            # an internal address if scrapers run on other machines
UPSTREAM = ("gate.proxyfield.io", 7000)
LOGIN, PASSWORD = "<login>", "<password>"
PROJECTS = {"price-monitoring": {"gb": 40, "target": "-country-us"},
            "real-estate": {"gb": 25, "target": "-country-us-city-new_york"}}
used = {}                                 # bytes per project since start-up

async def pipe(reader, writer, count, key):
    try:
        while True:
            data = await reader.read(65536)
            if not data:
                break
            count[key] += len(data)
            writer.write(data)
            await writer.drain()
    except OSError:
        pass
    finally:
        writer.close()

async def handle(c_reader, c_writer):
    try:
        head = (await c_reader.readuntil(b"\r\n\r\n")).decode("latin-1").split("\r\n")
        method, target = head[0].split(" ")[:2]
        auth = next(l.split(":", 1)[1].split()[-1] for l in head
                    if l.lower().startswith("proxy-authorization:"))
        project, person = base64.b64decode(auth).decode().split(":")[0].split(".", 1)
    except Exception:
        method = project = person = None
    p = PROJECTS.get(project)
    if method != "CONNECT" or p is None or used.get(project, 0) >= p["gb"] * 1e9:
        c_writer.write(b"HTTP/1.1 403 Forbidden\r\n\r\n")   # the meter's own refusal
        c_writer.close()
        return
    u_reader, u_writer = await asyncio.open_connection(*UPSTREAM)
    cred = base64.b64encode(f"{LOGIN}{p['target']}:{PASSWORD}".encode()).decode()
    u_writer.write(f"CONNECT {target} HTTP/1.1\r\nHost: {target}\r\n"
                   f"Proxy-Authorization: Basic {cred}\r\n\r\n".encode())
    count, start = {"up": 0, "down": 0}, time.time()
    await asyncio.gather(pipe(c_reader, u_writer, count, "up"), pipe(u_reader, c_writer, count, "down"))
    used[project] = used.get(project, 0) + count["up"] + count["down"]
    with open("meter.csv", "a", newline="") as f:
        csv.writer(f).writerow([time.strftime("%Y-%m-%d %H:%M:%S"), project, person, target,
                                count["up"], count["down"], round(time.time() - start, 1)])

async def main():
    server = await asyncio.start_server(handle, HOST, PORT)
    async with server:
        await server.serve_forever()

asyncio.run(main())

A scraper then needs only its label, for example proxies={"https": "http://price-monitoring.alice:[email protected]:8899"} in Python requests. What it does and does not do:

• HTTPS stays encrypted. The meter relays the CONNECT tunnel as bytes and never decrypts it, as ProxyField's gateway does not, so it counts every byte on the wire, TLS handshakes included, but cannot see URLs inside the tunnel; the target host and port are in the CSV.
• The budget is checked when each connection opens, so a long-lived connection can run past it by its own size.
• Plain http:// targets are refused (they arrive without CONNECT); send them straight to gate.proxyfield.io:7000 or extend the handler.
• The label is not a password. Bind the meter to an internal address and allow port 8899 only from your scraper hosts.

A decrypting proxy such as mitmproxy can log full URLs and status codes per request instead, but every client must then trust its certificate authority.

What 1,000 pages cost

The tier rate applies to the whole purchase, so the team's combined purchase sets the rate every project pays. To size a project before you buy, measure 10-20 real pages as below and multiply by your monthly requests, with a margin for retries. Account questions on billing and access: help.

What uses traffic is everything that passes through the proxy: HTML, images, scripts, fonts, video, plus retries and redirects. A page that weighs 3 MB in a browser costs about 3 MB through the proxy. In a headless browser, the biggest saving is to block what you don't need, and Playwright reports the bytes it received:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={"server": "http://gate.proxyfield.io:7000",
                                       "username": "<login>-country-us-city-new_york", "password": "<password>"})
    page = browser.new_page()
    page.route("**/*", lambda route: route.abort()
               if route.request.resource_type in ("image", "media", "font") else route.continue_())
    finished = []
    page.on("requestfinished", finished.append)
    page.goto("https://example.com/")
    print(sum(req.sizes()["responseBodySize"] for req in finished), "body bytes received")
    browser.close()

request.sizes() is part of Playwright's API (Playwright: Request.sizes). More savings on scraping projects: Playwright with proxies.

One ProxyField account works everywhere at once. The gateway allows unlimited concurrent threads, so the same login can run in a scraper, a browser, an antidetect browser and a phone at the same time, and all of them draw on one balance.

• With a username and password: paste the same credentials into every tool. Change the targeting per tool in the username, as in <login>-country-us-city-new_york for a New York IP.
• With the IP allowlist: add the public IP of each place you connect from, and connect to the port you set up in the dashboard, where the location and session are set. Devices behind one home router share one public IP, so one entry covers them all; a laptop that moves between networks needs the login instead.
• Per site or task: use rotation for public pages and a sticky session per logged-in account. Managing your own or your clients' accounts through ProxyField is allowed, one sticky or ISP IP per account.

Device and tool steps are in how to set up a proxy, and credentials in proxy authentication.

• Name sessions by account. Give each logged-in account its own sticky session, so it keeps its IP for the time you set (30 s to 36 h); use a dedicated ISP IP when an account needs one IP for weeks. The full line is gate.proxyfield.io:7000:<login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600:<password>: -session-<id> names the session and -ttl-<seconds> sets its time, 30 to 129,600 seconds, 20 minutes by default. Every part after the login is optional.
• Keep credentials in one place, a secret store or environment variables, so a password change is one edit.
• HTTP, HTTPS and SOCKS5 all run on gate.proxyfield.io:7000 with the same username and password, so one stored credential serves every tool.

The proxy rarely fails alone; the target site pushes back. Log these per target site, not just in total (the codes below are their standard HTTP meanings):

Retry each failed request a few times on a fresh IP, with growing delays, and stop retrying a target that blocks every IP; ProxyField's network already moves traffic to the next node if one slows down. If you use several providers, run the same checks with a provider label so you can compare them on your own targets. The full list of errors is on all proxy and HTTP errors.

A home-made rotator over a list of IPs is simple to start and hard to keep healthy: you track bans, cool-downs, locations and retries yourself. A rotating gateway moves that work to the provider and leaves you with one address. Desktop proxy managers and browser proxy switchers help on the user side, switching profiles in one click. If you would rather pay per page than per GB, ProxyField's Web Scraping API is in development: it is designed to bill successful results only, $0.50 per 1,000 over plain HTTP; see the API documentation and Contact sales.

• Traffic never expires and there is no subscription: buy what you need and spend it over months.
• The tier rate applies to the whole purchase, so one larger purchase costs less per GB than many small ones: $2.95/GB for 1-9 GB, $2.49 from 100 GB, $1.95 from 1,000 GB. From 1 TB, the price is individual: contact sales. See pricing.
• All purchases are final, so measure a sample of your pages first; the Easy Trial (1 GB for $2.95) is enough to estimate a project.