HTTP Headers for Web Scraping

A desktop Chrome sends about a dozen headers on a page load. These are the ones sites look at most, with what a browser sends:

The client hints matter because Chromium browsers now send a reduced User-Agent and move the details into Sec-CH-UA-* headers (MDN: User-Agent reduction; MDN: Client hints). A request that claims Chrome 156 in User-Agent but has no Sec-CH-UA is a mismatch a site can test for. Firefox and Safari don't send client hints, so a Firefox User-Agent without them is consistent.

What not to send: headers a browser never sends on a page load, such as X-Requested-With on a navigation, a Content-Type on a GET, or a header with your tool's name.

The easiest consistent set is the one your own browser sends:

1. Open the target page in Chrome or Firefox, open DevTools and go to the Network tab.
2. Reload, click the first document request, and read Request Headers.
3. Right-click it and choose Copy as cURL to get every header in one command.
4. Keep User-Agent, Accept, Accept-Language, Accept-Encoding, the Sec-* headers and Upgrade-Insecure-Requests. Drop Cookie unless you mean to reuse that session, and drop Content-Length and Host, which your client sets.

Refresh the set when your browser updates; the version in User-Agent and Sec-CH-UA moves every few weeks. Current strings for each browser are in our list of user agents for scraping.

requests sends User-Agent: python-requests/<version> unless you replace it (requests source, default_user_agent). Set the headers on a Session, so cookies and connections carry over:

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:158.0) Gecko/20100101 Firefox/158.0",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
    "Accept-Encoding": "gzip, deflate",
    "Upgrade-Insecure-Requests": "1",
}
proxy = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"

s = requests.Session()
s.headers.update(headers)
r = s.get("https://httpbin.org/headers", proxies={"http": proxy, "https": proxy}, timeout=30)
print(r.json()["headers"])  # what the site received

The example claims Firefox and leaves out br and zstd, which plain requests can't decode without extra packages. Even with perfect headers, requests has its own TLS fingerprint, which some anti-bot systems check; that is covered in TLS and TCP fingerprinting. Proxy setup in Python is on Python requests with proxies.

curl sends curl/VERSION by default; -A replaces it and -H adds any header (curl man page). wget sends Wget/VERSION; --user-agent and --header do the same (GNU Wget manual).

curl -x gate.proxyfield.io:7000 -U "<login>-country-us-city-new_york:<password>" \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36" \
  -H "Accept-Language: en-US,en;q=0.9" \
  https://httpbin.org/headers

wget --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:158.0) Gecko/20100101 Firefox/158.0" \
  --header="Accept-Language: en-US,en;q=0.5" -O - https://httpbin.org/headers

More on each tool: cURL with proxies, wget with proxies.

A real browser already sends browser headers, so in Selenium you usually change only the user agent, and only to match the proxy's location or a device. In Chrome, pass it as a launch argument; other headers go through the Chrome DevTools Protocol:

from selenium import webdriver

opts = webdriver.ChromeOptions()
opts.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36")
driver = webdriver.Chrome(options=opts)

driver.execute_cdp_cmd("Network.enable", {})
driver.execute_cdp_cmd("Network.setExtraHTTPHeaders", {"headers": {"Accept-Language": "de-DE,de;q=0.9"}})
driver.get("https://httpbin.org/headers")
print(driver.find_element("tag name", "body").text)
driver.quit()

In Firefox, set the preference general.useragent.override on FirefoxOptions. Keep the version in the string equal to the browser you run; a Chrome 120 string on Chrome 156 is a mismatch. Proxy setup for Selenium: Selenium with proxies.

Puppeteer's current form is page.setUserAgent({ userAgent }); the plain string form is marked obsolete in the Puppeteer 25 docs (Puppeteer: Page.setUserAgent). Extra headers go through page.setExtraHTTPHeaders:

import puppeteer from "puppeteer";

const browser = await puppeteer.launch({ args: ["--proxy-server=http://gate.proxyfield.io:7000"] });
const page = await browser.newPage();
await page.authenticate({ username: "<login>-country-us-city-new_york", password: "<password>" });
await page.setUserAgent({ userAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36" });
await page.setExtraHTTPHeaders({ "Accept-Language": "en-US,en;q=0.9" });
await page.goto("https://httpbin.org/headers");
console.log(await page.evaluate(() => document.body.innerText));
await browser.close();

Headless Chrome's default user agent contains HeadlessChrome, which is why many scrapers set it. More: Puppeteer with proxies.

Playwright sets these per browser context: user_agent, locale (which also sets Accept-Language) and extra_http_headers (Playwright: Browser.new_context):

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={"server": "http://gate.proxyfield.io:7000",
                                       "username": "<login>-country-us-city-new_york", "password": "<password>"})
    context = browser.new_context(
        user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
        locale="en-US",
        timezone_id="America/New_York",
        extra_http_headers={"Referer": "https://example.com/"},
    )
    page = context.new_page()
    page.goto("https://httpbin.org/headers")
    print(page.inner_text("body"))
    browser.close()

Setting locale and timezone_id to the proxy's country keeps the headers and the JavaScript side in agreement. More: Playwright with proxies.

Scrapy sends Scrapy/VERSION (+https://scrapy.org) unless you change the USER_AGENT setting, and its DEFAULT_REQUEST_HEADERS default to an Accept value and Accept-Language: en (Scrapy settings). Set both in settings.py:

# settings.py
USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:158.0) Gecko/20100101 Firefox/158.0"
DEFAULT_REQUEST_HEADERS = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
}

For one request, pass headers={...} to scrapy.Request. Proxies in Scrapy: Scrapy with proxies.

Postman sends User-Agent: PostmanRuntime/<version> as an auto-generated header. To change it, add a User-Agent key on the request's Headers tab; your value replaces the generated one. Postman proxy setup: Postman with proxies.

Browsers send headers in a fixed order, and some anti-bot systems compare the order with the one the claimed browser uses. Most HTTP libraries keep the order of the dictionary you pass but add their own headers first or last, so the result is rarely a browser's order. Over HTTP/2, header names are lowercase and the request starts with pseudo-headers (:method, :authority, :scheme, :path), whose order also differs by client. You can't fix that with headers alone; a real browser, or a client that imitates one at the TLS and HTTP/2 level, sends it right.

Rotate only within strings that are real and current, and keep each one for a whole session. A new random user agent on every request from the same IP and cookie looks less like a person, not more. What helps:

• A small set of current strings from the browsers you can imitate fully (the matching Sec-CH-UA included).
• One user agent per session or per sticky IP, so the IP, cookies and headers stay together.
• No outdated versions. A Chrome version from two years ago is rarer than a bot.

The fake-useragent Python package, a common source of random strings, was archived in March 2026 (GitHub), so a hand-kept list from your own browsers is the safer source. Why the user agent matters at all is explained in what is a user agent.