Scraping Behind a Login: Cookies and Sessions

Find out which one you have before you write code: log in by hand with DevTools open on the Network tab ("Preserve log" on) and look at the login request and the next one. The Python walkthrough of each type is in scraping a website behind a login with Python.

A cookie is a name and value the server asks the client to store and send back (MDN: Using HTTP cookies). The attributes in Set-Cookie decide where and how long:

• Domain and Path: which hosts and paths receive the cookie. A cookie set for .example.com also goes to shop.example.com.
• Expires or Max-Age: when the client deletes it. Without either, it is a session cookie, deleted when the browser closes; in a script it lives as long as your cookie jar does. This is not when the login ends: see below.
• HttpOnly: page scripts can't read it, but your HTTP client sends it like any other cookie.
• Secure: sent only over HTTPS.
• SameSite: limits cross-site sending in browsers; it rarely affects a scraper that talks to one site.

Login state is usually one or two of these cookies. Others hold preferences, cookie-banner choices, analytics IDs or anti-bot results. Send back everything the site set, in the same session, rather than picking cookies by hand.

A cookie jar stores what the server sets and replays it on the next request. Every HTTP tool has one; you only have to use it. We ran the curl, Python and Node.js examples on this page on 2026-10-27 (curl 8.5.0, Python 3.12 with requests 2.31.0, Node.js 22) against a local test server built like a typical form login with a CSRF token.

curl. -c file writes cookies, -b file sends them. A complete form login with a CSRF token:

BASE=https://example.com
TOKEN=$(curl -s -c cookies.txt "$BASE/login" | grep -o 'name="csrf_token" value="[^"]*"' | sed 's/.*value="//; s/"$//')
curl -s -b cookies.txt -c cookies.txt -d "csrf_token=$TOKEN&username=$SITE_USER&password=$SITE_PASS" "$BASE/login"
curl -s -b cookies.txt "$BASE/account"

On our test server the second call got a 302 redirect to /account with a session cookie, and the third returned the protected page. curl saves cookies in the Netscape format and writes HttpOnly cookies on lines that start with #HttpOnly_. They look like comments but are real cookies, so don't strip lines starting with # from the file. More curl options are on the curl proxy setup page.

Python. requests.Session() is the cookie jar: everything the site sets on one request goes back on the next (Requests: session objects). The full login-and-recover loop is in the section on expired sessions below.

Node.js. The built-in fetch keeps no cookies between calls. Wrap it with fetch-cookie (3.2.0, the current version on npm on 2026-10-27; it uses tough-cookie for the jar):

import makeFetchCookie from "fetch-cookie";

const fetchCookie = makeFetchCookie(fetch);            // fetch with a cookie jar
const html = await (await fetchCookie("https://example.com/login")).text();  // sets the first cookie
const token = html.match(/name="csrf_token" value="([^"]+)"/)[1];
await fetchCookie("https://example.com/login", {        // post the form; the redirect keeps the cookie
  method: "POST",
  body: new URLSearchParams({ csrf_token: token, username: process.env.SITE_USER, password: process.env.SITE_PASS }),
});
const account = await (await fetchCookie("https://example.com/account")).text();

This flow logged in and returned the protected page on our test server. Without the library, collect response.headers.getSetCookie() from each response and send the values back in a cookie header, with redirect: "manual" on the login POST so you can catch the cookie set on the redirect.

Browsers. Log in once with a browser, save its state to a file, and start the next run already logged in. In Playwright for Python, through a sticky ProxyField session (Playwright: authentication):

import os
from playwright.sync_api import sync_playwright

PROXY = {"server": "http://gate.proxyfield.io:7000",
         "username": "<login>-country-us-city-new_york-session-acct1-ttl-7200",
         "password": "<password>"}

with sync_playwright() as p:
    browser = p.chromium.launch(proxy=PROXY)
    if os.path.exists("auth/state.json"):                       # later runs: start logged in
        context = browser.new_context(storage_state="auth/state.json")
    else:                                                       # first run: log in and save
        context = browser.new_context()
        page = context.new_page()
        page.goto("https://example.com/login")
        page.fill("input[name=username]", os.environ["SITE_USER"])
        page.fill("input[name=password]", os.environ["SITE_PASS"])
        page.click("button[type=submit]")
        page.wait_for_url("**/account")
        os.makedirs("auth", exist_ok=True)
        context.storage_state(path="auth/state.json")           # cookies and local storage
    page = context.new_page()
    page.goto("https://example.com/account")
    browser.close()

The state file holds the cookies and local storage; recent Playwright versions can add IndexedDB with indexed_db=True. Session storage is not saved: the Playwright docs show how to copy it with page.evaluate and restore it with an init script. The file works like your password, so add auth/ to .gitignore and never print it in logs. Selenium has get_cookies() and add_cookie() for the same job. Log in with the browser, then pass its cookies to a plain HTTP client for the scraping itself; that is much faster. Proxy settings for Playwright are on the Playwright proxy setup page.

For a quick job on your own account, you can skip the scripted login: log in in your browser, open DevTools > Network, click any request to the site and copy the value of its Cookie request header. Send it as a header from your script:

curl -s -H "Cookie: session=PASTE_VALUE_HERE" https://example.com/account

Treat that value like your password: anyone who has it is logged in as you until the session ends. It also expires when you log out in the browser or the site ends the session. Use this for your own account only.

Sites tie a session to more than the cookie. A session cookie that arrives from a new IP, especially in another city or country, can be ended or sent to a security check. So for authenticated scraping:

• Use a sticky session, not per-request rotation. Rotation gives each request a new IP, which looks like the account hopping around the world.
• One IP per account. If you work with several accounts you are entitled to use, give each one its own sticky session id and keep it.
• Match the location you normally log in from (country, region or city).
• Make the session at least as long as the job, and plan for the moments the IP can still change (the table below).

With ProxyField, the sticky session lasts from 30 s to 36 h (ttl from 30 to 129,600 seconds), with presets of 1, 5, 10 and 30 minutes and 1, 3 and 6 hours. The gateway is gate.proxyfield.io:7000, and <login>-country-us-city-new_york in the username asks for an exit IP in New York. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds>, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.

curl -x gate.proxyfield.io:7000 -U "<login>-country-us-city-new_york-session-acct1-ttl-7200:<password>" -b cookies.txt -c cookies.txt https://example.com/account

The IP is held while its exit node is up. Whenever it changes, a careful site may end the login, which is one more reason to check every response, as below. How the two proxy modes differ is in sticky vs rotating proxies; the authentication options of the proxy itself (username and password or an IP allowlist) are in proxy authentication.

A logged-out scraper keeps running and collects login pages instead of data. Check every response for one of these signs:

• a redirect to the login URL, or a final URL that contains /login or /signin;
• status 401 or 403 from an API;
• the login form (a password field) or a "Sign in" link in the HTML where your account name used to be;
• a page much shorter than usual.

The cookie's expiry date is not the end of the session. Expires and Max-Age only tell the client when to delete the cookie. The server keeps its own session and can end it earlier: after a logout, a password change, an idle timeout, an absolute timeout or a security event (OWASP Session Management Cheat Sheet). Editing the date in your cookie jar changes nothing on the server, so don't do it. If the site issues a refresh token (common with bearer tokens), use its refresh flow instead of a full login.

What to do when a sign appears:

1. Stop and checkpoint. Save the last record you processed, so the run can resume there.
2. Log in again once, through the normal login flow, and keep the new cookies in the same jar.
3. Retry the failed request once. Retry only reads (GET). Don't repeat an action that changes something, such as a form submission or an order, unless you can check whether it already went through.
4. Stop and alert on a second failure. A failed re-login usually means a wrong password, a locked account or a new check.
5. Stop for a person on MFA or a CAPTCHA. A second factor, a CAPTCHA or a consent screen needs a human; don't try to get past it.

In Python with requests:

import json, os, re, requests

BASE = "https://example.com"
PROXY = "http://<login>-country-us-city-new_york-session-acct1-ttl-7200:<password>@gate.proxyfield.io:7000"
CHECKPOINT = "checkpoint.json"

class SessionLost(Exception):
    pass

def logged_out(r):
    return ("/login" in r.url or r.status_code in (401, 403)
            or 'type="password"' in r.text)

def login(s):
    html = s.get(f"{BASE}/login", timeout=30).text
    token = re.search(r'name="csrf_token" value="([^"]+)"', html).group(1)
    r = s.post(f"{BASE}/login", timeout=30, data={"csrf_token": token,
               "username": os.environ["SITE_USER"], "password": os.environ["SITE_PASS"]})
    if logged_out(r):
        raise SessionLost("login failed: check the password, or a second factor or check page")

def get(s, url):
    r = s.get(url, timeout=30)
    if not logged_out(r):
        return r
    login(s)                      # the session ended: log in once
    r = s.get(url, timeout=30)    # and retry this request once
    if logged_out(r):
        raise SessionLost(url)    # second failure: stop and alert
    return r

s = requests.Session()
s.proxies = {"http": PROXY, "https": PROXY}
start = json.load(open(CHECKPOINT))["next"] if os.path.exists(CHECKPOINT) else 1
login(s)
for page in range(start, 21):
    html = get(s, f"{BASE}/items?page={page}").text
    # ... parse and store html ...
    json.dump({"next": page + 1}, open(CHECKPOINT, "w"))   # resume point

We ran it on 2026-10-27, without the proxy, against a test server that ends each session after 5 requests although its cookie says Max-Age=3600. The loop logged in 5 times (once at the start, then 4 times after the server ended the session) and collected all 20 pages, and the checkpoint ended at page 21. With a wrong password it stopped at once with SessionLost: login failed. Add the fourth sign (a page much shorter than usual) once you know the normal size of your pages.

Log in again only when a sign appears, not at the start of every request or run. Frequent logins from a script are more likely to trigger security checks, emails to the account owner or lockouts than a session that is reused calmly.

A paywall is a login with a subscription behind it. Scraping articles you pay for, for your own reading or analysis, is a different matter from republishing them: the content is still protected by copyright (see is web scraping legal?), and subscription terms usually forbid automated downloading and sharing. Don't get around a paywall you haven't paid for; tricks that hide the paywall without logging in still mean taking content the publisher only sells. If you need news or members-only data at scale, ask the publisher about a license or an official feed.

Authentication changes the legal picture. Pages anyone can open are treated differently from areas behind a password:

• Computer-misuse law. In the US, the Supreme Court read the Computer Fraud and Abuse Act in Van Buren v. United States (June 3, 2021) as asking whether you entered parts of a system that are off limits to you (opinion). Using someone else's login, or getting into areas your account can't see, is where that risk sits. In the UK, section 1 of the Computer Misuse Act 1990 makes unauthorized access an offense.
• Contract. Creating an account means accepting the site's terms. In the hiQ Labs v. LinkedIn case, the district court ruled in November 2022 that hiQ had breached LinkedIn's User Agreement, and in December 2022 the case ended in a consent judgment against hiQ (docket).
• Privacy. Logged-in pages often show other people's personal data; in the EU and UK, the GDPR applies to collecting it.
• ProxyField's Acceptable Use Policy does not allow using the service to access data behind logins you are not entitled to. Managing your own or your clients' social media and marketplace accounts, one IP per account, is allowed. Some site categories are restricted to protect the network, including financial and government services. Scraping behind a login is allowed only for data you are entitled to and the site's terms allow.

The full picture, with more cases, is in is web scraping legal?. This is not legal advice.

Logged-in scraping on your own accounts? ProxyField residential proxies: sticky sessions from 30 s to 36 h, 412M+ residential IPs in 195+ countries, targeting by country, region (state) and city, from $1.95/GB with no subscription; start with the Easy Trial (1 GB). Traffic never expires, and all purchases are final.