Scraping a Website Behind a Login With Python

Open the login page, open DevTools (F12), go to the Network tab, tick "Preserve log", and log in by hand. The requests tell you which of three patterns you have:

Write down the form's field names (in the Payload tab of the POST), including hidden ones, and the URL the browser lands on after a successful login. Cookies and how sessions work in general, outside Python, are covered in scraping behind a login.

pip install requests beautifulsoup4 lxml

import os

import requests
from bs4 import BeautifulSoup

BASE = "https://example.com"
s = requests.Session()
s.headers["User-Agent"] = "Mozilla/5.0"

# 1. Open the login page: it sets the first cookie and holds the CSRF token
form_page = s.get(f"{BASE}/login", timeout=30)
soup = BeautifulSoup(form_page.text, "lxml")
token = soup.select_one('input[name="csrf_token"]')["value"]

# 2. Post the form with the same session, so the cookies go back
payload = {
    "csrf_token": token,
    "username": os.environ["SITE_USER"],
    "password": os.environ["SITE_PASS"],
}
r = s.post(f"{BASE}/login", data=payload, timeout=30)

# 3. Check that you are logged in before you scrape
if r.url.endswith("/login") or "Invalid username or password" in r.text:
    raise SystemExit("Login failed")

# 4. Scrape pages behind the login with the same session
account = s.get(f"{BASE}/account", timeout=30)
orders = [td.get_text(strip=True) for td in BeautifulSoup(account.text, "lxml").select("td.order")]

Replace the URL, the field names and the selectors with the ones you saw in DevTools. We ran this code on 2026-10-23 against a small local test server (not published) that works like a typical form login (a hidden CSRF token, a session cookie, a redirect to /account): it logged in and returned the protected rows. Posting the same form without the token got 403 CSRF token missing or wrong, which is the most common reason a Python login fails while the browser login works.

What each step is for:

• The session object stores every cookie the site sets and sends them back on later requests. Separate requests.get() calls start from nothing each time, so the site never sees you as logged in.
• The CSRF token is a one-time value the site puts in a hidden form field and checks on submit. It is tied to the cookie from the same page load, so fetch the form and post it in the same session. Some sites put the token in a <meta name="csrf-token"> tag or a cookie and expect it back in a header such as X-CSRF-Token.
• The login check matters because many sites answer a failed login with status 200 and the login form again. Test for something that only appears when you are logged in, or for a redirect away from the login URL.
• Credentials come from environment variables, not from the script, so they don't end up in version control.

If the form needs more hidden fields, collect them all: {i["name"]: i.get("value", "") for i in soup.select("form input[name]")}, then set the username and password on top.

Many modern sites log in through a JSON API and then send a token with each request. Reproduce that:

auth = requests.post(f"{BASE}/api/token",
                     json={"username": os.environ["SITE_USER"], "password": os.environ["SITE_PASS"]}, timeout=30)
auth.raise_for_status()
headers = {"Authorization": f"Bearer {auth.json()['access_token']}"}
orders = requests.get(f"{BASE}/api/orders", headers=headers, timeout=30).json()

The field names (access_token, token, jwt) and the header differ by site; copy them from DevTools. Tokens expire, often within an hour, so check for a 401 and log in again, or use the refresh token the API gives you. If the site offers an official API with keys for your account, use that instead of the web login; it is more stable and usually allowed by the terms.

If the login page builds its form with JavaScript, computes values in scripts, redirects through another domain or asks for a code, let a real browser do the login. Playwright can save the resulting session to a file, so you log in once and reuse it:

import os
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()

    # Log in once and save the cookies and local storage
    context = browser.new_context()
    page = context.new_page()
    page.goto("https://example.com/login")
    page.fill('input[name="username"]', os.environ["SITE_USER"])
    page.fill('input[name="password"]', os.environ["SITE_PASS"])
    page.click("button[type=submit]")
    page.wait_for_url("**/account")
    context.storage_state(path="state.json")
    context.close()

    # Later runs: start already logged in
    context = browser.new_context(storage_state="state.json")
    page = context.new_page()
    page.goto("https://example.com/account")
    rows = page.locator("td.order").all_inner_texts()

On our test server this logged in, saved state.json with the session cookie, and opened the account page in a new context without logging in again. storage_state keeps cookies and local storage; treat the file like a password.

For a two-factor login, run the first login with a visible browser (launch(headless=False)), enter the code yourself, and save the state; later runs reuse it until the session expires. The same flow in Selenium is driver.get, find_element(...).send_keys(...), then driver.get_cookies() to save the cookies. More about the tools in Python dynamic web scraping.

Hand the browser's cookies to Requests

A browser is slow. Log in with Playwright, then copy its cookies into a Requests session and scrape the pages over plain HTTP:

import json
import requests

state = json.load(open("state.json"))
s = requests.Session()
for c in state["cookies"]:
    s.cookies.set(c["name"], c["value"], domain=c["domain"], path=c["path"])
html = s.get("https://example.com/account", timeout=30).text

This worked on our test server. Some sites also check that the User-Agent and IP match the ones used at login, so send the browser's User-Agent and keep the same IP.

• Detect logouts. Sessions expire after inactivity or a fixed time. Check each response for the login form or a redirect to /login, and log in again when you see it.
• Don't log in on every run. Reuse the saved cookies or state file and log in only when they stop working. Repeated logins are a common trigger for security checks and lockouts.
• Go slowly. A logged-in account is easy to identify, so its request rate is visible to the site. Keep delays between requests and limit concurrency to one or two per account.
• Same IP for the whole session. A session cookie that suddenly arrives from a different IP or country can be ended by the site or flagged for a security check.

Logging in usually means you accepted the site's terms, and many terms forbid automated access; breaking them can end your account or lead to a contract claim. Getting into areas you are not authorized to see raises computer-misuse law (the CFAA in the US, the Computer Misuse Act in the UK). ProxyField's Acceptable Use Policy does not allow using the service to access data behind logins you are not entitled to; managing your own or your clients' social media and marketplace accounts (one IP per account) is allowed. Some site categories are restricted to protect the network, including financial and government services. Scraping behind a login is allowed only for data you are entitled to and the site's terms allow. Scrape your own data, data you have permission for, or data the terms allow, and keep personal data you collect within privacy law. The details, with court cases, are in is web scraping legal?. This is not legal advice.

Scraping your own account data? ProxyField residential proxies: sticky sessions from 30 s to 36 h, 412M+ residential IPs in 195+ countries, targeting by country, region (state) and city, from $1.95/GB with no subscription; start with the Easy Trial (1 GB). Traffic never expires, and all purchases are final.