How to Build a Web Scraper

A web scraper is a program that does three things in a loop: it fetches a page, extracts the fields you want from it, and stores them. Everything else (pagination, retries, proxies, scheduling) is there to keep that loop running on more pages without breaking.

The tutorial uses two pages of scrapethissite.com, a sandbox built for scraping practice: a list of countries on one page and a table of hockey teams spread over many pages. Its robots.txt blocks only the lesson and FAQ pages, and the selectors below were checked against the live HTML the same day.

Check two things before you write code: the site's robots.txt (at /robots.txt on the domain) and its terms of use. Scrape public pages only, never data behind a login you are not entitled to, and keep personal data out unless you have a lawful reason to collect it. For the details, see is web scraping legal and the guide to checking whether scraping is allowed.

Open the page in Chrome or Firefox, right-click the first value you want (a country name) and choose Inspect. The Elements panel jumps to its tag. Look for a class or attribute that repeats on every item:

<div class="col-md-4 country">
  <h3 class="country-name">Andorra</h3>
  <div class="country-info">
    <strong>Capital:</strong> <span class="country-capital">Andorra la Vella</span>
    <strong>Population:</strong> <span class="country-population">84000</span>
    <strong>Area (km2):</strong> <span class="country-area">468.0</span>
  </div>
</div>

Every country sits in a div.country, and each field has its own class. Those classes are your selectors: div.country, h3.country-name, span.country-capital.

Two checks save hours later:

Install Python 3.10 or newer, create a virtual environment and install two libraries. Requests 2.34.2 needs Python 3.10+, and Beautiful Soup 4.15.0 is MIT-licensed (PyPI, 2026-10-23).

python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install requests beautifulsoup4

Send a GET request with a timeout and a browser-like User-Agent, and stop if the status is not 200.

import requests

HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                         "(KHTML, like Gecko) Chrome/141.0 Safari/537.36"}

r = requests.get("https://www.scrapethissite.com/pages/simple/", headers=HEADERS, timeout=30)
r.raise_for_status()
html = r.text
print(len(html), "characters")

raise_for_status() turns a 403, 404 or 429 into an error, so you notice a block instead of parsing an error page.

Beautiful Soup turns the HTML into a tree; select() takes a CSS selector and returns every match, and select_one() returns the first.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
rows = []
for card in soup.select("div.country"):
    rows.append({
        "name": card.select_one("h3.country-name").get_text(strip=True),
        "capital": card.select_one("span.country-capital").get_text(strip=True),
        "population": int(card.select_one("span.country-population").get_text(strip=True)),
        "area_km2": float(card.select_one("span.country-area").get_text(strip=True)),
    })
print(len(rows), rows[0])

The page lists 250 countries. get_text(strip=True) drops the whitespace around the name, and the int() and float() calls turn text into numbers you can sort. For every way to find and select elements, see the Beautiful Soup web scraping tutorial.

Most lists are split into pages. On the hockey table, the page number is a URL parameter, ?page_num=2, so the scraper can count up until a page comes back with no rows. Where a site has a "Next" link instead, read its href and join it to the current URL with urllib.parse.urljoin.

import time

def scrape_teams(session):
    teams, page = [], 1
    while True:
        r = session.get("https://www.scrapethissite.com/pages/forms/",
                        params={"page_num": page}, timeout=30)
        r.raise_for_status()
        found = BeautifulSoup(r.text, "html.parser").select("tr.team")
        if not found:
            break
        for tr in found:
            teams.append({
                "team": tr.select_one("td.name").get_text(strip=True),
                "year": int(tr.select_one("td.year").get_text(strip=True)),
                "wins": int(tr.select_one("td.wins").get_text(strip=True)),
                "losses": int(tr.select_one("td.losses").get_text(strip=True)),
            })
        page += 1
        time.sleep(1)          # one request per second: polite and enough for a practice site
    return teams

A requests.Session reuses the connection between pages, which makes a long run faster.

Write the rows to CSV with the standard library; Excel and Google Sheets open the file as is.

import csv

def save_csv(rows, path):
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=rows[0].keys())
        writer.writeheader()
        writer.writerows(rows)

with requests.Session() as s:
    s.headers.update(HEADERS)
    save_csv(scrape_teams(s), "teams.csv")

For JSON, json.dump(rows, f, indent=2) does the same job. Once a scraper runs every day, a database (SQLite to start) makes it easy to keep only new or changed rows.

A scraper that runs once on your laptop and a scraper you trust every day differ in four habits:

• Retry with backoff. Retry timeouts and 429 or 5xx responses 3 times, waiting longer each time; give up on 403 and 404.
• Check what you parsed. If a selector returns None, the layout changed: log the URL and stop, rather than writing empty rows.
• Pace the requests. Keep a delay between requests to the same site and honor a Retry-After header. Speed comes from fetching several sites in parallel, not from hammering one; see how to make web scraping faster.
• Log every run. Pages fetched, rows written, errors by status code. When a number drops, you know where to look.

Sites limit how many requests one IP may send. A scraper that fetches a few pages from your home IP is fine; one that fetches thousands of pages a day from one site gets 429 and 403 responses, then a block. Rotating proxies spread the requests over many IPs, and a location in the username lets you see the page the way local visitors see it.

With Requests, the proxy is one argument. The ProxyField gateway is gate.proxyfield.io:7000; replace <login> and <password> with your credentials. <login>-country-us-city-new_york asks for an exit IP in New York, United States. Region, session and session-time options: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default) after the location, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600.

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"

with requests.Session() as s:
    s.headers.update(HEADERS)
    s.proxies.update({"http": PROXY, "https": PROXY})
    print(s.get("https://api.ipify.org", timeout=30).text)   # the exit IP the site sees
    save_csv(scrape_teams(s), "teams.csv")

The gateway can give each request a new IP, or keep one IP for a sticky session from 30 s to 36 h when a site needs the same visitor across pages. Full setup, SOCKS5 included: Python proxy setup. Which proxy type to pick, and how much traffic a job needs: proxies for web scraping.

If Step 1 showed the data only in the Elements panel, Requests will fetch an empty template. You have three ways out, in order of cost: call the JSON API the page uses, parse JSON embedded in a <script> tag, or render the page with Playwright or Selenium. The Python versions are in Python dynamic web scraping. If you would rather not run browsers at all, the ProxyField Web Scraping API fetches any URL with plain HTTP or with JavaScript rendering, billed on successful results only; ask for it through Contact sales.

The basics take a weekend if you already write a little code: the script above covers fetching, parsing, pagination and saving. What takes longer is the part this tutorial only touches: pages that need JavaScript, sites that block automated traffic, and keeping scrapers working as layouts change. Practice on a sandbox, then on one real site you care about; the web scraping projects list has ideas by level.

• Run it every day: how to schedule and deploy web scrapers.
• Follow links across a whole site: how to build a web crawler.
• Other languages: Ruby web scraping, web scraping in Rust.
• Images rather than text: how to scrape images from a website.

Ready to scrape past your home IP? ProxyField residential proxies: 412M+ residential IPs in 195+ countries, a new IP per request or sticky sessions up to 36 h, unlimited concurrent threads. Pay per GB, no subscription; start with the Easy Trial (1 GB). Traffic never expires, and all purchases are final.