Web Scraping With Python: Step-by-Step Tutorial

A scraper for books.toscrape.com, a practice site made for learning to scrape, that collects the title, price, rating and stock of every book across all catalog pages and writes them to books.csv. The same steps work for any site whose data is in the HTML.

What you need:

• Python 3.10 or newer (the current Requests, 2.34.2, requires it), a terminal, and 20 minutes. Versions on PyPI: Requests 2.34.2, Beautiful Soup 4.15.0.
• Enough HTML to read one card. This is one book on the page, trimmed:

<article class="product_pod">
  <p class="star-rating Three"> … </p>
  <h3><a href="catalogue/a-light-in-the-attic_1000/index.html" title="A Light in the Attic">A Light in the ...</a></h3>
  <p class="price_color">£51.77</p>
  <p class="instock availability"> In stock </p>
</article>

These are the 4 CSS selectors that read it, the only CSS this tutorial needs:

CSS for scraping means selecting elements, not styling pages. What else a first scraper needs, from Python basics to a proxy: web scraping requirements.

Create a virtual environment so the scraper's packages stay separate from the rest of your system, then install the two libraries.

python3 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install requests beautifulsoup4

Open the page in your browser, right-click a book title and choose Inspect. You will see each book in an <article class="product_pod">, the title in the title attribute of an <a> inside <h3>, the price in <p class="price_color"> and the rating as a class on <p class="star-rating Three">. Write these down: they become your selectors. This is also how you learn HTML for any other site: inspect the element that holds your data and read its tag and class.

Then press Ctrl+U (Cmd+Option+U on a Mac) to view the page source. If the data you want is in the source, Requests will get it. If it is not, the page builds its content with JavaScript, and you need the page's JSON requests or a browser (step 9).

import requests

HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; books-tutorial/1.0; [email protected])"}
r = requests.get("https://books.toscrape.com/", headers=HEADERS, timeout=30)
r.encoding = "utf-8"                 # the site sends no charset; without this "£" reads as "£"
print(r.status_code, len(r.text))   # 200 and the size of the HTML
r.raise_for_status()                 # stop on 4xx and 5xx errors

Always set a timeout, or a stuck request can hang your script forever. A descriptive User-Agent with a contact tells the site who you are; headers in more depth: web scraping headers.

from bs4 import BeautifulSoup

soup = BeautifulSoup(r.text, "html.parser")
print(soup.title.get_text(strip=True))
books = soup.select("article.product_pod")
print(len(books))                    # 20 books per page

select() takes CSS selectors and returns a list; select_one() returns the first match or None. Beautiful Soup's own guide covers every method: Beautiful Soup web scraping.

RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}

def parse_book(card):
    link = card.select_one("h3 a")
    rating = card.select_one("p.star-rating")["class"][1]
    return {
        "title": link["title"],
        "price": card.select_one("p.price_color").get_text(strip=True).lstrip("£"),
        "rating": RATINGS.get(rating),
        "in_stock": "In stock" in card.select_one("p.availability").get_text(),
        "url": requests.compat.urljoin(r.url, link["href"]),
    }

print(parse_book(books[0]))

Two habits save hours later: read attributes with tag["attr"] and text with get_text(strip=True), and turn relative links into absolute ones with urljoin. CSS selectors, XPath and extracting links and text in general: HTML scraping.

Each catalog page has a "next" link (li.next a) until the last one. Loop until it disappears, and pause between pages.

import time

url = "https://books.toscrape.com/"
rows = []
while url:
    r = requests.get(url, headers=HEADERS, timeout=30)
    r.raise_for_status()
    r.encoding = "utf-8"
    soup = BeautifulSoup(r.text, "html.parser")
    rows += [parse_book(c) for c in soup.select("article.product_pod")]
    nxt = soup.select_one("li.next a")
    url = requests.compat.urljoin(r.url, nxt["href"]) if nxt else None
    time.sleep(1)                    # be polite: one page per second
print(len(rows))                     # 1,000 books on 50 pages

Pages numbered with ?page=2 work the same way with a counter; infinite scroll usually loads a JSON endpoint you can call with a page or offset parameter.

import csv

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)

For JSON, json.dump(rows, f, ensure_ascii=False, indent=2); for analysis, pandas.DataFrame(rows). To open the result in a spreadsheet or to scrape into Excel directly, see web scraping in Excel.

Real sites fail now and then: timeouts, 429 Too Many Requests, 503 errors. Retry a few times with a growing pause, and slow down when the site tells you to.

from requests.adapters import HTTPAdapter, Retry
from urllib.robotparser import RobotFileParser

session = requests.Session()
session.headers.update(HEADERS)
session.mount("https://", HTTPAdapter(max_retries=Retry(
    total=4, backoff_factor=2, status_forcelist=[429, 500, 502, 503, 504], respect_retry_after_header=True)))

robots = RobotFileParser("https://books.toscrape.com/robots.txt")
robots.read()
print(robots.can_fetch(HEADERS["User-Agent"], "https://books.toscrape.com/catalogue/page-2.html"))   # True

books.toscrape.com has no robots.txt (it returns 404), which RFC 9309 treats as "no rules", so can_fetch returns True.

What you see when a request fails, and what the code above does about it:

robots.txt states which paths a site asks crawlers to avoid; it is a request, not access control, and the site's terms still apply (RFC 9309). How to read both before you scrape: how to check if scraping is allowed. What a 429 means and how to fix it: 429 Too Many Requests.

If the data is not in the page source, open the browser's developer tools, go to the Network tab, filter by Fetch/XHR and reload. Most modern sites load their data as JSON from an API endpoint, which Requests can call directly and which is easier to parse than HTML:

data = session.get("https://example.com/api/products?page=1", timeout=30).json()

If there is no such request, or it needs tokens the page computes, render the page in a browser with Playwright or Selenium. That is the subject of Python dynamic web scraping and web scraping with Selenium.

The steps above in one file, with the robots.txt check applied before every page:

# pip install requests beautifulsoup4   ->   python books.py
import csv, time
from urllib.robotparser import RobotFileParser
import requests
from bs4 import BeautifulSoup
from requests.adapters import HTTPAdapter, Retry

START = "https://books.toscrape.com/"                       # 1. your start URL
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; books-tutorial/1.0; [email protected])"}
RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
PROXY = None  # e.g. "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"

session = requests.Session()
session.headers.update(HEADERS)
session.mount("https://", HTTPAdapter(max_retries=Retry(total=4, backoff_factor=2,
              status_forcelist=[429, 500, 502, 503, 504], respect_retry_after_header=True)))
if PROXY:
    session.proxies = {"http": PROXY, "https": PROXY}

robots = RobotFileParser(requests.compat.urljoin(START, "/robots.txt"))
robots.read()


def parse_book(card, base):                                  # 3. your field selectors
    link = card.select_one("h3 a")
    return {"title": link["title"],
            "price": card.select_one("p.price_color").get_text(strip=True).lstrip("£"),
            "rating": RATINGS.get(card.select_one("p.star-rating")["class"][1]),
            "in_stock": "In stock" in card.select_one("p.availability").get_text(),
            "url": requests.compat.urljoin(base, link["href"])}


def scrape(start):
    url, rows = start, []
    while url and robots.can_fetch(HEADERS["User-Agent"], url):
        r = session.get(url, timeout=30)
        r.raise_for_status()
        r.encoding = "utf-8"             # the site sends no charset header
        soup = BeautifulSoup(r.text, "html.parser")
        rows += [parse_book(c, r.url) for c in soup.select("article.product_pod")]  # 2. card selector
        nxt = soup.select_one("li.next a")                   # 4. next-link selector
        url = requests.compat.urljoin(r.url, nxt["href"]) if nxt else None
        time.sleep(1)
    return rows


rows = scrape(START)
with open("books.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=rows[0].keys())
    w.writeheader()
    w.writerows(rows)
print(f"saved {len(rows)} books")

Run it with python books.py. Expected output, after about 80 seconds (50 pages at one per second):

saved 1000 books

and the first rows of books.csv:

title,price,rating,in_stock,url
A Light in the Attic,51.77,3,True,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet,53.74,1,True,https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
Soumission,50.10,1,True,https://books.toscrape.com/catalogue/soumission_998/index.html

Adapt it to your site

The code is marked where a different site needs changes. Open the site in the browser's Inspect panel (step 2) and replace:

1. START: the first page of the listing you want.
2. The card selector article.product_pod: the element that wraps one item.
3. The field selectors in parse_book (h3 a, p.price_color, p.star-rating, p.availability): one per field you need, and the dictionary keys, which become the CSV columns.
4. The next-link selector li.next a: the "Next" link or button; numbered pages, offsets and infinite scroll are covered in web scraping pagination.

If rows stays empty, the selectors do not match or the data loads with JavaScript (step 9).

• Thousands of pages from one site: move to Scrapy, which adds concurrency, retries, deduplication and exports (what Scrapy is), or fetch concurrently with httpx and asyncio.
• Running every day: schedule the script with cron or a job runner (schedule web scrapers).
• Many sites or a whole site: crawl with link discovery and limits (Python web crawler).
• Other libraries: the full comparison of clients, parsers, frameworks and browsers is in Python libraries for web scraping.

Sites cap how many requests one IP may send. When your scraper starts getting 429 or 403 responses at a polite pace, or when you need pages as they appear in another country, route the requests through rotating proxies. In the complete scraper, set PROXY to the ProxyField gateway:

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"

Each request then leaves from a new residential IP in New York. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional. SOCKS5, sticky sessions and rotation patterns are in the Python proxy setup guide. ProxyField residential proxies cost $2.95/GB for 1 GB and fall to $1.95/GB from 1,000 GB, with no subscription, unlimited concurrent threads and traffic that never expires; all purchases are final. Each catalog page in this tutorial is about 51 KB of HTML (Requests downloads no images), so the 50-page run moves about 2.5 MB. Scrape public data and follow each site's rules (is web scraping legal), then start with residential proxies and the Easy Trial: 1 GB for $2.95.