BeautifulSoup Web Scraping Tutorial

Beautiful Soup (the beautifulsoup4 package, version 4.15.0, MIT license, per PyPI on 2026-10-23) parses HTML and XML into Python objects and gives you ways to search and walk them. Its documentation is the reference for every method below.

pip install beautifulsoup4 lxml requests

Versions on PyPI on 2026-10-27: beautifulsoup4 4.15.0 (MIT), lxml 6.1.3 (BSD-3-Clause), soupsieve 3.0.4 (MIT, installed with Beautiful Soup), requests 2.34.2 (Apache-2.0).

The target is a practice sandbox: 250 countries, each a card like this:

<div class="col-md-4 country">
  <h3 class="country-name"><i class="flag-icon flag-icon-ad"></i> Andorra</h3>
  <div class="country-info">
    <strong>Capital:</strong> <span class="country-capital">Andorra la Vella</span><br/>
    <strong>Population:</strong> <span class="country-population">84000</span><br/>
    <strong>Area (km<sup>2</sup>):</strong> <span class="country-area">468.0</span><br/>
  </div>
</div>

The script loops over every card twice, once with find_all() and once with CSS selectors, and writes the rows to CSV:

import csv
import requests
from bs4 import BeautifulSoup

url = "https://www.scrapethissite.com/pages/simple/"
response = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=30)
response.raise_for_status()                      # stop on 403, 404 or 5xx instead of parsing an error page
soup = BeautifulSoup(response.text, "lxml")

def text(el):
    return el.get_text(" ", strip=True) if el else None   # None instead of AttributeError

# 1. find_all() for the cards, find() for each field
rows_find = []
for card in soup.find_all("div", class_="country"):
    rows_find.append({
        "name": text(card.find("h3", class_="country-name")),
        "capital": text(card.find("span", class_="country-capital")),
        "population": text(card.find("span", class_="country-population")),
        "area_km2": text(card.find("span", class_="country-area")),
    })

# 2. the same fields with CSS selectors
rows_css = []
for card in soup.select("div.country"):
    rows_css.append({
        "name": text(card.select_one("h3.country-name")),
        "capital": text(card.select_one("span.country-capital")),
        "population": text(card.select_one("span.country-population")),
        "area_km2": text(card.select_one("span.country-area")),
    })

print(len(rows_find), rows_find == rows_css)    # 250 True
print(rows_find[0])

with open("countries.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=rows_find[0].keys())
    writer.writeheader()
    writer.writerows(rows_find)

What it printed when we ran it on 2026-10-27 (Python 3, Beautiful Soup 4.15.0):

250 True
{'name': 'Andorra', 'capital': 'Andorra la Vella', 'population': '84000', 'area_km2': '468.0'}

and countries.csv starts:

name,capital,population,area_km2
Andorra,Andorra la Vella,84000,468.0
United Arab Emirates,Abu Dhabi,4975593,82880.0

Why each line is there:

• raise_for_status() turns a blocked or missing page into an HTTPError. Without it, a 403 page parses into a soup with no cards and the script writes an empty file without complaint.
• class_="country" matches a tag whose class list contains country, so col-md-4 country matches; the underscore is there because class is a Python keyword.
• The text() guard returns None for a missing field instead of raising AttributeError: 'NoneType' object has no attribute 'get_text', so one odd card doesn't stop 249 good ones.
• get_text(" ", strip=True) drops the whitespace around "Andorra" that the flag icon leaves.

Both methods return the same 250 rows. Pick the one you read more easily: find_all() takes Python arguments (functions, regexes, limit), select() takes the selector you already tested in DevTools.

The second argument to BeautifulSoup() picks the parser. They differ in speed and in how they repair broken HTML, so the same messy page can produce slightly different trees.

Always name the parser. If you don't, Beautiful Soup picks whichever installed parser it ranks first and warns you, and the script may behave differently on another machine.

find() returns the first matching element or None; find_all() returns a list of every match. Both take a tag name, attributes, a class, text or a function.

soup.find("h3", class_="country-name")           # first country name
soup.find_all("span", class_="country-capital")   # every capital
soup.find_all("div", attrs={"class": "country"}, limit=10)
soup.find("a", href=True)                         # first link that has an href

import re
soup.find_all(string=re.compile("Andorra"))      # text nodes that contain "Andorra"
soup.find_all("span", class_=re.compile(r"^country-"))   # regex on the class

Three traps:

• string= tests a tag's .string, which is None when the tag holds more than one child. On this page soup.find("h3", string=re.compile("Andorra")) returns None, because the <h3> holds an <i> icon as well as the text; search the text nodes (soup.find_all(string=...)) or use the CSS :-soup-contains("Andorra") instead.
• The keyword is class_ (with an underscore; supported since Beautiful Soup 4.1.2).
• find() returns None when nothing matches, so soup.find("h9").text raises AttributeError: 'NoneType' object has no attribute 'text'. Check for None before you read the value.

The full argument list is find_all(name, attrs, recursive, string, limit, **kwargs); the official docs explain each one under Searching the tree.

If you already know CSS, select() is usually shorter. It returns a list; select_one() returns the first match or None.

soup.select("div.country h3.country-name")          # descendant
soup.select("div.country > div.country-info")       # direct child
soup.select('meta[property="og:title"]')            # attribute equals
soup.select('a[href^="https://"]')                  # attribute starts with
soup.select("tr:nth-of-type(2) td")                 # position
soup.select_one("span.country-population")

The selector engine is Soup Sieve, part of Beautiful Soup since 4.7.0; it supports CSS selectors through Level 4, including :is(), :not() and :has() (Soup Sieve docs). Since 4.12.0 the same engine is also reachable as soup.css.select(...); the official section is CSS selectors through the .css property.

find_all() and select() equivalents

Each pair below returned the same elements on the sandbox page on 2026-10-27:

from urllib.parse import urljoin

card = soup.select_one("div.country")
name = card.select_one("h3.country-name").get_text(strip=True)    # "Andorra"
info = card.select_one("div.country-info").get_text(" ", strip=True)
link = soup.select_one("a[href]")
href = urljoin(url, link["href"])          # absolute URL
alt = soup.select_one("img").get("alt")    # None instead of KeyError if missing

• get_text(strip=True) removes surrounding whitespace; get_text(" ", strip=True) joins nested pieces with a space.
• tag["href"] raises KeyError when the attribute is missing; tag.get("href") returns None.
• Relative links (/page/2) need urljoin with the page URL before you fetch them.

When the value has no class of its own, start from something nearby and walk:

label = soup.find("strong", string="Population:")
value = label.find_next_sibling("span").get_text(strip=True)   # the span after the label
row = soup.find("td", class_="name").find_parent("tr")          # the whole table row

.parent, .find_parent(), .next_sibling, .find_next_sibling() and .find_next() cover most cases. Note that .next_sibling can be a whitespace string; the find_* versions skip to the next tag.

from urllib.parse import urljoin

page = "https://www.scrapethissite.com/pages/forms/"
r = requests.get(page, timeout=30)
r.raise_for_status()
soup = BeautifulSoup(r.text, "lxml")
headers = [th.get_text(strip=True) for th in soup.select("table th")]
rows = [[td.get_text(strip=True) for td in tr.select("td")] for tr in soup.select("tr.team")]
pages = [urljoin(page, a["href"]) for a in soup.select("ul.pagination a[href*='page_num']")]
print(len(rows), headers[:4], rows[0][:4])
print(pages[1])

It prints 25 rows (one page of teams), the first four headers ['Team Name', 'Year', 'Wins', 'Losses'], and https://www.scrapethissite.com/pages/forms/?page_num=2: urljoin turns the relative href="/pages/forms/?page_num=2" into a URL you can fetch next.

For simple tables, pandas.read_html(r.text) returns them as DataFrames in one line; Beautiful Soup is better when cells hold links or nested tags you need.

Beautiful Soup has no XPath support. When you want XPath (for example, to select by text or by position in ways CSS can't), parse the same HTML with lxml.html:

from lxml import html as lh

tree = lh.fromstring(r.text)
names = tree.xpath('//tr[@class="team"]/td[@class="name"]/text()')
wins_1991 = tree.xpath('//tr[td[@class="year" and normalize-space()="1991"]]/td[@class="wins"]/text()')
print([n.strip() for n in names][:3], [w.strip() for w in wins_1991][:3])

Use whichever you read more easily; many scrapers mix both. More on choosing selectors in the HTML scraping guide.

Start with Beautiful Soup; move to Scrapy when you crawl thousands of pages, and to a browser only for the pages that need one. The step-by-step project is in how to build a web scraper.

Parsing pages from many sites? ProxyField residential proxies: 412M+ residential IPs in 195+ countries, a new IP per request, from $1.95/GB with no subscription; start with the Easy Trial (1 GB). Traffic never expires, and all purchases are final.