HTML Scraping With CSS Selectors and XPath

The DOM (Document Object Model) is the tree a parser builds: the <html> element at the root, <head> and <body> below it, and every tag, attribute and text node as a branch. Selectors are paths through that tree. Parsers repair broken HTML the way browsers do, so missing closing tags rarely stop you.

Check first that the data is in the HTML. View the page source (Ctrl+U). If what you need is missing, the page renders it with JavaScript; look for its JSON request in the Network tab or render the page in a browser (how to scrape dynamic web pages).

# pip install requests beautifulsoup4 lxml
import requests
from bs4 import BeautifulSoup

html = requests.get("https://quotes.toscrape.com/", timeout=30).text
soup = BeautifulSoup(html, "lxml")
for q in soup.select("div.quote"):
    print(q.select_one("small.author").get_text(strip=True), "-", q.select_one("span.text").get_text(strip=True)[:50])

The same in Node.js with Cheerio (1.2.0, MIT), which gives you a jQuery-like API over parsed HTML:

// npm i cheerio   (Node.js 18+ has fetch built in)
import * as cheerio from 'cheerio';
const $ = cheerio.load(await (await fetch('https://quotes.toscrape.com/')).text());
$('div.quote').each((_, el) => console.log($(el).find('small.author').text(), '-', $(el).find('span.text').text().slice(0, 50)));

CSS selectors are the same patterns stylesheets use, and every parser supports them. They are short, readable and fast.

To get an attribute in Beautiful Soup, select the element and read tag["href"]; parsel and Scrapy also accept a::attr(href) and span::text pseudo-elements. Test a selector in the browser console with document.querySelectorAll("div.quote span.text") before you put it in code.

XPath is a query language for XML and HTML trees. It is longer than CSS, but it can do two things CSS cannot: match elements by their text, and move up to a parent or ancestor.

# pip install requests lxml
import requests
from lxml import html as lh
tree = lh.fromstring(requests.get("https://quotes.toscrape.com/", timeout=30).content)
print(tree.xpath('//small[@class="author"]/text()')[:3])
print(tree.xpath('//li[@class="next"]/a/@href'))

Case-insensitive text matching needs a trick in XPath 1.0, which most parsers implement: translate(text(), "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz"). If an XPath with text() returns nothing although you see the text, the text is probably split across child elements; use contains(., "word") (the dot means all text inside the element) instead.

Use CSS by default and XPath for the cases CSS cannot express. parsel, the library behind Scrapy's selectors (1.12.1), lets you mix both on the same object, and lxml translates CSS to XPath through cssselect.

from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup

base = "https://quotes.toscrape.com/"
soup = BeautifulSoup(requests.get(base, timeout=30).text, "lxml")
links = {urljoin(base, a["href"]) for a in soup.select("a[href]")}
internal = sorted(u for u in links if u.startswith(base))
print(len(links), "links,", len(internal), "on the same site")

Three details matter: make every href absolute with urljoin, drop fragments (#section) and mailto: or javascript: links, and deduplicate with a set. Collecting the links of a whole site, page by page, is crawling: how to crawl a website.

For the text of one element, use get_text(" ", strip=True) in Beautiful Soup or html_text2() in rvest. For the main text of an article, without menus, ads and footers:

For a real <table>, let a library do the work: pandas.read_html(io.StringIO(html)) returns every table as a DataFrame, and rvest's html_table() does the same in R (web scraping in R). Tables built from <div> grids need row and cell selectors. More on exporting tables to spreadsheets: web scraping in Excel.

Before you write selectors, search the page source for application/ld+json and for large JSON blobs in <script> tags. Many product, article and event pages carry structured data in schema.org JSON-LD, and many JavaScript frameworks embed the page's data as JSON (for example a __NEXT_DATA__ script). Parsing that JSON is more stable than parsing the visible layout.

import json
# soup: the parsed page from the first example
for s in soup.select('script[type="application/ld+json"]'):
    data = json.loads(s.string or "{}")
    print(data.get("@type"), data.get("name"))

Not for parsing HTML: a regular expression cannot follow nested tags, attribute order or the broken markup that parsers repair, so it breaks on small layout changes. Regex is the right tool after the parser, on the text you extracted: prices (r"[\d,.]+"), dates, IDs in URLs, phone formats, or a value inside a <script> block that no selector reaches.

import re
price_text = "Price: $1,299.00 (incl. VAT)"
price = float(re.search(r"[\d,]+\.\d{2}", price_text).group().replace(",", ""))

• The selector works in the browser but not in the script: the browser shows the DOM after JavaScript; your script sees the raw HTML. Compare with the page source.
• Garbled characters: the page's encoding was guessed wrong. Pass the bytes (r.content) to the parser, or set r.encoding before reading r.text.
• Fields shift between rows: select within each item card, not across the whole page, so a missing field becomes empty instead of misaligning rows.
• Classes like css-1x2y3z change on every deploy: select by stable attributes (data-testid, itemprop, aria-label), by structure, or by nearby text with XPath.
• Parsing is slow at volume: switch Beautiful Soup to the lxml parser, or use lxml or selectolax directly (Python libraries for web scraping).

Parsing and scraping are separate steps; the difference is explained in parsing vs scraping.

Parsing is local, but fetching happens on the target's servers, and sites limit how many requests one IP may send. When a scraper fetches thousands of pages from one site, route the HTTP client through rotating proxies, so each request leaves from a new IP:

import requests

url = "https://quotes.toscrape.com/"
proxy = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
html = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=30).text

<login>-country-us-city-new_york asks for an exit IP in New York. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional. ProxyField residential proxies give each request a new IP or keep a sticky session from 30 s to 36 h, over HTTP, HTTPS or SOCKS5, with unlimited concurrent threads. Raw HTML is typically small, so a GB of traffic covers thousands of pages. They cost $2.95/GB for 1 GB and fall to $1.95/GB from 1,000 GB, with no subscription; traffic never expires, and all purchases are final. Setup for every Python client: Python proxy setup. Start with residential proxies and the Easy Trial: 1 GB for $2.95.