Python Libraries for Web Scraping

Versions, licenses and release dates from PyPI. Each name links to the library's official documentation.

All of these are open source, so the library costs nothing; at scale you pay for servers and proxy traffic.

Fetching: Requests and httpx are equally fast one page at a time, as long as you reuse one requests.Session or httpx.Client; the network decides the time, not the library. For many pages, go async: httpx's AsyncClient or aiohttp fetched 200 pages at 100 ms latency in 0.63 s and 0.24 s, against 20.3 s one at a time. Parsing: selectolax is the fastest, lxml is close behind, and Beautiful Soup is 25 to 45 times slower than either; that matters only from hundreds of thousands of pages.

We measured both on 2026-10-27, on a server with an AMD Ryzen 9 5950X (8 vCPUs), Linux and Python 3.12.3, with the library versions in the table above. The fetch tests called a local test server in a separate process that returned a 16 KB page, so they measure the client, not the internet; the 100 ms delay stands in for a real site's latency. Each figure is the median of 3 runs.

Fetching: seconds for the whole batch (lower is better)

Parsing: milliseconds per page for a 55.7 KB product listing: build the tree, then extract 100 product names (lower is better; median of 3 runs of 1,000 pages)

What the numbers say:

• Reuse one client. httpx.get() in a loop was 37 times slower than one httpx.Client (15.7 s against 0.43 s per 1,000), because each call builds a new client and its SSL context. requests.get() costs less, but a Session still keeps connections open.
• Set a timeout everywhere. Requests has no default timeout and can wait forever on a stalled server; httpx's default is 5 seconds; aiohttp's is 5 minutes in total. Pass timeout= on every call or client.
• Cap the requests in flight. httpx's pool allows 100 connections and keeps 20 alive by default (aiohttp: 100 connections). Queuing all 1,000 requests on httpx at once took 21.1 s; the same 1,000 with an asyncio.Semaphore(100) took 1.4 s. On a real site, cap it far lower, at the pace the site allows.
• Beautiful Soup's parser choice matters less than people think. Switching it to lxml saved about 30%; the tree that Beautiful Soup builds on top costs the rest. If parsing is your bottleneck, change the library, not the backend.
• Query style matters. lxml's XPath query was 3.5 times faster than its CSS query on the same tree. With one simple query, lxml and selectolax come out nearly even.
• The network still wins. At 100 ms per page, a one-by-one scraper spends 100 ms waiting for each page and under 1 ms parsing it with lxml or selectolax.

Measure on a sample of your own pages before optimizing; this script reproduces the parse test on any saved page:

# pip install beautifulsoup4 lxml parsel selectolax
import time
import lxml.html
from bs4 import BeautifulSoup
from parsel import Selector
from selectolax.lexbor import LexborHTMLParser

html = open("page.html", encoding="utf-8").read()   # any saved page

def bench(name, parse, n=1000):
    t = time.perf_counter()
    for _ in range(n):
        parse(html)
    print(f"{name:30} {(time.perf_counter() - t) / n * 1000:.2f} ms per page")

bench("Beautiful Soup + html.parser", lambda h: [a.get_text() for a in BeautifulSoup(h, "html.parser").select("h3 a")])
bench("Beautiful Soup + lxml", lambda h: [a.get_text() for a in BeautifulSoup(h, "lxml").select("h3 a")])
bench("lxml + XPath", lambda h: lxml.html.fromstring(h).xpath("//h3/a/text()"))
bench("parsel", lambda h: Selector(text=h).css("h3 a::text").getall())
bench("selectolax (Lexbor)", lambda h: [a.text() for a in LexborHTMLParser(h).css("h3 a")])

More ways to speed a scraper up, from concurrency to caching: how to make web scraping faster.

Requests is the default way to fetch a page in Python: one call, readable code, sessions that keep cookies. It is synchronous, so one script fetches one page at a time unless you add threads.

# pip install requests beautifulsoup4
import requests
from bs4 import BeautifulSoup

proxy = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
with requests.Session() as s:
    s.proxies = {"http": proxy, "https": proxy}
    r = s.get("https://quotes.toscrape.com/", timeout=30)
    soup = BeautifulSoup(r.text, "html.parser")
    print([q.get_text() for q in soup.select("span.text")][:3])

httpx has a Requests-like API plus an async client and optional HTTP/2, which makes it the usual upgrade when a script must fetch hundreds of pages concurrently. aiohttp is the older async client with the same purpose, and the fastest in our fetch test. curl_cffi wraps libcurl and can make its TLS handshake match a chosen browser's; some sites treat Python's default TLS handshake differently from a browser's, which is when people reach for it. It is still marked beta. Through ProxyField, the Acceptable Use Policy applies whatever client you use: public pages only, and follow the site's terms; fraud, attacks and access to data behind logins you are not entitled to are not allowed.

# pip install httpx  ->  async fetching of many pages, at most 10 in flight
import asyncio, httpx

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"

async def main(urls):
    sem = asyncio.Semaphore(10)
    async with httpx.AsyncClient(proxy=PROXY, timeout=30) as client:
        async def fetch(u):
            async with sem:
                return await client.get(u)
        pages = await asyncio.gather(*(fetch(u) for u in urls))
        print([p.status_code for p in pages])

asyncio.run(main([f"https://quotes.toscrape.com/page/{n}/" for n in range(1, 6)]))

Proxy setup for each client, with SOCKS5 and rotation: Python proxy setup. Async patterns in depth: async web scraping in Python.

A parser turns the HTML string into a tree you can query.

• Beautiful Soup is the most forgiving and the easiest to read: soup.select("div.price"), tag.get_text(), tag["href"]. It builds its own tree on top of a parser; passing "lxml" instead of "html.parser" saved about 30% in our test. Tutorial: Beautiful Soup web scraping.
• lxml is a fast C-based parser with full XPath 1.0 and CSS selectors through the cssselect package.
• parsel is the selector library inside Scrapy: one object with .css() and .xpath(), usable without Scrapy.
• selectolax is a Cython parser on the Lexbor engine, built for speed with CSS selectors. Since version 1.0, import LexborHTMLParser from selectolax.lexbor: the old from selectolax.parser import HTMLParser (the Modest engine) now raises an ImportError that says the backend is deprecated.

Which selectors to write, CSS or XPath, and how to extract links and article text: HTML scraping with CSS selectors and XPath.

When a script turns into a crawler, with link following, retries, deduplication, pacing and exports, a framework saves you from writing that plumbing.

• Scrapy has been the standard for more than a decade: asynchronous downloads, spiders, item pipelines, middlewares and feed exports. It does not render JavaScript by itself. See what Scrapy is and the Scrapy tutorial.
• Crawlee for Python combines HTTP crawlers (with Beautiful Soup or parsel) and a Playwright crawler behind one API, with request queues, sessions and proxy rotation.

Use a browser only for pages that need one: it costs far more CPU, memory and traffic per page than an HTTP request.

• Playwright for Python is the one to start a new project with: it installs its own Chromium, Firefox and WebKit builds (playwright install), has sync and async APIs, and its locators wait for elements by themselves. Tutorial: Playwright web scraping.
• Selenium drives browsers through WebDriver and is everywhere in QA teams; choose it if your team already has Selenium code. Selenium Manager downloads drivers by itself since 4.6. Tutorial: web scraping with Selenium and Python.
• SeleniumBase adds waits, helpers and a test runner on top of Selenium.

Pyppeteer, a Python port of Puppeteer, is unmaintained: its README says "This repo is unmaintained" and asks users to "consider playwright-python as an alternative" (github.com/pyppeteer/pyppeteer). Scraping dynamic pages step by step: Python dynamic web scraping.

A step-by-step tutorial with the first stack: Python web scraping. Tools beyond Python, from no-code apps to scraping APIs: best web scraping tools.