Scrapy vs BeautifulSoup vs Selenium vs Playwright

All four are open source. Current Scrapy, Selenium and Playwright releases need Python 3.10 or later, and Beautiful Soup 4.15.0 needs Python 3.7 or later. Versions are from PyPI.

Beautiful Soup describes itself as "a Python library for pulling data out of HTML and XML files" (docs); Scrapy as "a high-level Web Crawling and Web Scraping framework" (PyPI). That one line explains most of the comparison: a parser reads pages, a framework runs the whole crawl.

The task: collect the title and price of all 1,000 books on Books to Scrape, a practice site with 50 pages of 20 books.

With Requests and Beautiful Soup, you write the loop, the pagination and the pause yourself:

# pip install requests beautifulsoup4
import time
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

url = "https://books.toscrape.com/catalogue/page-1.html"
books = []
while url:
    soup = BeautifulSoup(requests.get(url, timeout=30).content, "html.parser")
    for card in soup.select("article.product_pod"):
        books.append({"title": card.h3.a["title"], "price": card.select_one(".price_color").text})
    nxt = soup.select_one("li.next a")
    url = urljoin(url, nxt["href"]) if nxt else None
    time.sleep(0.5)  # one page at a time, with a pause
print(len(books), books[0])

With Scrapy, you declare what to extract and which links to follow; the framework schedules the requests, runs several at once, retries failures and writes the file:

# pip install scrapy, then: scrapy runspider books_spider.py
import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]
    custom_settings = {"CONCURRENT_REQUESTS": 8, "AUTOTHROTTLE_ENABLED": True,
                       "FEEDS": {"books.json": {"format": "json"}}}

    def parse(self, response):
        for card in response.css("article.product_pod"):
            yield {"title": card.css("h3 a::attr(title)").get(),
                   "price": card.css(".price_color::text").get()}
        yield from response.follow_all(css="li.next a", callback=self.parse)

We ran both on October 23, 2026 (Requests 2.34.2, Beautiful Soup 4.15.0, Scrapy 2.19.0). Both saved all 1,000 books. The sequential script with a 0.5 s pause took about 61 s; the Scrapy spider, with 8 concurrent requests and AutoThrottle adjusting the pace, took about 19 s. That is one run on one small site, not a benchmark: what you should take from it is that Scrapy's gain comes from running requests in parallel, which you would otherwise have to build yourself.

One detail the test caught: pass response.content (bytes) to Beautiful Soup, not response.text. Requests guessed the wrong encoding for this site and turned "£" into "£"; with bytes, Beautiful Soup reads the charset from the page.

• You scrape tens or hundreds of pages, once or on a schedule.
• You already have the HTML, from a file, an email, an API field or a browser.
• The script lives inside a larger program, a notebook or a data pipeline, and you do not want a framework's project layout.

Beautiful Soup is forgiving with broken HTML and easy to read. For speed on large documents, use it with the lxml parser, or move to lxml or parsel directly; more in Beautiful Soup web scraping.

• Thousands to millions of pages, where concurrency, retries and deduplication of URLs matter.
• Crawls that follow links across a site, with rules for what to follow (CrawlSpider).
• Recurring jobs that need clean exports, item pipelines, logging and stats.
• Scrapy also parses: its CSS and XPath selectors (parsel) replace Beautiful Soup inside a spider, though you can still call Beautiful Soup from a callback if you prefer its API.

Scrapy has a learning curve: spiders, callbacks, settings and middlewares. What Scrapy is explains the parts, and the Scrapy tutorial builds a project step by step. For runs across several servers, see large-scale web scraping.

Neither Beautiful Soup nor Scrapy runs JavaScript. If the data you want is missing from the page source, you have two routes:

Between the two browsers, Playwright is the easier start for new Python scrapers, with auto-waiting and request blocking built in; Selenium suits teams that already run it. The full comparison is in Playwright vs Selenium vs Puppeteer, and the same parser-or-browser question for Node.js in Cheerio vs Puppeteer.

More Python options, including httpx, lxml and selectolax, are in our guide to Python scraping libraries.