What Is Scrapy?

The Scrapy documentation describes the core advantage this way: "requests are scheduled and processed asynchronously", so Scrapy "doesn't need to wait for a request to be finished and processed" before it sends the next one (Scrapy at a glance). That is why one Scrapy process can fetch many pages concurrently, as long as the target site allows that pace.

Scrapy is used for jobs where you fetch many pages with the same structure and turn them into rows:

Scrapy or something else?

The longer comparison is in Scrapy vs Beautiful Soup and Python libraries for web scraping.

A Scrapy project is a set of small components that the engine connects. You write the spider and, when needed, pipelines and middlewares; the framework does the rest.

This spider collects quotes and follows the "Next" link on the practice site the Scrapy tutorial uses:

# quotes.py
import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for q in response.css("div.quote"):
            yield {"text": q.css("span.text::text").get(),
                   "author": q.css("small.author::text").get()}
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Run it with no project:

pip install scrapy
scrapy runspider quotes.py -O quotes.json

Expect 10 pages and 100 items in quotes.json. The first two:

[
{"text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d", "author": "Albert Einstein"},
{"text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d", "author": "J.K. Rowling"},

The \u201c sequences are curly quotes: JSON exports escape non-ASCII characters by default. Add -s FEED_EXPORT_ENCODING=utf-8 to write them as-is; projects made with startproject already set it.

The commands you use next:

If you write a custom start method, current Scrapy uses async def start(): start_requests() was deprecated in 2.13.0 (2025-05-08) and removed in 2.16.0 (2026-05-19), where it "is removed and no longer called" (release notes). Older tutorials that override start_requests() need that one change, inside the spider class:

    async def start(self):
        for url in ["https://quotes.toscrape.com/page/1/", "https://quotes.toscrape.com/page/2/"]:
            yield scrapy.Request(url, callback=self.parse)

The step-by-step version, from scrapy startproject to pipelines and exports, is the Scrapy tutorial.

Both. A crawler finds pages by following links; a scraper extracts fields from them. Scrapy's spiders do the two in one pass: parse yields items and new requests from the same response. Its CrawlSpider class goes further and follows links that match rules you set, which is how you crawl a whole site; see how to crawl a website. The difference between the two jobs is explained in web crawling vs web scraping.

Yes, for the jobs it was built for: many pages, recurring runs, data in the HTML or in JSON endpoints. What makes it good:

• Throughput. Asynchronous downloads with per-domain concurrency limits.
• Robustness. Built-in retries, redirect handling, duplicate filtering and a pause-and-resume mode for long crawls.
• Politeness controls. Download delays, AutoThrottle (which adapts the pace to the server's response times) and robots.txt handling are settings, not code.
• Clean data flow. Items go through pipelines, so cleaning and storage stay out of the spider.
• Exports with no code. -O items.json or -O items.csv on the command line.

Scrapy's three biggest limits are JavaScript (it downloads only the HTML the server sends), scale (one process uses one CPU core, and nothing built in spreads one crawl over several machines) and upkeep (selectors break when a site changes its layout). Each has a documented workaround, at a cost.

JavaScript-rendered pages

Scrapy does not run JavaScript, so a page that builds its content in the browser arrives as an empty shell. The documentation's order of work (Selecting dynamically-loaded content):

Code for each route: Scrapy and JavaScript pages.

Very large crawls

Scrapy's defaults are tuned for crawling one site politely, and its documentation lists what breaks first at scale (Optimization, Common practices):

For queues shared between machines (scrapy-redis), monitoring and storage at that size, see large-scale web scraping.

Other limits

• Upkeep. A spider is tied to a site's HTML; when the layout changes, the selectors return nothing and the item count drops. Watch it per run.
• A learning curve. Projects, settings, callbacks and the asynchronous model take longer to learn than a 10-line Requests script.
• Overkill for small jobs. One page or one table is quicker with Requests and Beautiful Soup.

Yes. Scrapy ships a release every one to two months: 2.16.0 on 2026-05-19, 2.17.0 on 2026-07-07, 2.18.0 on 2026-08-20 and 2.19.0 on 2026-09-10. 2.19.0 added a RemoteControl extension to inspect and control a running crawl. It remains the standard open-source framework for large crawls in Python. Newer tools, such as browser-based crawlers and AI extractors, sit next to it rather than replacing it: many teams use Scrapy for the crawl and a browser only for the pages that need one.

• Scheduling. Run spiders from cron or a job scheduler, or deploy them to Scrapyd, an open-source service that runs spiders over an HTTP API. More in how to schedule web scrapers.
• Pacing. Set DOWNLOAD_DELAY and enable AutoThrottle so the crawl slows down when the site does.
• Pause and resume. Run with -s JOBDIR=crawls/run1, stop with one Ctrl-C, and start the same command again to continue.
• Monitoring. Watch the item count and the share of failed responses per run; a sudden drop usually means a layout change.
• Proxies. Put the proxy in each request's proxy meta key or in the http_proxy and https_proxy environment variables, which Scrapy's HttpProxyMiddleware reads.

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
yield scrapy.Request(url, meta={"proxy": PROXY}, callback=self.parse)

With a rotating gateway, one proxy URL is enough: each request leaves from a new IP, and <login>-country-us-city-new_york asks for exit IPs in New York. The full setup, including sticky sessions and a middleware, is in the Scrapy proxy setup guide.