proxyfield
Start with 1 GB
Tutorial8 min read

Scrapy Tutorial: Build Your First Spider

Updated · Part of how to scrape a website
Short answer

To scrape with Scrapy, install it with pip install scrapy, create a project with scrapy startproject, write a spider class whose parse() method yields one dictionary per item and follows the "next" link, and run it with scrapy crawl <name> -O items.json. That is the whole loop; this tutorial builds it step by step on Scrapy 2.19, then adds error handling, item pipelines, proxies and scheduled runs. ProxyField's rotating residential proxies plug into Scrapy with one proxy request key, with 412M+ IPs in 195+ countries from $1.95/GB; start with the Easy Trial: 1 GB for $2.95.

residential IPs
412M+
from, traffic never expires
$1.95/GB
Easy Trial
1 GB for $2.95
Start with 1 GB See pricing
On this page 13 sections
  1. What you will build
  2. Step 1: install Scrapy and start a project
  3. Step 2: write your first spider
  4. Step 3: run it and export the data
  5. Step 4: find selectors with the Scrapy shell
  6. Step 5: modify the spider to scrape the data you need
  7. Step 6: clean and validate items in a pipeline
  8. Step 7: handle errors in Scrapy
  9. Step 8: crawl politely and at scale
  10. Step 9: pages that need JavaScript
  11. Step 10: proxies in Scrapy
  12. Step 11: schedule Scrapy runs
  13. Scrapy Tutorial FAQ

What you will build

A spider that collects every quote, author and tag from the practice site quotes.toscrape.com, follows pagination to the last page, opens each author's page for their birth date, cleans and validates the items, survives errors, and runs on a schedule. What Scrapy is, and how it compares with other tools, is in what Scrapy is; this page is the hands-on part.

Step 1: install Scrapy and start a project

Use a virtual environment so Scrapy's dependencies stay separate. Scrapy 2.19 needs Python 3.10 or newer (Scrapy installation guide).

Shell
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install scrapy
scrapy startproject quotes_project
cd quotes_project
scrapy genspider quotes quotes.toscrape.com

startproject creates this layout:

quotes_project/
    scrapy.cfg              # deploy configuration
    quotes_project/
        items.py            # item definitions
        middlewares.py      # downloader and spider middlewares
        pipelines.py        # item pipelines: clean, validate, store
        settings.py         # project settings
        spiders/
            quotes.py       # created by genspider

The generated settings.py already sets polite defaults: ROBOTSTXT_OBEY = True, one concurrent request per domain and a one-second delay.

Step 2: write your first spider

Open spiders/quotes.py and replace its contents:

Python
import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
            }

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

How it works:

  1. Scrapy requests each URL in start_urls and calls parse() with the response.
  2. response.css() selects elements; ::text and ::attr(href) extract text and attributes; .get() returns the first match or None, .getall() a list.
  3. Each yield {...} emits one item.
  4. response.follow() turns the relative "next" link into a new request with the same callback, so the spider walks every page. Scrapy drops duplicate URLs for you.
Easy Trial: 1 GB for $2.95.
Try for the price of 1 GB, no subscription, traffic never expires.
Start with 1 GB See pricing

Step 3: run it and export the data

Shell
scrapy crawl quotes -O quotes.json      # -O overwrites, -o appends
scrapy crawl quotes -O quotes.csv       # the extension picks the format
scrapy crawl quotes -O quotes.jsonl     # one JSON object per line, best for big runs

The log ends with statistics: requests, responses by status code, items scraped and the finish reason. Read them after every run; they are the first place a problem shows.

Step 4: find selectors with the Scrapy shell

The shell fetches a page and lets you try selectors interactively before you put them in the spider:

Shell
scrapy shell "https://quotes.toscrape.com/"
>>> response.css("div.quote span.text::text").get()
>>> response.xpath("//small[@class='author']/text()").getall()
>>> view(response)        # opens the page as Scrapy received it in your browser

view(response) matters: if the data is missing there but visible in your browser, the page builds it with JavaScript (see Step 9).

Step 5: modify the spider to scrape the data you need

Real projects need more than one page type. To add each author's birth date, follow the author link and pass the quote along with cb_kwargs:

Python
    def parse(self, response):
        for quote in response.css("div.quote"):
            item = {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
            }
            author_url = quote.css("span a::attr(href)").get()
            yield response.follow(author_url, callback=self.parse_author, cb_kwargs={"item": item})

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

    def parse_author(self, response, item):
        item["born"] = response.css("span.author-born-date::text").get()
        yield item

To change what you collect: find the new field in scrapy shell, add one line to the item, and add a callback for each new page type. Note that Scrapy filters duplicate requests, so a second quote by the same author will not refetch that author's page; pass dont_filter=True on that request if every quote needs the field, or store authors separately.

For a fixed schema, declare the item as a dataclass in items.py; Scrapy accepts dicts, scrapy.Item classes, dataclasses and attrs classes:

Python
from dataclasses import dataclass, field

@dataclass
class Quote:
    text: str
    author: str
    tags: list = field(default_factory=list)
    born: str | None = None

Step 6: clean and validate items in a pipeline

Pipelines process every item after the spider yields it: clean, validate, deduplicate, store. Add one to pipelines.py:

Python
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem


class CleanAndValidatePipeline:
    def process_item(self, item):
        adapter = ItemAdapter(item)
        if not adapter.get("text") or not adapter.get("author"):
            raise DropItem("missing text or author")       # counted in the stats, not saved
        adapter["text"] = adapter["text"].strip("“”")   # remove curly quotes
        return item

Enable it in settings.py with ITEM_PIPELINES = {"quotes_project.pipelines.CleanAndValidatePipeline": 300}; the number sets the order when you have several.

Step 7: handle errors in Scrapy

Scrapy already retries some failures for you: with the default settings, a request that times out, loses its connection or gets 408, 429, 500, 502, 503, 504, 522 or 524 is retried twice (RETRY_TIMES = 2). Other non-2xx responses are dropped before your callback by the HttpError middleware. To see and record the failures, add an errback:

Python
import scrapy
from scrapy.spidermiddlewares.httperror import HttpError


class QuotesSpider(scrapy.Spider):
    # name, allowed_domains, start_urls, parse() and parse_author() as above

    async def start(self):
        for url in self.start_urls:
            yield scrapy.Request(url, callback=self.parse, errback=self.on_error)

    def on_error(self, failure):
        request = failure.request
        if failure.check(HttpError):
            response = failure.value.response
            self.logger.warning("HTTP %s on %s", response.status, response.url)
        else:
            self.logger.error("%r on %s", failure.value, request.url)
        yield {"url": request.url, "error": repr(failure.value)}   # keep a record of the failure

start() replaces the old start_requests(), which was deprecated in Scrapy 2.13 and removed in 2.16 (Scrapy release notes). Other settings for errors:

  • RETRY_TIMES and RETRY_HTTP_CODES change how many and which failures are retried.
  • HTTPERROR_ALLOWED_CODES = [404] (or handle_httpstatus_list on the spider) passes those responses to your callback, for example to record removed products.
  • CLOSESPIDER_ERRORCOUNT = 50 stops a run that is clearly broken instead of letting it fail for hours.
  • DOWNLOAD_TIMEOUT (180 s by default) is worth lowering to 30-60 s for most sites.

When retries are exhausted, look at the status codes in the final stats: many 429 or 403 responses mean you are too fast or too visible: slow down first and, where a site limits requests per IP, spread the requests over more IPs; more retries only add load. More in web scraping best practices.

Step 8: crawl politely and at scale

Settings that matter once a spider runs on someone else's site:

SettingDefault in a new projectWhat it does
ROBOTSTXT_OBEYTrueSkips URLs the site's robots.txt disallows
CONCURRENT_REQUESTS_PER_DOMAIN1Parallel requests to one site
DOWNLOAD_DELAY1Seconds between requests to one site (randomized by default)
AUTOTHROTTLE_ENABLEDoffAdjusts the delay to the site's response times
USER_AGENTScrapy's ownSet a descriptive one, ideally with a contact
JOBDIRunsetSaves the queue to disk so scrapy crawl ... -s JOBDIR=crawls/run1 can pause and resume

For a site-wide crawl that follows links by rules instead of hand-written follow() calls, use CrawlSpider; see how to build a Python web crawler.

Step 9: pages that need JavaScript

Scrapy downloads HTML; it does not run JavaScript. If view(response) shows an empty page, open your browser's DevTools, Network tab, and look for the JSON request that brings the data: requesting that URL from Scrapy is the fastest fix. If there is none, the scrapy-playwright plugin renders selected requests in a headless browser while the rest of the crawl stays plain HTTP. When a browser is worth it is covered in headless browser scraping.

Step 10: proxies in Scrapy

Scrapy's built-in HttpProxyMiddleware reads the proxy from each request's meta, credentials included:

Python
PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"

yield scrapy.Request(url, callback=self.parse, meta={"proxy": PROXY})

With ProxyField's rotating gateway every request gets a new residential IP, so one proxy URL is enough; no proxy list or rotation middleware is needed. <login>-country-us-city-new_york asks for an IP in New York, United States. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional. Setting the proxy for the whole project, sticky sessions and country targeting are covered in the Scrapy proxy setup guide.

Step 11: schedule Scrapy runs

A spider that collects prices or listings usually runs daily or hourly. Three common ways:

  1. cron (Linux, macOS) or Task Scheduler (Windows). Run the crawl command from the project folder, with a dated output file:
Shell
# crontab -e : every day at 03:15
15 3 * * * cd /srv/quotes_project && .venv/bin/scrapy crawl quotes -O data/quotes-$(date +\%F).jsonl >> logs/quotes.log 2>&1

  1. Scrapyd

    An open-source service that runs Scrapy projects you deploy to it and schedules spiders through an HTTP API; pair it with cron or a job runner for the timetable.

  2. A Python script

    CrawlerProcess runs spiders from your own code, so an existing scheduler (Airflow, a CI pipeline, a container job) can start them:

Python
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())
process.crawl("quotes")
process.start()

Whatever runs it, keep the logs and stats of each run and alert when items drop or errors rise. More approaches are in how to schedule web scrapers.

ProxyField residential proxies for Scrapy cost $2.95/GB for 1 GB down to $1.95/GB from 1,000 GB, with no subscription; the traffic never expires, and all purchases are final (residential and mobile proxy pricing). Collect only public data and check each site's terms; see is web scraping legal.

Scrapy Tutorial FAQ

How do I scrape with Scrapy?

ProxyField's residential proxies (412M+ residential IPs, from $1.95/GB) plug into Scrapy with one meta={"proxy": ...} key, and you scrape with Scrapy in four steps: install it, run scrapy startproject, write a spider whose parse() yields items and follows the "next" link, and run scrapy crawl <name> -O items.json. Test selectors in scrapy shell first.

How do I install Scrapy and start a project?

Create a virtual environment, run pip install scrapy (it needs Python 3.10 or newer), then scrapy startproject myproject and scrapy genspider name example.com inside it. The project gets a settings file with polite defaults and a spiders folder.

How do I handle errors in Scrapy?

Let the retry middleware retry timeouts, 429 and 5xx responses (twice by default), attach an errback to requests to log or record what still fails, and use HTTPERROR_ALLOWED_CODES for statuses you want in your callback. CLOSESPIDER_ERRORCOUNT stops a run that keeps failing.

How do I modify a Scrapy spider to scrape the data I need?

Find the new field's selector in scrapy shell, add it to the item in your callback, and add a new callback with response.follow() for each extra page type, passing data along with cb_kwargs. Declare the item as a dataclass if you want a fixed schema.

What is the purpose of callbacks and errbacks in Scrapy?

A callback receives the response of a successful request and yields items or new requests; an errback receives the failure when the request errors or returns a filtered status. Errbacks can log, retry differently or yield a record of the failure.

How can I handle websites with pagination in Scrapy?

Select the "next" link in parse() and yield response.follow(next_url, callback=self.parse) until there is none. For numbered pages or APIs, generate the page URLs in start(), or use a CrawlSpider rule that follows the pagination links.

What are Scrapy items and why should we use them?

Items are the records your spider yields, as dicts, scrapy.Item classes, dataclasses or attrs classes. Declared items give every record the same fields, catch typos in field names, and make pipelines and exports predictable.

How can you host Scrapy spiders?

Any machine that runs Python can host a Scrapy project: a server or VPS that starts the crawl from cron, a container that a job scheduler starts, or a Scrapyd instance you deploy the project to and control through its HTTP API. Keep the output files and logs outside the container, so each run's data and stats survive.

Is Scrapy faster than other Python scraping tools?

For crawling many pages, usually yes: it sends requests concurrently and asynchronously, while a simple Requests loop waits for each page. For one page, or for pages that need a browser, the difference disappears or reverses.

When should I use Selenium instead of Scrapy?

When the pages need a real browser for clicks, logins to your own account or JavaScript rendering, and there is no JSON request to call. For large crawls of HTML pages, Scrapy is far faster; scrapy-playwright combines both by rendering only selected requests.

Show 3 more questionsShow fewer questions

Why choose Scrapy over Requests and Beautiful Soup for crawling?

Scrapy ships with what a crawl needs and a script lacks: concurrency, deduplication, retries, robots.txt handling, throttling, pipelines, export formats and pause-and-resume. Requests and Beautiful Soup are simpler for a handful of pages.

What is the best way to manage large-scale web scraping projects with Scrapy?

Give each site its own spider and share pipelines for validation and storage, with JOBDIR for resumable runs and AutoThrottle plus per-domain limits for the pace. Run it from a scheduler such as cron or Scrapyd, monitor the run stats, and use rotating proxies for sites that limit requests per IP.

How do I schedule Scrapy spiders to run automatically?

Run scrapy crawl from cron or Task Scheduler, deploy the project to Scrapyd and schedule spiders through its API, or start them from your own scheduler with CrawlerProcess. Write a dated output file and keep each run's log.

Keep reading

Related pages

Easy Trial: 1 GB for $2.95.

Try for the price of 1 GB, no subscription, traffic never expires.

Start with 1 GB See pricing