How to Crawl a Website

Crawling finds pages; scraping extracts data from them. A crawler follows links from page to page to discover URLs, and a scraper pulls the fields you need from each page it visits. Most data jobs do both in one pass. The two jobs in detail: web crawling vs web scraping; what a crawler is in general: what a web crawler is.

Decide before the first request:

• Which hosts: only www.example.com, or subdomains too.
• Which paths: for example only /products/ and /category/, never /cart or /account.
• How deep: how many clicks from the start page you follow.
• How many pages: a hard cap, so a mistake cannot run for days.
• What you keep: the full HTML, extracted fields, or only the list of URLs.

A tight scope is the most effective way to make a crawl faster, cheaper and kinder to the site.

https://example.com/robots.txt lists the paths the site asks crawlers to avoid and often the location of its sitemaps (Sitemap: lines). The rules are a request, not access control: RFC 9309 says they "are not a form of access authorization" (RFC 9309), and the site's terms of service apply as well. Respect both; how to read them: how to check if scraping is allowed.

Sitemaps are the fastest way to discover URLs: an XML list the site publishes for search engines, each file holding up to 50,000 URLs and up to 50 MB uncompressed, often grouped under a sitemap index (sitemaps.org protocol). Many sites list every product or article there, with a lastmod date that tells you what changed since your last crawl.

# list the URLs in a sitemap (and in the sitemaps of a sitemap index)
import requests
from lxml import etree

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

def sitemap_urls(url):
    root = etree.fromstring(requests.get(url, timeout=30).content)
    if root.tag.endswith("sitemapindex"):
        for loc in root.xpath("//sm:sitemap/sm:loc/text()", namespaces=NS):
            yield from sitemap_urls(loc)
    else:
        yield from root.xpath("//sm:url/sm:loc/text()", namespaces=NS)

When there is no sitemap, or it is incomplete, follow links. A crawler keeps a queue of URLs to visit and a set of URLs it has seen; it fetches the next URL, extracts its links, keeps those in scope and adds the new ones to the queue. Fetching in queue order (breadth-first) visits pages closest to the start first, which is usually what you want.

# pip install requests beautifulsoup4   ->   python crawl.py
import time
from collections import deque
from urllib.parse import urljoin, urldefrag, urlparse
from urllib.robotparser import RobotFileParser
import requests
from bs4 import BeautifulSoup

START = "https://books.toscrape.com/"
MAX_PAGES, MAX_DEPTH, DELAY = 200, 3, 1.0
UA = "Mozilla/5.0 (compatible; site-crawl-tutorial/1.0; [email protected])"
SKIP = (".jpg", ".jpeg", ".png", ".gif", ".pdf", ".zip", ".css", ".js")

host = urlparse(START).netloc
robots = RobotFileParser(urljoin(START, "/robots.txt")); robots.read()
session = requests.Session(); session.headers["User-Agent"] = UA

def normalize(base, href):
    url, _ = urldefrag(urljoin(base, href))          # absolute, no #fragment
    return url

queue, seen, pages = deque([(START, 0)]), {START}, []
while queue and len(pages) < MAX_PAGES:
    url, depth = queue.popleft()
    if not robots.can_fetch(UA, url):
        continue
    r = session.get(url, timeout=30)
    if r.status_code != 200 or "text/html" not in r.headers.get("Content-Type", ""):
        continue
    soup = BeautifulSoup(r.text, "html.parser")
    pages.append({"url": url, "depth": depth, "title": soup.title.get_text(strip=True) if soup.title else ""})
    if depth < MAX_DEPTH:
        for a in soup.select("a[href]"):
            link = normalize(r.url, a["href"])
            if urlparse(link).netloc == host and link not in seen and not link.lower().endswith(SKIP):
                seen.add(link)
                queue.append((link, depth + 1))
    time.sleep(DELAY)

print(len(pages), "pages crawled,", len(seen), "URLs found")

This is enough for a site of a few thousand pages. For bigger sites, store the queue and the seen set on disk so a crash does not restart the crawl, and fetch several pages at once with a per-host limit. Building a crawler in full: how to build a web crawler and the Python web crawler tutorial.

List crawling means crawling only the pages that list items, such as category pages, search results and their pagination, and then the detail pages they link to. For data collection it beats a full-site crawl: if you want every product, you need the category tree and the product pages, not the blog, the help center or the 40 filter combinations of every category.

How to do it:

1. Find the list pages: the category menu, a sitemap section, or a search with an empty query.
2. Follow pagination on each list page (the "next" link or ?page=N) until it ends.
3. Collect the detail-page links from each list page, deduplicated.
4. Fetch the detail pages and extract the fields.

List pages also tell you when to stop: when a category's last page is reached, you have all its items.

Crawl depth is how many links away from the start page a URL is. A depth limit matters because the number of URLs grows quickly with each level, and because pages far from the home page are often archives, filters or duplicates. Set three limits together:

• Max depth, often 3 to 5 for content sites; list crawls need only the levels down to the detail pages.
• Max pages in total and per URL pattern, so one runaway section cannot take the whole budget.
• Path rules, an allow list of patterns you want and a deny list of patterns you never fetch.

A crawl trap, also called a spider trap, is a part of a site that generates endless distinct URLs. A depth limit alone does not prevent it, because the trap can be shallow and wide. The common ones and their fixes:

Normalize every URL the same way before you check whether you have seen it: absolute form, lowercase host, no fragment, tracking parameters removed, the rest sorted. Watch the crawl as it runs: if one URL pattern grows much faster than the others, it is probably a trap.

• Pace: start at about one request per second per site and slow down if responses slow down. Scrapy's AutoThrottle does this automatically.
• Concurrency: a few parallel requests per host at most.
• Respect 429 and 503: wait for the Retry-After time, then back off further; see 429 Too Many Requests.
• Identify yourself: a User-Agent with a contact address.
• Crawl off-peak for large jobs, and recrawl only what changed, using sitemap lastmod dates or HTTP If-Modified-Since.

With Scrapy, link-following rules replace most of the code above:

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class BooksCrawler(CrawlSpider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]
    custom_settings = {"DEPTH_LIMIT": 3, "CLOSESPIDER_PAGECOUNT": 500, "DOWNLOAD_DELAY": 1, "ROBOTSTXT_OBEY": True}
    rules = [Rule(LinkExtractor(allow=r"/catalogue/category/"), follow=True),              # list pages
             Rule(LinkExtractor(allow=r"/catalogue/[^/]+_\d+/index\.html"), callback="parse_book")]  # detail pages

    def parse_book(self, response):
        yield {"url": response.url, "title": response.css("h1::text").get(),
               "price": response.css("p.price_color::text").get()}

Comparisons of ready-made crawlers: best web crawlers.

To get all the pages, or all the text, of a site: list the URLs from the sitemaps, fill the gaps by following links within your scope, then fetch each URL once and store the HTML or the extracted text. wget --mirror --convert-links --wait=1 https://example.com/ saves a browsable copy of a small static site. Check the site's terms first: copying a whole site is a much bigger request than scraping a few fields, and content may be protected by copyright (is web scraping legal).