How to Build a Web Crawler

Every crawler, from a weekend script to a search engine, has the same parts. What changes with scale is where each part lives.

The frontier is the crawler's to-do list: every discovered URL that has not been fetched yet, plus the rules for which one comes next. Its order shapes the crawl:

• FIFO (first in, first out) visits pages in breadth-first order: all pages one click from the seeds, then two clicks, and so on. Good for site-wide crawls with a depth limit.
• Priority orders URLs by a score (fewer clicks from the home page, a recent lastmod in the sitemap, a URL pattern you care about). Good when you can't fetch everything.
• Per-host queues keep one queue per host and take from a host only when its politeness delay has passed. That is how large crawlers stay fast overall while slow on each site; the design goes back to the Mercator crawler described by Heydon and Najork in 1999.

The seen set sits next to the frontier: a URL is added to the frontier only if it was never seen before. Without it, a crawler loops forever between pages that link to each other.

The same page hides behind many URLs: HTTP://Example.com/a#top, http://example.com/a, http://example.com/a?utm_source=x. Normalize every link before you check the seen set:

1. Make it absolute with urljoin(page_url, href).
2. Drop the #fragment; it never changes what the server sends.
3. Lowercase the scheme and host, and drop default ports.
4. Remove tracking parameters (utm_*, gclid, session IDs) and, if order never matters on the site, sort the remaining query parameters.
5. If the page declares <link rel="canonical">, store that as the page's URL.

For millions of URLs, a Bloom filter answers "seen before?" in a few bits per URL, with a small chance of a false "yes" and never a false "no". Deduplicate content too: hash the page body, and treat two URLs with the same hash as one page.

robots.txt is the site's instruction to crawlers, standardized as RFC 9309 in 2022. Read it once per host, cache it, and check every URL against it before you fetch. Two practical notes:

• Fetch robots.txt with your own client and user agent. Python's RobotFileParser.read() uses its own default user agent, and if a server answers that with 401 or 403, the parser treats the whole site as disallowed. The code below fetches the file with the crawler's session and passes the lines to parse().
• Crawl-delay is not part of RFC 9309, but some sites set it; honoring it is a good default.

Pacing matters more than robots.txt for staying welcome: keep at least one second between requests to the same host for a small crawl, send a user agent that names your crawler and a contact URL, and back off when a host answers 429 or 503 or gets slower.

This crawler starts from one seed, stays inside the allowed hosts, checks robots.txt, waits between requests to the same host, stores each page's status and title in SQLite, and adds new links to the frontier. It needs pip install requests beautifulsoup4. It was tested on the practice sandbox scrapethissite.com.

import sqlite3, time, urllib.robotparser
from collections import deque
from urllib.parse import urljoin, urldefrag, urlparse
import requests
from bs4 import BeautifulSoup

SEEDS = ["https://www.scrapethissite.com/pages/"]
ALLOWED = {"www.scrapethissite.com"}
MAX_PAGES, DELAY = 50, 1.0
UA = "my-crawler/1.0 (+https://example.com/contact)"
PROXY = None   # e.g. "http://<login>:<password>@gate.proxyfield.io:7000"

def normalize(url, base):
    url = urldefrag(urljoin(base, url))[0]          # absolute, no #fragment
    p = urlparse(url)
    if p.scheme not in ("http", "https") or p.hostname not in ALLOWED:
        return None
    return p._replace(netloc=p.hostname, path=p.path or "/").geturl()

s = requests.Session()
s.headers["User-Agent"] = UA
if PROXY:
    s.proxies.update({"http": PROXY, "https": PROXY})

robots = {}
def allowed(url):
    host = urlparse(url).scheme + "://" + urlparse(url).netloc
    if host not in robots:
        rp = urllib.robotparser.RobotFileParser()
        r = s.get(host + "/robots.txt", timeout=30)    # with our own User-Agent and proxy
        rp.parse(r.text.splitlines() if r.status_code == 200 else [])
        robots[host] = rp
    return robots[host].can_fetch(UA, url)

db = sqlite3.connect("crawl.db")
db.execute("CREATE TABLE IF NOT EXISTS pages (url TEXT PRIMARY KEY, status INT, title TEXT, fetched_at REAL)")

frontier = deque(SEEDS)
seen = set(SEEDS)
last_hit = {}
fetched = 0
while frontier and fetched < MAX_PAGES:
    url = frontier.popleft()
    if not allowed(url):
        continue
    host = urlparse(url).hostname
    wait = DELAY - (time.time() - last_hit.get(host, 0))
    if wait > 0:
        time.sleep(wait)                           # politeness per host
    try:
        r = s.get(url, timeout=30)
    except requests.RequestException as e:
        print("error", url, e); continue
    last_hit[host] = time.time()
    fetched += 1
    soup = BeautifulSoup(r.text, "html.parser") if "html" in r.headers.get("Content-Type", "") else None
    title = soup.title.get_text(strip=True) if soup and soup.title else None
    db.execute("INSERT OR REPLACE INTO pages VALUES (?, ?, ?, ?)", (url, r.status_code, title, time.time()))
    db.commit()
    if not soup or r.status_code != 200:
        continue
    for a in soup.select("a[href]"):
        link = normalize(a["href"], url)
        if link and link not in seen:
            seen.add(link)
            frontier.append(link)
print(fetched, "pages fetched,", len(frontier), "URLs left in the frontier")

To use it on your own site, change SEEDS, ALLOWED and UA, then add your field extraction where the title is read. The Python-specific deep dive, with async fetching, is in the Python web crawler guide; a framework that does all of this for you is Scrapy.

Store crawl state and content separately, so you can rerun parsing without refetching.

SQLite is enough for a few hundred thousand URLs on one machine. Past that, Postgres handles the frontier with an index on (status, host, priority), and raw HTML goes to object storage, compressed.

Some sites generate endless URLs: calendars with a "next month" link forever, faceted filters that combine endlessly, session IDs in links, and relative links that build ever-longer paths. Guard against them with a maximum depth, a maximum number of URLs per host, a maximum URL length, and rules that skip parameter patterns you don't need (?sort=, ?filter=, ?sessionid=).

When one machine is not enough, the parts above split into services:

• A shared frontier (Redis, Kafka or Postgres) that all workers read, partitioned by host so one worker owns each host's politeness.
• Async fetchers: one process with asyncio and httpx can keep hundreds of requests in flight across many hosts while staying slow on each. Patterns: how to make web scraping faster.
• Rendering only where needed: send pages that need JavaScript to a separate pool of headless browsers (how to scrape dynamic web pages).
• Recrawl scheduling: revisit pages by how often they change, using the lastmod in sitemaps and the content hash from the last fetch.
• Near-duplicate detection such as SimHash, for templated pages that differ only in a date or an ad.

Build your own when you need to understand every part, crawl a few sites with custom rules, or learn. Use a framework when you need retries, concurrency, exports and middleware on day one: Scrapy (Python) and Crawlee (Node.js and Python) are the common open-source choices, and Apache Nutch and StormCrawler are built for very large crawls. Comparisons: web crawler tools compared. For the difference between crawling and scraping, and how to plan a whole-site crawl, see how to crawl a website.

Crawling at volume? ProxyField rotating residential proxies: 412M+ residential IPs in 195+ countries, a new IP per request, pay per GB from $1.95/GB with no subscription. Start with the Easy Trial (1 GB); traffic never expires, and all purchases are final.