What Is a Web Crawler?

A web crawler is software that browses the web by following links, to find and download pages. "Crawling" is what it does: moving from page to page the way a spider moves along its web. The crawler itself does not decide what a page means; it collects pages, and other software indexes, analyzes or extracts data from them. In computing, "crawler" almost always means a web crawler (the word also names cranes, tracked vehicles and TV news tickers).

Every crawler has the same 6 parts:

There is no central list of every page on the web, so a crawler discovers pages from 3 sources:

A page that no other page links to and no sitemap lists (an "orphan" page) is hard for any crawler to find.

Then the crawler repeats one loop until its queue is empty or it reaches a limit:

1. Take the next URL from the frontier. A breadth-first crawler takes the oldest; a priority crawler takes the most important.
2. Check robots.txt for that host (cached, refreshed at least every 24 hours) and skip the URL if it is disallowed for this crawler.
3. Fetch the page, waiting long enough since the last request to the same host.
4. Store the page or the fields you need from it.
5. Extract the links, turn them into absolute URLs, drop duplicates and URLs outside the crawl's scope, and add the rest to the frontier.
6. Repeat, and schedule revisits, so pages that change often are fetched again sooner.

A crawler that followed every link would never stop, so real crawlers set a scope (one site, one section, a list of domains), a depth limit and a page limit. They also skip URL traps: endless calendars, faceted filters and tracking parameters that create new URLs for the same page. For the architecture with working Python code, see how to build a web crawler and the Python web crawler guide; for crawling one site end to end, see how to crawl a website.

Crawlers differ by what they crawl and why.

The crawler names and their purposes are documented by their operators: Google's overview of Google crawlers, Bing's crawlers, Applebot, OpenAI's bots, Common Crawl's CCBot and Heritrix (pages as published in October 2026). Tools for running your own crawls are compared in our guide to website crawler tools.

A polite crawler obeys robots.txt, slows down when the server asks, identifies itself and never goes past logins or technical blocks. robots.txt is a request to crawlers, not a lock: RFC 9309 says its rules "are not a form of access authorization" (§1), and listing a path in it makes that path public (§3).

robots.txt rules, with the numbers (RFC 9309)

The Robots Exclusion Protocol was standardized as RFC 9309 in September 2022.

Longest match in practice:

User-agent: *
Disallow: /shop/
Allow: /shop/public/
Disallow: /*?sessionid=

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Here /shop/public/item-1 is allowed (the 13-character Allow beats the 6-character Disallow), /shop/cart is not, any URL with ?sessionid= is not, and GPTBot may fetch nothing. Google follows the same rules and documents its own limits: it caches robots.txt for up to 24 hours, follows at least 5 redirect hops, ignores content past 500 KiB, and treats every 4xx except 429 as "no robots.txt" (Google's robots.txt specification).

Crawl-delay, 429 and Retry-After

• Crawl-delay is not part of RFC 9309. Some crawlers read it; Google says "other fields such as crawl-delay aren't supported" and offers no robots.txt delay at all (Google's robots.txt specification). Bing documents its own crawl-rate setting in Bing Webmaster Tools (Bing crawl control). If a file sets Crawl-delay: 10, a polite crawler waits 10 seconds between requests to that host.
• No delay stated? Pace yourself anyway: one request at a time per host, and back off when responses slow down. That is the common practice, not a number from a standard.
• 429 Too Many Requests and 503 Service Unavailable mean stop and wait. If the response carries Retry-After (seconds or a date), wait at least that long (MDN: 429); without it, double the wait after each one. More in 429 Too Many Requests.

A minimal check in Python's standard library:

import time, urllib.robotparser, requests

UA = "ExampleCrawler/1.0 (+https://example.com/bot)"
rp = urllib.robotparser.RobotFileParser("https://example.com/robots.txt")
rp.read()

def polite_get(url, wait=1.0):
    if not rp.can_fetch(UA, url):
        return None                            # disallowed for this user agent
    time.sleep(rp.crawl_delay(UA) or wait)     # honor Crawl-delay if the site sets one
    r = requests.get(url, headers={"User-Agent": UA}, timeout=30)
    if r.status_code in (429, 503):
        retry = r.headers.get("Retry-After", "60")
        time.sleep(int(retry) if retry.isdigit() else 60)
    return r

One caveat: Python's urllib.robotparser applies rules in file order (the first match wins), not RFC 9309's longest match, so a file with overlapping Allow and Disallow rules can parse differently; it also treats a 401 or 403 on robots.txt as "disallow everything", where RFC 9309 treats every 4xx as "no rules". For strict RFC behavior, use a parser that implements it, such as Google's open-source robotstxt library.

Beyond robots.txt

• An honest user agent with a contact URL; never pose as Googlebot or another known crawler.
• Respect access controls: logins, paywalls, CAPTCHA pages and rate limits are the site saying no, whatever robots.txt says.
• Crawl only what you need: a scope, a depth limit, a page limit, and no URL traps.
• Cache and revisit by change rate, instead of refetching everything.
• Personal and copyrighted data need care whatever the crawl rules; our guide is web scraping legal covers when a crawl is allowed.

Crawling finds and downloads a page; indexing analyzes it and decides whether to store it so it can be searched. Google describes 3 stages (Google Search Central: how Search works):

Three points that confuse site owners:

• A robots.txt disallow does not remove a page from the index. Google says "a page that's disallowed in robots.txt can still be indexed if linked to from other sites", shown without a description (Google: robots.txt introduction).
• To keep a page out of results, use noindex (a meta tag or an HTTP header), and leave the page crawlable: if robots.txt blocks it, Google never sees the noindex.
• Crawled does not mean indexed. In Google Search Console, the URL Inspection tool shows whether a URL is crawled and indexed, and the Page indexing report lists statuses such as "Crawled - currently not indexed" and "Discovered - currently not indexed" (Search Console: Page indexing report).

A crawler finds pages; a scraper extracts data from them. Most data projects use both: the crawler builds the URL list, the scraper reads the fields from each page. The full comparison, with when to use which, is in our guide on web crawling vs web scraping.

Crawlers announce themselves in the User-Agent header, such as Googlebot or bingbot, but anyone can copy that string. To check that a visit really comes from a search engine, do a reverse DNS lookup on the IP, check that the host name belongs to the search engine's domain, then do a forward lookup to confirm it resolves back to the same IP. Google documents this method for Googlebot (Google Search Central: verifying Googlebot), and some operators also publish their IP ranges.

A crawler sends many requests to one site, so it meets the same limits as any scraper: rate limits per IP, blocks on IP ranges with a poor reputation, and CAPTCHAs when traffic looks automated. For a few hundred pages, a polite delay is enough. For large crawls:

• Spread requests over many IPs. Rotating proxies give each request a new IP, and a sticky session keeps one IP for steps that must share it, such as paging through search results. Spreading load is not a reason to ignore robots.txt or a site's 429s.
• Use datacenter IPs where the site allows them. Rotating datacenter proxies cost from $0.22/GB at volume; residential proxies suit sites that block datacenter ranges.
• Estimate the traffic: pages × average page size. 50,000 pages of 150 KB is about 7.5 GB, which costs $22.13 at $2.95/GB (the 1–9 GB residential tier).

ProxyField works with Scrapy, Crawlee, Playwright, Puppeteer, Selenium and Python requests over HTTP, HTTPS and SOCKS5, with unlimited concurrent threads, country, region and city targeting, and sticky sessions from 30 s to 36 h. You pay per GB with no subscription, the traffic never expires, and all purchases are final. Cost per job is worked out on proxies for web scraping.