Website Crawler Tools Compared

Licenses, versions, prices and limits were checked on the projects' and vendors' own sites and repositories on October 23 and 25, 2026.

Proxy support: Scrapy, Crawlee, Colly and Wget take a proxy URL (Scrapy and Wget also read the https_proxy variable), Screaming Frog has a proxy setting, and Nutch, StormCrawler and Heritrix take a proxy host and port in their configuration.

Start with the site's XML sitemap (listed in robots.txt or at /sitemap.xml), then crawl its links to find the pages the sitemap misses. Four ways to get the full list:

1. Screaming Frog SEO Spider (no code): enter the start URL, let the crawl finish, then open the Internal tab and export the URLs to CSV or a spreadsheet. It can also crawl an XML sitemap on its own or check an uploaded list of URLs. The free version stops at 500 URLs per crawl; the paid licence, £199 or €245 per year, removes the limit and lets you save crawls.
2. Semrush or Ahrefs Site Audit (hosted, no install): if you already pay for one of them, its audit crawls from the home page, the robots.txt sitemap or an uploaded URL list, and lists the crawled pages. Semrush allows 20,000 pages per audit on Pro and Guru and 100,000 on Business; Ahrefs gives 5,000 crawl credits per project per month free for a site whose ownership you verify.
3. GNU Wget (one command, small sites): --spider visits pages without saving them and prints each URL it checks.
4. Scrapy (code, any size): a CrawlSpider follows every internal link once and writes each URL with its status code.

The Wget command, with the lines reduced to page URLs (images, CSS and scripts dropped):

export https_proxy="http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"  # optional
wget --spider --recursive --level=1 --wait=1 --no-verbose -e robots=on https://books.toscrape.com/ 2>&1 \
  | grep -oE 'URL: ?https?://[^ ]+' | sed -E 's/^URL: ?//' | grep -vE '/(static|media)/' | sort -u > urls.txt

We ran it without the proxy on October 25, 2026 (Wget 1.21.4) against books.toscrape.com, a public scraping sandbox: it downloaded 77 files in about 3.5 minutes, reported "Found no broken links." and left 74 page URLs in urls.txt. Replace --level=1 with --level=inf to follow links to any depth.

The Scrapy spider (save as list_urls.py):

from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class ListUrls(CrawlSpider):
    name = "list_urls"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]
    rules = [Rule(LinkExtractor(), callback="parse_item", follow=True)]
    custom_settings = {"ROBOTSTXT_OBEY": True, "DOWNLOAD_DELAY": 1, "AUTOTHROTTLE_ENABLED": True}

    def parse_start_url(self, response):
        return self.parse_item(response)

    def parse_item(self, response):
        yield {"url": response.url, "status": response.status}

Run it with scrapy runspider list_urls.py -O urls.csv. Scrapy reads the same https_proxy variable, so the proxy line above applies to it too. Our test run on October 25, 2026 (Scrapy 2.19), stopped early with -s CLOSESPIDER_ITEMCOUNT=60, wrote 76 rows such as https://books.toscrape.com/catalogue/page-2.html,200 in about 2 minutes; without the limit it runs until no new links are left. Scrapy skips URLs it has already seen, so each URL appears once. The full method, with sitemaps and duplicate URLs: how to crawl a website.

Yes, there are: Apache Nutch, StormCrawler and Heritrix are free under the Apache-2.0 license and built for crawls of millions of pages across many sites. The software costs nothing; the servers, bandwidth, storage and the team to run them are the real cost, so a crawl at this scale needs a cluster and someone to operate it. If the pages you need are already public, Common Crawl publishes crawls of billions of pages as free open data, which can save the crawl altogether.

All four fetch plain HTML by default. StormCrawler can crawl with a headless browser through its Playwright or Selenium protocol, Scrapy through an add-on; plan for that only on the pages that need it.

How to start each one, from the projects' own guides (October 2026):

# Apache Nutch 1.x (binary release, Java installed): set http.agent.name in conf/nutch-site.xml first
mkdir -p urls && echo "https://example.com/" > urls/seed.txt
bin/crawl -s urls crawl 2            # usage: crawl [options] <crawl_dir> <num_rounds>

# StormCrawler: generate a project from the archetype, then follow its README
mvn archetype:generate -DarchetypeGroupId=org.apache.stormcrawler \
  -DarchetypeArtifactId=stormcrawler-archetype -DarchetypeVersion=3.7.0

# Heritrix (Java 17+): start it, then open https://localhost:8443/ and create a crawl job
$HERITRIX_HOME/bin/heritrix -a admin:CHANGE_ME

Guides: the Nutch tutorial, StormCrawler getting started and Heritrix getting started.

Scrapy is a Python framework that crawls and scrapes in one: it follows links, keeps a queue of URLs, skips duplicates, retries failures, throttles itself per site and exports items to JSON or CSV. Its CrawlSpider follows links by rules, such as "every URL under /products/". It does not render JavaScript by default; an add-on connects it to a headless browser. Introduction: what is Scrapy; proxy setup: Scrapy proxy setup.

Crawlee is a crawling and scraping library for Node.js and Python with plain HTTP crawlers and browser crawlers (Playwright, Puppeteer) behind one interface, plus queues, retries and sessions. Its ProxyConfiguration rotates through a list of proxy URLs; with a rotating gateway, one URL is enough. It suits sites that need a browser for some pages and not others.

Colly is a crawling framework for Go: fast, light on memory, with callbacks for each element, rate limits per domain and a proxy switcher. It does not render JavaScript. A good fit if your team works in Go; see Golang web scraping.

GNU Wget crawls from the command line. With --spider it visits pages without saving them and reports broken links, as in the URL-list command above; with --mirror it copies a site for offline use. HTTrack does the mirroring job with a graphical interface on Windows. Both suit small sites; neither runs JavaScript.

SEO crawlers crawl one site the way a search engine would and report what hurts its search performance: broken links, redirect chains, missing or duplicate titles and descriptions, thin pages, canonical and hreflang errors, orphan pages and crawl depth. Screaming Frog SEO Spider runs on Windows, macOS and Linux, is free up to 500 URLs per crawl, costs £199 or €245 per year without the limit, and exports its URL lists to CSV; Sitebulb runs on Windows and macOS, with a cloud version for very large sites, and has paid plans only. Hosted audits crawl on a schedule and share reports with a team: Semrush Site Audit (20,000 pages per audit on Pro and Guru, 100,000 on Business) and Ahrefs Site Audit (free for a verified site, 5,000 crawl credits per project per month). Many SEO crawlers render JavaScript and extract custom fields, which makes them simple scrapers too. Running one through proxies, for example to see a site from another country: proxies for SEO and proxies in SEO software.

Online website crawlers run in your browser: you paste a URL and get a list of pages, links or errors, with a free tier limited to a small number of URLs. They suit a quick check, not a recurring job. Crawling APIs take a start URL and return the crawled pages or their content, running the IPs and browsers on the provider's side, billed per page. For single pages, ProxyField's web scraping API fetches any URL as plain HTTP for $0.50 down to $0.20 per 1,000 successful results, or with JavaScript rendering for $2.00 down to $1.00, billed on successful results only. Contact sales.

Common Crawl is a nonprofit that crawls the web and publishes the results as free, open data: raw pages (WARC files), extracted text and metadata, with an index you can query by URL. Its crawler identifies itself as CCBot. If your question is about the web at large, such as which sites link to a domain or how often a phrase appears, start with Common Crawl before running a crawler of your own. Its data is a snapshot, so for current prices or rankings you still need a fresh crawl.

How crawlers work in general: what is a web crawler. Building your own: how to build a web crawler.