AI Web Crawlers: Crawl4AI and LLM-Ready Crawling

The phrase names two different things, and search results mix them.

This page is about the first kind: tools you run. The second kind matters to you as a site owner, or because sites that block AI companies' bots now challenge other automated traffic more often.

An AI crawler keeps the crawling part of a classic crawler and changes what comes out: text a model can read, instead of raw HTML or fields you mapped by hand.

What does not change: the crawler still downloads every page from a website, so it meets the same rate limits, blocks and robots.txt rules as any other crawler. For the basics of crawling, see what a web crawler is.

Crawl4AI turns pages into input for language models, and its documentation groups the features in five areas:

• Markdown generation. Clean Markdown that keeps headings, lists, tables and code blocks. "Fit Markdown" filters drop menus, footers and boilerplate, and links can become a numbered reference list.
• Structured extraction. CSS and XPath schemas extract fields with no LLM at all; generate_schema can write such a schema once from a description; LLMExtractionStrategy sends the page to a model and asks for JSON that matches your schema.
• Browser control. Chromium, Firefox and WebKit through Playwright, sessions that keep state across steps, persistent profiles, custom headers and user agents, and proxies with authentication and rotation.
• Crawling. Deep crawls (breadth-first, depth-first, best-first) with page and depth limits, URL discovery from sitemaps and Common Crawl, arun_many() for lists of URLs, caching, screenshots and PDFs.
• Ways to run it. A Python library, a command line tool (crwl), and a Docker server with a REST API and an MCP endpoint for AI agents.

Is it free? The library is free and open source under Apache-2.0, and its README asks for attribution (a badge or a link). Your costs are the machines that run the browsers, the model tokens if you use LLM extraction, and proxies once a site starts limiting you. The project also sells a hosted service; this page covers only the open-source library.

Install the package, then let its setup command download the browser once:

pip install -U crawl4ai
crawl4ai-setup      # installs the browser
crawl4ai-doctor     # checks the installation

Crawl4AI's documentation recommends setting the proxy per request, in CrawlerRunConfig.proxy_config. The example below sends the page through the ProxyField gateway and prints the start of the Markdown:

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, ProxyConfig

proxy = ProxyConfig(
    server="http://gate.proxyfield.io:7000",
    username="<login>-country-us-city-new_york",
    password="<password>",
)

async def main():
    config = CrawlerRunConfig(proxy_config=proxy, check_robots_txt=True)
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com", config=config)
        print(result.success, result.url)
        print(result.markdown.raw_markdown[:500])

asyncio.run(main())

Replace <login> and <password> with your credentials. <login>-country-us-city-new_york asks for an exit IP in New York, United States; the later examples write the username as <login>. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.

Three things to know about proxies in Crawl4AI:

• One gateway is enough for rotation. proxy_config also takes a list of proxies, tried in order, and proxy_rotation_strategy rotates through a list. With a rotating gateway you do not need either: in rotating mode, ProxyField gives every request a new IP behind the same address.
• Sessions need the same IP. When a crawl logs in or clicks through steps that belong together, keep one IP with a sticky session (30 seconds to 36 hours on ProxyField) instead of a new IP per request.
• SOCKS5 strings are accepted. ProxyConfig.from_string accepts http://user:pass@host:port and socks5://host:port strings. ProxyField's SOCKS5 runs on the same gate.proxyfield.io:7000 as HTTP and HTTPS, with the same username and password.

A deep crawl starts from one URL, follows the links it finds and returns one result per page. This example stays on the starting domain, goes 2 levels deep, stops at 50 pages and writes each page to a Markdown file:

import asyncio, pathlib, re
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, ProxyConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

async def main():
    config = CrawlerRunConfig(
        deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=2, include_external=False, max_pages=50),
        proxy_config=ProxyConfig(server="http://gate.proxyfield.io:7000", username="<login>", password="<password>"),
        check_robots_txt=True,
    )
    out = pathlib.Path("pages")
    out.mkdir(exist_ok=True)
    async with AsyncWebCrawler() as crawler:
        results = await crawler.arun(url="https://example.com/docs/", config=config)
        saved = 0
        for r in results:
            if r.success:
                name = re.sub(r"[^a-z0-9]+", "-", r.url.lower()).strip("-")[:80]
                (out / f"{name}.md").write_text(f"<!-- {r.url} -->\n" + r.markdown.raw_markdown)
                saved += 1
        print(saved, "pages crawled,", len(results) - saved, "failed")

asyncio.run(main())

Set max_pages before you start: its default is no limit, so the crawl keeps going as long as it finds new links within max_depth. More on depth and limits: how to crawl a website.

What a run gives you (the shape; file names and text come from the site you crawl):

$ python crawl_site.py
48 pages crawled, 2 failed

$ ls pages | head -3
https-example-com-docs.md
https-example-com-docs-getting-started.md
https-example-com-docs-install.md

$ head -6 pages/https-example-com-docs-install.md
<!-- https://example.com/docs/install -->
# Install

Install the package with pip:

    pip install example

Each file is one page: its URL on the first line, then the page's headings, lists, tables and code as Markdown, ready for the RAG steps below.

Crawl from the sitemap instead of following links

When a site publishes a sitemap, list its URLs first and crawl exactly those: you see the page count before you spend anything, and pages that no link reaches are included. Crawl4AI's AsyncUrlSeeder reads the sitemap, pattern keeps one section, and arun_many crawls the list:

import asyncio, pathlib, re
from crawl4ai import AsyncUrlSeeder, AsyncWebCrawler, CrawlerRunConfig, ProxyConfig, SeedingConfig

async def main():
    async with AsyncUrlSeeder() as seeder:
        found = await seeder.urls("example.com", SeedingConfig(source="sitemap", pattern="*/docs/*", max_urls=500))
    urls = [u["url"] for u in found if u["status"] != "not_valid"]
    print(len(urls), "URLs in the sitemap")

    config = CrawlerRunConfig(
        proxy_config=ProxyConfig(server="http://gate.proxyfield.io:7000", username="<login>", password="<password>"),
        check_robots_txt=True,
        stream=True,  # handle each page as it finishes
    )
    out = pathlib.Path("pages")
    out.mkdir(exist_ok=True)
    saved = 0
    async with AsyncWebCrawler() as crawler:
        async for r in await crawler.arun_many(urls, config=config):
            if r.success:
                name = re.sub(r"[^a-z0-9]+", "-", r.url.lower()).strip("-")[:80]
                (out / f"{name}.md").write_text(f"<!-- {r.url} -->\n" + r.markdown.raw_markdown)
                saved += 1
    print(saved, "pages crawled")

asyncio.run(main())

source="sitemap" is the fastest source in Crawl4AI's documentation; max_urls=-1 means no limit. Each item the seeder returns is a dictionary with url and status (valid, not_valid or unknown). The seeder reads the sitemap directly; the page fetches go through the proxy in proxy_config.

Do not embed the crawl as it comes. Four steps make the Markdown retrievable and citable:

import hashlib, re

def rag_chunks(results, max_chars=4000):
    seen, chunks = set(), []
    for r in results:
        if not r.success:
            continue
        md = r.markdown.raw_markdown
        digest = hashlib.sha256(md.encode()).hexdigest()
        if digest in seen:                      # 2. the same page under another URL
            continue
        seen.add(digest)
        title = (r.metadata or {}).get("title") or r.url
        for section in re.split(r"\n(?=#{1,3} )", md):   # 3. one chunk per heading
            for i in range(0, len(section), max_chars):
                text = section[i:i + max_chars].strip()
                if text:
                    chunks.append({"text": text, "url": r.url, "title": title})  # 1. source kept
    return chunks

Each dictionary is ready for your embedding model and vector store (LangChain, LlamaIndex or your own code): embed text and keep url and title as metadata. Re-crawl on a schedule and replace a page's chunks when its hash changes.

Crawl4AI has two ways to turn pages into fields, and the cheaper one does not use a model.

Without an LLM. JsonCssExtractionStrategy takes a schema of CSS selectors (a base selector for each item and a list of fields) and returns JSON. It is fast, exact and costs nothing per page, but you maintain the selectors.

With an LLM. LLMExtractionStrategy sends the page, as Markdown by default, to a model with your schema and instruction. It works with any provider that LiteLLM supports, through a provider string such as openai/gpt-4o-mini or, for a local model, ollama/llama3.3:

import asyncio, os
from pydantic import BaseModel
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig, LLMExtractionStrategy, ProxyConfig

class Product(BaseModel):
    name: str
    price: float
    currency: str
    in_stock: bool

strategy = LLMExtractionStrategy(
    llm_config=LLMConfig(provider="openai/gpt-4o-mini", api_token=os.getenv("OPENAI_API_KEY")),
    # a local model instead: LLMConfig(provider="ollama/llama3.3", api_token="no-token")
    schema=Product.model_json_schema(),
    extraction_type="schema",
    instruction="Extract every product on the page with its price, currency and stock status.",
)

async def main():
    config = CrawlerRunConfig(
        extraction_strategy=strategy,
        proxy_config=ProxyConfig(server="http://gate.proxyfield.io:7000", username="<login>", password="<password>"),
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com/category/shoes", config=config)
        print(result.extracted_content)

asyncio.run(main())

Long pages are split into chunks (chunk_token_threshold, apply_chunking) so they fit the model's context. A practical pattern: use the LLM on a few pages to learn the layout, or let generate_schema write a CSS schema, then run the schema on the rest. Check a sample of the LLM's output against the pages, because models can return plausible values that are not on the page.

Scrapy is the stronger choice for large crawls of plain HTML, and Crawl4AI for JavaScript pages and output that goes straight into a model.

You do not have to pick one for a whole project. A common split: Scrapy crawls and discovers URLs cheaply, and Crawl4AI renders and converts the pages that need a browser or need to go to a model. For Scrapy against Python's simplest parser, see Scrapy vs Beautiful Soup.

Crawl4AI is one of five ways to get a site into LLM-ready Markdown. The choice comes down to who runs the browsers and who pays for them:

• Other open-source AI scrapers. Libraries that send a page to an LLM with a prompt, or let an LLM drive a browser: open-source AI scraping libraries compares them with code.
• ProxyField's own API. The web scraping API is designed to fetch any URL as plain HTTP or with JavaScript rendering, billed on successful results only; it is in development (Contact sales).
• MCP servers for agents. An agent calls a crawler as a tool: web crawl and scraping MCP servers.

For a ranking of AI scraping tools by use case, see AI scraping tools compared.

An AI crawler is still a crawler, and the same rules apply:

• Turn on check_robots_txt=True in Crawl4AI, because it is off by default, and respect the rules that apply to all agents: how to check if a website allows scraping.
• Crawl public pages only, never data behind logins you are not entitled to, as ProxyField's acceptable use policy also requires.
• Keep the pace human-scale and identify a contact if you crawl a site often.
• Using crawled pages to train or ground a model raises copyright and personal-data questions of its own: is AI scraping legal.

Read on October 23, 2026 (the PyPI versions and the max_pages default checked again on October 27, 2026):

• Crawl4AI README (features, install, license and attribution) and package on PyPI (version 0.9.5, Apache-2.0, Python 3.10+)
• Crawl4AI documentation: Proxy & Security, Browser, crawler and LLM config, parameters (check_robots_txt, default off), deep crawling, LLM strategies, extraction without an LLM
• Read on October 27, 2026: Crawl4AI's URL seeding page (AsyncUrlSeeder, SeedingConfig, source="sitemap", the status values) and the Jina Reader README (https://r.jina.ai/ prefix, PDFs, Apache-2.0)
• Scrapy settings: ROBOTSTXT_OBEY and package on PyPI (version 2.19.0, BSD-3-Clause, Python 3.10+)