Web Scraping for AI Training: From Crawl to Clean Dataset

Who crawls for training. OpenAI says GPTBot "is used to crawl content that may be used in training our generative AI foundation models"; Anthropic says ClaudeBot collects "web content that could potentially contribute to their training." Common Crawl's CCBot builds an open archive that "contains petabytes of data, regularly collected since 2008," free to access on Amazon's cloud. Most teams outside the big labs work at the scale of one domain, one language or one product category.

robots.txt is a request from the site, not access control: the RFC says its rules are "not a form of access authorization." AI opt-outs name bots such as GPTBot or CCBot; for your own crawler, follow the rules for * and any group that names it. Python's built-in urllib.robotparser differs from the RFC in two ways: it applies the first matching rule, not the longest, and its read() treats 401 and 403 as disallow-all. Scrapy's default parser, Protego, follows the longest-match rule. How to read a site's file and terms: how to check if scraping is allowed.

1. Define the dataset. Write down the domains, languages, page types and the size you need. Check whether Common Crawl already has the pages; filtering an archive costs less than crawling.
2. Check what each site allows. Read robots.txt as the table above says (cache it for 24 hours at most; a 5xx means stop) and the site's terms, and record any text-and-data-mining reservation.
3. Crawl politely. Use a crawler with a queue, retries and per-site rate limits, such as Scrapy or Crawlee. Identify your bot in the user agent, honor Retry-After on a 429 Too Many Requests, and keep the raw responses.
4. Extract the main content with trafilatura or a similar extractor; keep headings and lists, and store the clean text next to the raw HTML.
5. Filter and clean. Detect the language with fastText, drop thin and repetitive pages, and remove or mask personal data such as names, emails and phone numbers.
6. Deduplicate and store provenance. Hash each text for exact copies and use MinHash for near-copies. For every record keep the URL, the fetch date, the robots.txt status and any license. In the EU, general-purpose AI model providers must publish a summary of their training content, and provenance makes that possible.
7. Refresh and version. Re-crawl on a schedule, keep dataset versions, and record which version trained which model.

Collect (steps 2 to 4 and exact deduplication). It reads robots.txt as RFC 9309 says, through the same proxy as the pages, backs off when a site asks, extracts the main text and writes one JSON line per page:

# pip install requests protego trafilatura
import hashlib, json, os, time
from datetime import datetime, timezone
from urllib.parse import urlsplit
import requests, trafilatura
from protego import Protego

PROXY = os.environ.get("PROXY_URL")  # http://<login>-country-us:<password>@gate.proxyfield.io:7000
PROXIES = {"http": PROXY, "https": PROXY} if PROXY else None
UA = "ExampleDatasetBot/1.0 (+https://example.com/bot)"
ROBOTS_TTL = 24 * 3600  # RFC 9309: don't use a cached robots.txt for more than 24 hours
robots = {}             # host -> (time fetched, rules)

def robots_rules(url):
    host = "{0.scheme}://{0.netloc}".format(urlsplit(url))
    cached = robots.get(host)
    if cached and time.time() - cached[0] < ROBOTS_TTL:
        return cached[1]
    try:
        r = requests.get(host + "/robots.txt", headers={"User-Agent": UA}, proxies=PROXIES, timeout=30)
        if r.status_code == 200:
            rules = Protego.parse(r.text)      # longest match wins, as RFC 9309 says
        elif 400 <= r.status_code < 500:
            rules = "allow"                    # RFC 9309: 4xx means there are no rules
        else:
            rules = "disallow"                 # 5xx: assume complete disallow
    except requests.RequestException:
        rules = "disallow"                     # unreachable: assume complete disallow
    robots[host] = (time.time(), rules)
    return rules

def allowed(url):
    rules = robots_rules(url)
    return rules == "allow" if isinstance(rules, str) else rules.can_fetch(url, UA)

def collect(urls, out="raw.jsonl", delay=2.0):
    seen = set()
    with open(out, "a", encoding="utf-8") as f:
        for url in urls:
            if not allowed(url):
                continue  # the site asks crawlers to stay away: a request, not access control
            r = requests.get(url, headers={"User-Agent": UA}, proxies=PROXIES, timeout=30)
            if r.status_code in (429, 503):
                wait = r.headers.get("Retry-After", "")
                time.sleep(int(wait) if wait.isdigit() else 60)  # back off as the site asks
                continue
            if r.status_code != 200:
                continue
            text = trafilatura.extract(r.text, url=url, include_comments=False)
            if not text or len(text.split()) < 50:
                continue  # empty or thin page
            digest = hashlib.sha256(" ".join(text.lower().split()).encode()).hexdigest()
            if digest in seen:
                continue  # exact duplicate
            seen.add(digest)
            f.write(json.dumps({"id": "sha256:" + digest, "url": url,
                                "fetched": datetime.now(timezone.utc).isoformat(timespec="seconds"),
                                "robots": "allowed", "text": text}, ensure_ascii=False) + "\n")
            time.sleep(delay)

if __name__ == "__main__":
    collect(["https://example.com/docs/page-1", "https://example.com/docs/page-2"])

Filter (steps 5 and 6). It labels the language with fastText's lid.176 model, drops near-copies with MinHash and counts what each filter removed:

# pip install fasttext-numpy2-wheel datasketch  (model: https://fasttext.cc/docs/en/language-identification.html)
import json
import fasttext
from datasketch import MinHash, MinHashLSH

lid = fasttext.load_model("lid.176.bin")
lsh = MinHashLSH(threshold=0.8, num_perm=128)  # pages sharing ~80% of their 5-word shingles count as copies

def minhash(text):
    words = text.lower().split()
    m = MinHash(num_perm=128)
    for i in range(max(1, len(words) - 4)):
        m.update(" ".join(words[i:i + 5]).encode())
    return m

counts = {"read": 0, "language": 0, "near_duplicate": 0, "kept": 0}
with open("raw.jsonl", encoding="utf-8") as src, open("dataset.jsonl", "w", encoding="utf-8") as out:
    for line in src:
        rec = json.loads(line)
        counts["read"] += 1
        labels, probs = lid.predict(rec["text"].replace("\n", " "))
        rec["lang"], rec["lang_score"] = labels[0].replace("__label__", ""), round(float(probs[0]), 2)
        if rec["lang"] != "en" or rec["lang_score"] < 0.65:
            counts["language"] += 1
            continue
        m = minhash(rec["text"])
        if lsh.query(m):
            counts["near_duplicate"] += 1
            continue
        lsh.insert(rec["id"], m)
        rec["words"] = len(rec["text"].split())
        out.write(json.dumps(rec, ensure_ascii=False) + "\n")
        counts["kept"] += 1
print(counts)

We ran both scripts on a small test site in October 2026: an English page, a near-copy of it and a German page. The filter printed {'read': 3, 'language': 1, 'near_duplicate': 1, 'kept': 1}, and the kept record looks like this:

{"id": "sha256:98bd61eb…", "url": "https://example.com/docs/page-1", "fetched": "2026-10-27T02:43:37+00:00",
 "robots": "allowed", "text": "Title\nThis is paragraph 0 of a long documentation page…",
 "lang": "en", "lang_score": 0.91, "words": 145}

Log those counts for every run: they are the funnel from pages fetched to pages kept, and they show which filter to tune. The original fasttext package fails on NumPy 2 with ValueError: Unable to avoid copy while creating an array as requested.; fasttext-numpy2-wheel is the same library built for NumPy 2. Add a license field where a site states one.

For millions of pages, move the same logic into Scrapy, which has robots.txt handling, auto-throttling and retries built in (Scrapy tutorial), or into datatrove, the library FineWeb was built with. <login>-country-us in the proxy username asks for a US IP; other targeting goes after it: -region-<region>, -city-<city>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600.

Retrieval-augmented generation answers a question by first retrieving relevant passages from your own index, then giving them to the model as context. Web scraping keeps that index current. To scrape websites for a RAG pipeline:

RAG crawls are smaller than training crawls but run all the time, and they often need the same page as seen from different countries, such as local prices or regional regulations, which is where geo-targeted IPs come in.

A compliant AI dataset rests on the same rules as any scraping, plus three that are specific to AI:

• Text-and-data-mining opt-outs. In the EU, Directive (EU) 2019/790, Article 4, allows text and data mining unless the rightholder has expressly reserved it "in an appropriate manner, such as machine-readable means in the case of content made publicly available online". Honor robots.txt rules and other machine-readable reservations, and keep a record that you did.
• The AI Act's duties for model providers. Under Regulation (EU) 2024/1689, Article 53, providers of general-purpose AI models need a policy to comply with EU copyright law, including those reservations, and must publish a summary of the content used for training, from August 2, 2025.
• Personal data. Public pages still hold personal data, and the GDPR applies to it: filter it before it reaches the dataset. The European Data Protection Board published Guidelines 03/2026 on web scraping in the context of generative AI on July 8, 2026, for public consultation; read the current version, because the text is not final.
• Copyright outside the EU. In the US, courts have decided fair use for AI training case by case.

The full legal picture, with the cases: is AI scraping legal. The general rules for any scraper: is web scraping legal.

Start with rotating datacenter proxies and move only the domains that block them to residential proxies. Datacenter IPs cost the least per GB and suit bulk crawls of sites that accept them; residential IPs get through sites that block or challenge datacenter ranges, and give you country, region and city targeting; dedicated ISP IPs fit sources that allowlist your addresses or sessions that must keep one IP for weeks. Proxies spread requests over many IPs and locations; they do not make a crawl lawful or polite. Pacing and the checks above do that.

One job, priced four ways: 1,000,000 pages at 100 KB of HTML each is about 100 GB.

The mixed row assumes a tenth of the traffic needs residential IPs; measure the real share on your targets. The tier rate applies to the whole purchase, and fetching HTML only, without images and scripts, keeps the page size low. Rotating datacenter proxies cover the same countries as residential and are targeted by country only.

What to look for in a proxy for AI data collection:

• Per-GB pricing that suits bursts: no subscription, and traffic that never expires, so a quarterly re-crawl uses what is left from the last one. All purchases are final.
• Location: country, region and city targeting at the same price per GB, for datasets and RAG indexes that should reflect one market.
• Sessions: a new IP on every request for bulk crawls, or a sticky IP from 30 s to 36 h for paginated sources.
• Scale: unlimited concurrent threads, HTTP, HTTPS and SOCKS5 on one gateway, gate.proxyfield.io:7000, and 412M+ residential IPs in 195+ countries.
• Rules you can follow: public pages only, following each site's terms (proxies for web scraping).

Or skip the proxies. For a smaller project, a scraping API can replace running proxies and a crawler's retry logic. ProxyField's Web Scraping API is in development (Contact sales). Its prices are set: $0.50 per 1,000 successful results over plain HTTP, or $0.20 from 10,000 results in one purchase, and $2.00 down to $1.00 per 1,000 with JS rendering. The same 1,000,000 pages would cost $200 over plain HTTP.

Scraped data suits machine learning when it is cleaned, labeled and documented. Common projects:

• Sentiment and review analysis: product reviews with their star ratings as labels.
• Price prediction: product listings with prices over time.
• Classification: news articles or job ads with their categories.
• NLP corpora: forum threads, documentation or articles in one language or field, for fine-tuning or evaluation.
• Entity extraction: company pages and directories for names, addresses and products.

Each starts with the same crawl-extract-clean steps above; the difference is which fields you keep as labels. More ideas: web scraping projects.

Get started: run your dataset crawl through residential proxies. Start with the Easy Trial: 1 GB for $2.95, no subscription; traffic never expires.

• Penedo et al., The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (arXiv 2406.17557, June 2024, revised October 2024)
• Soldaini et al., Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research (arXiv 2402.00159)
• RFC 9309, Robots Exclusion Protocol (September 2022)
• Common Crawl, Get started (WARC, WAT and WET files), Overview and CCBot
• OpenAI, Overview of OpenAI Crawlers
• Anthropic Help Center, Does Anthropic crawl data from the web? (updated April 7, 2026)
• fastText, Language identification
• Directive (EU) 2019/790
• AI Act, Regulation (EU) 2024/1689
• EDPB, Guidelines 03/2026 on web scraping in the context of generative AI (July 8, 2026, public consultation)

This is not legal advice. Laws and guidance change; the primary sources above are the authority.