AI Web Scraping: How It Differs From Traditional Scraping

The phrase is used for two different things, and it helps to know which one a page is talking about.

Both meet the same websites, the same blocks and the same rules; the law on both is in is AI scraping legal.

A traditional scraper finds each field by its place in the HTML, with a CSS selector or XPath you write for each site; an AI scraper finds each field by its meaning, from a schema you write once for every site. The difference in code:

# Traditional: one selector per field, per site (Scrapy)
price = response.css(".product-card .price::text").get()

# AI: one schema for every shop; the model finds the field on each page
schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "price": {"type": ["number", "null"]},
        "currency": {"type": "string"},
        "in_stock": {"type": "boolean"},
    },
    "required": ["name", "price", "currency", "in_stock"],
}

The selector line is Scrapy's selector syntax; .get() returns the first match or None. When the shop renames .price, that line returns None until you fix it. The schema goes to the model with the cleaned page text, and the model answers with JSON in that shape. Open-source tools package this step: Crawl4AI, for example, fetches a page, turns it into Markdown and can pass it to an LLM with your schema. Selectors in detail: HTML scraping with CSS selectors and XPath.

How the two behave (typical behaviour of the tools, not measured figures):

The model cost per month, to fill in with your own figures: input tokens per page ÷ 1,000,000 × the model's price per million input tokens × pages per month, plus the output tokens of the JSON. At 20,000 characters (about 5,000 tokens) per page, 100,000 pages a month are about 500 million input tokens; cleaning pages to text before the call is what keeps that number down.

The hybrid most teams end up with: the model writes the selectors, not the data. You ask an AI assistant to generate the scraper for a site, review the code, and run it as an ordinary scraper: cheap and exact per page, and when the layout changes you ask the model to repair the selectors. Keep model-per-page extraction for varied or messy pages, such as hundreds of shops with different layouts.

An AI scraper is a pipeline of 5 steps, and only one of them is AI.

The fetch and clean steps in Python, through a ProxyField residential IP:

import requests
from bs4 import BeautifulSoup

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36"  # use your browser's current string

html = requests.get(
    "https://example.com/product/123",
    proxies={"http": PROXY, "https": PROXY},
    headers={"User-Agent": UA},
    timeout=30,
).text

soup = BeautifulSoup(html, "html.parser")
for tag in soup(["script", "style", "nav", "footer"]):
    tag.decompose()
page_text = soup.get_text(" ", strip=True)[:20000]  # about 5,000 tokens; what you send to the model with the schema

Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.

The extract step: any LLM API with structured output (a JSON schema) takes page_text and the schema. With OpenAI's Python SDK, for example:

import json
from openai import OpenAI  # other providers' structured-output APIs work the same way

schema["additionalProperties"] = False  # strict mode needs a closed schema
client = OpenAI()  # reads OPENAI_API_KEY
resp = client.responses.create(
    model="gpt-5-mini",
    input=[
        {"role": "system", "content": "Extract the product from the page text. Use null for a missing price; never guess."},
        {"role": "user", "content": page_text},
    ],
    text={"format": {"type": "json_schema", "name": "product", "schema": schema, "strict": True}},
)
product = json.loads(resp.output_text)
print(product)  # e.g. {'name': 'Trail Runner 2', 'price': 89.0, 'currency': 'USD', 'in_stock': True}

Then validate before you store: a price that is not a positive number, or a name that does not occur in page_text, sends the page back for a re-run or a review. Step-by-step versions with specific assistants are in can ChatGPT scrape websites and web scraping with Claude.

Vibe scraping is the newest variant: you describe the data in plain language, and an AI coding assistant writes, runs and fixes the scraper for you. It is fast for one-off jobs; for a recurring job, keep the generated code, review it and run it like any other scraper.

Put the proxy in the tool that fetches pages, not in the model: an agent reaches the web only through its tools. Either pass the proxy to the loader or tool you use, or set HTTP_PROXY and HTTPS_PROXY for tools you did not write.

In LangChain, the page loader WebBaseLoader (package langchain-community) takes a proxies dictionary:

from langchain_community.document_loaders import WebBaseLoader

# one sticky US IP for the whole task (30 min); drop -session-...-ttl-... to rotate per request
PROXY = "http://<login>-country-us-session-agent01-ttl-1800:<password>@gate.proxyfield.io:7000"

loader = WebBaseLoader(
    ["https://example.com/pricing", "https://example.com/faq"],
    proxies={"http": PROXY, "https": PROXY},
    requests_per_second=2,
)
docs = loader.load()          # a list of Documents, one per URL
print(docs[0].page_content[:300])

What to expect: docs[0].page_content holds the page text, fetched from a ProxyField US IP; docs[0].metadata["source"] holds the URL. Things that trip people up:

• proxies covers load() only. In the current source the async path (aload, alazy_load) opens its own aiohttp session and ignores proxies. For async loading, set the environment variables below and pass trust_env=True.
• Tools you did not write: use environment variables. requests and httpx read HTTP_PROXY, HTTPS_PROXY and NO_PROXY. List your model provider's API host in NO_PROXY, so LLM calls go direct and only page fetches use (and are counted on) your proxy balance:

export HTTP_PROXY="http://<login>-country-us:<password>@gate.proxyfield.io:7000"
export HTTPS_PROXY="$HTTP_PROXY"
export NO_PROXY="localhost,127.0.0.1,<your model provider's API host>"

• httpx changed its keyword. Current httpx takes httpx.Client(proxy=PROXY); proxies= is the older keyword, and httpx 0.28 and later reject it with TypeError: Client.__init__() got an unexpected keyword argument 'proxies'.
• SOCKS5 works too: the same gateway and credentials as socks5h://<login>-country-us:<password>@gate.proxyfield.io:7000, after pip install "requests[socks]"; with socks5h the proxy resolves the host names. HTTPS pages go through a CONNECT tunnel that the gateway does not decrypt, so no certificate setup is needed.
• Encode special characters in the password inside a proxy URL (@ becomes %40; urllib.parse.quote(password, safe="")).
• A browser tool takes the proxy at launch, for example Playwright's proxy={"server": "http://gate.proxyfield.io:7000", "username": "<login>", "password": "<password>"}.

A fetch tool for the agent. To let the agent decide which pages to open, give it a tool of your own that fetches through the proxy:

import requests
from bs4 import BeautifulSoup
from langchain_core.tools import tool

@tool
def fetch_page(url: str) -> str:
    """Fetch a public web page and return its visible text."""
    r = requests.get(url, proxies={"http": PROXY, "https": PROXY}, timeout=30)
    r.raise_for_status()
    return BeautifulSoup(r.text, "html.parser").get_text(" ", strip=True)[:20000]

print(fetch_page.invoke({"url": "https://api.ipify.org"}))  # expect the proxy's exit IP, not your server's

Pass fetch_page in the agent's list of tools; every page the agent opens then leaves from a ProxyField IP. Only if your network requires the model calls to go through a proxy too, give the chat model its own client: ChatOpenAI(model=..., http_client=httpx.Client(proxy=PROXY)). Otherwise keep the model's API host in NO_PROXY, so model calls are not counted on your proxy balance.

Three checks before an agent runs at volume:

Which proxy type an agent needs, one session per agent and fleet sizing: the guide to proxies for AI agents. Agents connect to scraping tools through MCP servers more and more often: web crawl and scraping MCP servers. For self-hosted agent frameworks, see web scraping with OpenClaw.

Six AI-visibility platforms name Perplexity, and one publishes a plan with hundreds of prompts on it: OtterlyAI Premium, 400 prompts checked daily on a monthly plan. Larger Perplexity prompt sets are on custom Enterprise plans (OtterlyAI from 1,000 prompts, Gauge, GetMentioned). For the raw answers in your own database, a pipeline on a Perplexity scraper API costs $13.50 for 300 prompts a day for 30 days (ProxyField's, in development).

Vendor pages (plans and prices change, so check before you buy):

Pick by prompt count:

Past a platform's quota, or when you want the raw answers in your own database, building your own is the cheaper route. ProxyField's Perplexity scraper API is in development (Contact sales), so the pipeline column is its design and price, not a service you can call today:

ProxyField's scraper APIs are in development: the design is in the API documentation, and access goes through Contact sales. They are billed on successful results only and paid from the same crypto balance as proxies, with no subscription:

All four are on the AI search scraper page.

The other side of AI scraping is the crawlers AI companies run. OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot and Claude-SearchBot, and Common Crawl's CCBot announce themselves by user agent and follow robots.txt rules addressed to them, and many sites now block some or all of them. Two consequences for your own scraping:

• robots.txt is getting longer and stricter. Read it before you scrape, and respect rules that apply to all agents: how to check if a website allows scraping.
• Bot protection is tighter. Sites that fight AI crawlers also challenge other automated traffic more often; the checklist in how to avoid getting blocked while scraping applies to AI scrapers too.

The same rules apply as to any scraping: public data, no logins you are not entitled to, respect for copyright and personal data law, and extra rules when the data trains a model. The detail, with the main court cases and the EU rules, is in is AI scraping legal. ProxyField's acceptable use policy allows public data collection and forbids access to data behind logins you are not entitled to. This is not legal advice.

• Can ChatGPT scrape websites?: what it can fetch, and when you still need a scraper.
• Web scraping with Claude: web fetch, browser use and Claude Code.
• Web scraping with OpenClaw: a self-hosted agent that browses.
• Web crawl and scraping MCP servers: tools AI clients call.
• Web scraping for AI training and RAG: datasets, pipelines and costs.
• Is AI scraping legal?

• Scrapy: selectors: CSS and XPath selectors, .get() and .getall().
• LangChain WebBaseLoader source: the proxies, requests_per_second and trust_env parameters; proxies is applied to the requests session used by load().
• HTTPX: API: the proxy argument.
• Ahrefs Brand Radar, OtterlyAI pricing, Gauge pricing, GetMentioned pricing, Semrush: Brand Performance reports, Peec AI pricing: engines, prompt quotas, prices, check rules and metrics as each vendor states them.
• OpenAI: overview of OpenAI crawlers, Anthropic: crawling and how site owners can block it, Common Crawl: CCBot.