Free AI Web Scrapers: Open-Source Tools

Versions on October 23, 2026: Crawl4AI 0.9.5, ScrapeGraphAI 2.3.1, Browser Use 0.13.11, LLM Scraper 2.0.0. All four are free for commercial use under their licenses. Crawl4AI's README asks for attribution (a badge or a link) when you use it.

An AI scraper has three costs, and an open-source library removes only the first:

Hosted AI scraping apps often have free tiers too, limited by credits or pages per month; check each one's limits before you build on it. For free scrapers without AI (libraries, extensions, spreadsheet formulas), see free web scraping tools.

Crawl4AI is the first tool to try when you want Python, no paid API key and more than a handful of pages. It crawls with a headless browser, writes each page as clean Markdown, and extracts JSON in one of two ways: CSS or XPath schemas with no model at all, or LLMExtractionStrategy, which sends the Markdown to any model LiteLLM supports, including a local one through Ollama. A local model needs no API token: the README's hint is LLMConfig(provider="ollama/llama3.3", api_token="no-token").

Setup, with Ollama installed and running:

pip install -U crawl4ai
crawl4ai-setup          # installs the browser Crawl4AI drives
crawl4ai-doctor         # optional: checks the install
ollama pull llama3.3    # or a smaller model, such as llama3.2, on a laptop

Extraction with the local model, through a residential proxy:

import asyncio
from pydantic import BaseModel
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig, LLMExtractionStrategy, ProxyConfig

class Product(BaseModel):
    name: str
    price: float

strategy = LLMExtractionStrategy(
    llm_config=LLMConfig(provider="ollama/llama3.3", api_token="no-token"),  # local model, no key, no per-page bill
    schema=Product.model_json_schema(),
    extraction_type="schema",
    instruction="Extract every product on the page with its name and price.",
    input_format="markdown",
)

async def main():
    config = CrawlerRunConfig(
        extraction_strategy=strategy,
        proxy_config=ProxyConfig(server="http://gate.proxyfield.io:7000",
                                 username="<login>-country-us", password="<password>"),
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com/category/shoes", config=config)
        print(result.markdown[:500])        # the page as Markdown
        print(result.extracted_content)     # JSON from the local model

asyncio.run(main())

What to expect: the first run is slow because the browser and the model are downloaded once. llama3.3 is a 70-billion-parameter model and needs a machine with a lot of memory; on a laptop, start with a small model such as llama3.2 and move up only if fields come back wrong. result.extracted_content is a JSON string, so check its fields against the page before you store them. Version 0.9.5 was current on October 23, 2026. Deep crawls, CSS schemas and a comparison with Scrapy are in AI web crawlers: Crawl4AI.

The simplest AI scraper is 4 steps in about 30 lines: fetch the page, strip it to text, ask a model for JSON that matches a schema, and check the result. This version uses Requests and Beautiful Soup for the page and a local model through Ollama, so nothing is billed per page except the proxy traffic:

# pip install requests beautifulsoup4   (and install Ollama, then: ollama pull llama3.1)
import json
import requests
from bs4 import BeautifulSoup

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
SCHEMA = {
    "type": "object",
    "properties": {"products": {"type": "array", "items": {
        "type": "object",
        "properties": {"name": {"type": "string"}, "price": {"type": "number"}, "currency": {"type": "string"}},
        "required": ["name", "price"]}}},
    "required": ["products"],
}

# 1. Fetch the page through a residential IP
html = requests.get("https://example.com/category/shoes", proxies={"http": PROXY, "https": PROXY}, timeout=30).text

# 2. Clean it: drop scripts, styles and navigation, keep the text
soup = BeautifulSoup(html, "html.parser")
for tag in soup(["script", "style", "nav", "header", "footer"]):
    tag.decompose()
text = soup.get_text(" ", strip=True)[:12000]

# 3. Ask the local model for JSON that matches the schema
reply = requests.post("http://localhost:11434/api/chat", json={
    "model": "llama3.1",
    "messages": [{"role": "user", "content": "Extract every product with its name, price and currency:\n\n" + text}],
    "format": SCHEMA,
    "stream": False,
    "options": {"temperature": 0},
}, timeout=300).json()
data = json.loads(reply["message"]["content"])

# 4. Validate before you store anything
rows = [p for p in data["products"] if p.get("name") and isinstance(p.get("price"), (int, float)) and p["price"] > 0]
print(len(rows), "products", rows[:3])

Replace <login> and <password> with your credentials. <login>-country-us-city-new_york asks for an exit IP in New York, United States. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.

How this scales: Ollama's format field takes a JSON schema and makes the model answer in that shape. To use a hosted model instead, send the same text and schema to its API with your own key; the fetch, clean and validate steps stay the same. For pages built by JavaScript, swap step 1 for a headless browser such as Playwright. More Python options: Python proxy setup.

ScrapeGraphAI is a Python library that builds a scraping pipeline from a prompt: you give it a URL and a sentence such as "list every product with its price", and it fetches the page, sends it to the model and returns a dictionary. Its main pipelines are SmartScraperGraph (one page), SmartScraperMultiGraph (a list of pages), SearchGraph (the top results of a search engine) and ScriptCreatorGraph, which writes a Python scraper for you instead of extracting.

It fetches pages with Playwright, so install the browser after the package, and it takes a proxy through loader_kwargs:

# pip install scrapegraphai && playwright install
from scrapegraphai.graphs import SmartScraperGraph

graph_config = {
    "llm": {"model": "ollama/llama3.2", "model_tokens": 8192, "format": "json"},
    # hosted model instead: {"api_key": "YOUR_OPENAI_API_KEY", "model": "openai/gpt-4o-mini"}
    "loader_kwargs": {
        "proxy": {
            "server": "http://gate.proxyfield.io:7000",
            "username": "<login>-country-us-city-new_york",
            "password": "<password>",
        },
    },
    "headless": True,
}

scraper = SmartScraperGraph(
    prompt="List every product on the page with its name and price",
    source="https://example.com/category/shoes",
    config=graph_config,
)
print(scraper.run())

What to know: the library needs Python 3.12 or newer, and you bring your own model (a hosted API key or Ollama). Its README says proxies and anti-bot handling are your responsibility.

Browser Use gives a model a real browser: you describe a task, and the agent clicks, types, scrolls and reads pages until it has the answer. That suits jobs a fixed scraper handles badly, such as searching a site, filling a form or paging through results whose layout you do not know in advance. It is slower and uses more tokens per page than extraction from a fetched page, so it is not the tool for 10,000 product pages.

# uv add browser-use   (Python 3.11+; and install Ollama, then: ollama pull qwen3)
import asyncio
from browser_use import Agent, Browser, ChatOllama
from browser_use.browser.profile import ProxySettings

async def main():
    browser = Browser(proxy=ProxySettings(
        server="http://gate.proxyfield.io:7000",
        username="<login>-country-us-city-new_york",
        password="<password>",
    ))
    agent = Agent(
        task="Open https://example.com/category/shoes and return the first 10 products with name and price as JSON",
        llm=ChatOllama(model="qwen3"),  # or ChatOpenAI(...) / ChatAnthropic(...) with your own API key
        browser=browser,
    )
    history = await agent.run()
    print(history.final_result())

asyncio.run(main())

The library is free and MIT-licensed, and its README shows how to run a local browser with a local model through Ollama. Hosted models bill tokens for every step. Multi-step browsing is hard for small models, so anything beyond simple tasks often needs a larger local model or a hosted one.

LLM Scraper turns a Playwright page into typed data: you define a Zod schema, and the model fills it from the page's HTML, Markdown, text or a screenshot. It works with models through the provider packages of Vercel's ai library, including OpenAI, Anthropic, Google, Groq and Ollama, and it can also generate a reusable Playwright script from the schema. The proxy goes into Playwright's launch options:

// npm i zod playwright llm-scraper ai ollama-ai-provider-v2
import { chromium } from 'playwright'
import { z } from 'zod'
import { Output } from 'ai'
import { ollama } from 'ollama-ai-provider-v2'
import LLMScraper from 'llm-scraper'

const browser = await chromium.launch({
  proxy: { server: 'http://gate.proxyfield.io:7000', username: '<login>-country-us-city-new_york', password: '<password>' },
})
const scraper = new LLMScraper(ollama('llama3'))
const page = await browser.newPage()
await page.goto('https://news.ycombinator.com')

const schema = z.object({
  top: z.array(z.object({ title: z.string(), points: z.number() })).length(5).describe('Top 5 stories'),
})
const { data } = await scraper.run(page, Output.object({ schema }), { format: 'html' })
console.log(data.top)

await page.close()
await browser.close()

Full Node.js proxy setup: Node.js proxy guide.

• You are learning, or you want the lowest cost per page: the do-it-yourself Python script above, with a local model.
• You want Python, a local LLM and no API key: Crawl4AI with Ollama, as above.
• You need a whole site as Markdown for RAG or a dataset: Crawl4AI.
• You want JSON from a few pages by writing one sentence: ScrapeGraphAI.
• The task needs clicks, searches or forms: Browser Use.
• Your stack is Node.js or TypeScript: LLM Scraper.
• You would rather not run browsers, models or IPs at all: a hosted scraping API, billed per result. ProxyField's web scraping API fetches any URL as plain HTTP or with JavaScript rendering, billed on successful results only.

Paid AI scraping tools, and how to judge them, are in AI scraping tools compared.

Read on October 23, 2026:

• Crawl4AI README and PyPI (0.9.5, Apache-2.0, Python 3.10+)
• ScrapeGraphAI README (pipelines, models, proxies, MIT), its proxy example and PyPI (2.3.1, Python 3.12+)
• Browser Use README (Python 3.11+, MIT, models, local Ollama), its ProxySettings and PyPI (0.13.11)
• LLM Scraper README (2.0.0, MIT, formats, providers, code generation)
• Ollama API: structured outputs (the format field takes a JSON schema)