Web Scraping With Claude

Yes, but which Claude you use decides how far it goes. A chat can read pages for you; Claude Code can build and run a scraper; the API tools let you build Claude into your own pipeline. The table above sums it up; the sections below show how each one works and where a proxy fits.

One rule holds for all of them: Claude is good at reading pages, extracting fields and writing scraping code, and the volume work, a queue of thousands of URLs that runs every day, belongs in a script.

On the Claude API, two server tools bring web content into the conversation:

• Web fetch "allows Claude to retrieve full content from specified webpages and PDF documents." Claude fetches only URLs that already appeared in the conversation, and the tool "currently does not support websites dynamically rendered with JavaScript." You can cap it with max_uses, limit it to allowed_domains and truncate long pages with max_content_tokens. Web fetch costs nothing beyond standard token costs.
• Web search finds pages and answers with citations, for $10 per 1,000 searches plus token costs.

Both run on Anthropic's side, so you cannot add a proxy to them, and both suit a few pages per request, not a crawl. When a question to Claude needs a web page, Anthropic says Claude "may access websites using a Claude-User agent," which site owners can block in robots.txt.

Claude Code is where Claude becomes a practical scraper, because it runs on your machine with the same network access as any other program. Two of its tools matter here:

• WebFetch fetches a URL, converts the page to Markdown and runs your extraction prompt over it in a separate model call. Anthropic calls it "lossy by design": Claude receives the extracted answer, not the raw page. It reads up to 100,000 characters per call and caches each response for 15 minutes.
• Bash runs anything you could run in a terminal, including curl for the raw HTML and the Python or Node.js scrapers Claude writes.

A good session looks like this: ask Claude to fetch one page with curl and save it, let it study the HTML and write a scraper, run the scraper on a small sample, check the output, then run the full job. Claude fixes errors as they come up, and you review each command before it runs.

Put the proxy in the scraper, not in Claude Code's environment. Claude Code respects the standard HTTPS_PROXY and HTTP_PROXY variables for its own traffic, so setting them before you launch it would also send Claude Code's connection to Anthropic through the proxy and spend proxy traffic on it. Give the proxy to the script instead:

# scrape.py: written by Claude, run by Claude Code through Bash
# pip install requests beautifulsoup4
import json, os, sys
import requests
from bs4 import BeautifulSoup

PROXY = os.environ["SCRAPER_PROXY"]  # http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000
proxies = {"http": PROXY, "https": PROXY}

for url in sys.argv[1:]:
    html = requests.get(url, proxies=proxies, timeout=30).text
    soup = BeautifulSoup(html, "html.parser")
    title = soup.title.string if soup.title else None
    print(json.dumps({"url": url, "title": title, "h1": [h.get_text(strip=True) for h in soup.find_all("h1")]}))

Run it with SCRAPER_PROXY=... python scrape.py https://example.com/a https://example.com/b, so only the scraper's requests use the proxy. <login>-country-us-city-new_york in the username asks the ProxyField gateway for a New York IP; other targeting in the username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default) after the location, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600. For JavaScript-heavy pages, ask Claude for a Playwright version; the proxy setup is in Playwright proxy setup.

A skill is a folder with a SKILL.md file of instructions, plus any scripts and reference files it needs. Claude loads the skill's name and description at startup and reads the rest only when a request matches, so a scraper skill costs almost nothing until you use it. In Claude Code, personal skills live in ~/.claude/skills/ and project skills in .claude/skills/.

A minimal scraper skill, .claude/skills/page-scraper/SKILL.md:

---
name: page-scraper
description: Scrape public web pages into JSON lines with the bundled script. Use when the user asks to scrape, extract or collect data from one or more URLs.
---

# Page scraper

1. Check that the user named public pages, and read the site's robots.txt first.
2. Run `python scripts/scrape.py <url> [<url> ...] > out.jsonl`. The script reads the
   proxy from the SCRAPER_PROXY environment variable; never print its value.
3. Fetch at most one page every 2 seconds per site. Stop on 403 or 429 responses
   and report them instead of retrying in a loop.
4. Show the user the first 5 records, then write the rest to the file they asked for.

Put the script from the previous section in .claude/skills/page-scraper/scripts/scrape.py, and ask Claude to "scrape these three URLs". Where the skill can reach the network depends on where it runs: in Claude Code it has your machine's network access; skills on the Claude API run "in a sandboxed container with no network access"; on claude.ai, network access depends on your settings. Before you install a skill someone else wrote, read every file in it, as Anthropic recommends, and look for network calls that do not match its purpose.

The Model Context Protocol lets Claude call tools on an MCP server, and browser servers give it a real browser. To add Microsoft's Playwright MCP server to Claude Code:

claude mcp add playwright npx @playwright/mcp@latest

Claude can then open pages, take accessibility snapshots, click and read rendered content. To route that browser through a proxy, start the server with its --proxy-server option, for example --proxy-server=http://gate.proxyfield.io:<port> with your server's IP on the ProxyField allowlist, where <port> is the port you set up in the dashboard, or pass a username and password through the server's configuration file. Setup for each client, other servers and how to choose: MCP servers for scraping.

Claude Code can connect to the Claude in Chrome extension and work in your own browser: it "opens new tabs for browser tasks and shares your browser's login state." Anthropic lists data extraction among its uses: "pull structured information from web pages and save it locally." The actions run in a visible Chrome window, and "when Claude encounters a login page or CAPTCHA, it pauses and asks you to handle it manually."

This is the right tool for small jobs on sites you use, such as a report inside a web app you have an account with. Collect only data your account is entitled to, and check the site's terms first. Because the extension uses your browser, the requests come from your IP, or from whatever proxy your Chrome uses (Chrome proxy setup).

Websites can block Anthropic's own robots by name. Anthropic lists three: ClaudeBot collects content that may be used for training, Claude-User fetches pages when users ask Claude something, and Claude-SearchBot improves search results. All three honor robots.txt, and Anthropic says its bots "respect anti-circumvention technologies."

Scrapers that Claude writes are a different matter: they run as ordinary programs from your machine, so sites see your IP and your request rate. A few pages work from your own connection. Hundreds or thousands from one site will run into rate limits and errors such as 429 Too Many Requests unless the requests are paced and spread over many IPs.

ProxyField gives Claude's scraper those IPs through one gateway, gate.proxyfield.io:7000: a new IP on every request or a sticky session from 30 s to 36 h, over HTTP, HTTPS or SOCKS5, with unlimited concurrent threads. Residential proxies start at $2.95/GB for 1 GB and fall to $1.95/GB from 1,000 GB, with no subscription, and the traffic never expires. All purchases are final.

Claude's tools do not change the rules: public pages are generally fine to read; getting past logins or blocks, ignoring terms you accepted, or collecting personal data carries legal risk. Read is web scraping legal for the general rules and is AI scraping legal for using models on scraped data.

Get started: let Claude write the scraper, then run it through residential proxies. Start with the Easy Trial: 1 GB for $2.95; traffic never expires.

• Anthropic, Web fetch tool, Web search tool and Browser use tool
• Anthropic, Agent Skills overview
• Claude Code docs: Tools reference, Enterprise network configuration, Use Claude Code with Chrome and MCP
• Anthropic Help Center, Does Anthropic crawl data from the web, and how can site owners block the crawler? (updated April 7, 2026)
• Microsoft, Playwright MCP README