Web Crawl MCP: Scraping Tools With an MCP Server for AI Agents

MCP, the Model Context Protocol, is "an open-source standard for connecting AI applications to external systems"; its own documentation calls it "a USB-C port for AI applications." An MCP server exposes tools, each with a name, a description and a typed input. The AI client lists the tools to the model, and the model decides when to call one, for example browser_navigate with a URL, then browser_snapshot to read the page.

For scraping, that means an agent can collect data as part of a larger task, "find the pricing pages of these five companies and compare the plans," without you writing glue code. It also means the agent, not your code, decides how many pages to open, which is why MCP suits exploration and small jobs better than a nightly crawl of 100,000 URLs.

MCP server vs scraping API. A scraping API is an HTTP endpoint your code calls with fixed parameters. A scraping API's MCP server wraps the same service as tools with descriptions, so an agent can call it on its own. The fetching, IPs and billing are the same; the difference is who decides when to call: your code or the model.

Searchers ask for web crawl MCP options with "top-tier scalability," "great data quality," "excellent accuracy" or "top crawling speed." Here is what each depends on:

• Scalability. A local browser server runs on one machine; Playwright MCP's persistent profile "can only be used by one browser instance at a time," so parallel agents need --isolated or separate profiles. Past a few hundred pages per site, the limit is usually blocking, not CPU: every request leaves from the same IP unless the server uses a proxy.
• Data quality and accuracy. Check what the server returns. Accessibility snapshots keep structure and element references; Markdown keeps text and drops layout; the Fetch server truncates to 5,000 characters unless you raise max_length. If the model then extracts fields from that output, validate them, because a model can misread a value.
• Speed. A plain fetch takes one HTTP request; a browser page loads scripts, images and fonts. Use a fetch server for static pages and a browser only where the page needs JavaScript.
• Customization. Playwright MCP takes a JSON config file with Playwright launch and context options: browser, viewport, storage state, proxy.
• Documentation and usability. Look for a one-line install for your client, a full option list and a security section. Playwright MCP's README has install steps for 20 clients (counted October 27, 2026) and says plainly that it "is not a security boundary."

Playwright MCP needs Node.js 18 or newer. Add it to your client with its standard command, npx @playwright/mcp@latest:

# Claude Code
claude mcp add playwright npx @playwright/mcp@latest
# Codex CLI
codex mcp add playwright npx "@playwright/mcp@latest"
# VS Code (GitHub Copilot agent)
code --add-mcp '{"name":"playwright","command":"npx","args":["@playwright/mcp@latest"]}'

Clients that read a JSON config, such as Claude Desktop, Cursor or Windsurf, take the same server as "command": "npx", "args": ["@playwright/mcp@latest"]. Then ask the agent in plain words: "Open https://example.com/pricing, read the plan names and prices, and give me a table."

Add a proxy. The --proxy-server option takes an address such as http://myproxy:3128 or socks5://myproxy:8080 but no password. With an IP allowlist, add your machine's IP in the ProxyField dashboard and pass the gateway with the port you set up there for it in place of <port>:

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@latest", "--headless", "--proxy-server=http://gate.proxyfield.io:<port>"]
    }
  }
}

With a username and password, use a config file (--config proxy.json), whose launchOptions are passed to Playwright's own launch, which accepts proxy credentials:

{
  "browser": {
    "launchOptions": {
      "proxy": {
        "server": "http://gate.proxyfield.io:7000",
        "username": "<login>-country-us-city-new_york",
        "password": "<password>"
      }
    }
  }
}

<login>-country-us-city-new_york asks for a New York IP; other targeting in the username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default) after the location, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600. An agent's browsing session clicks through several pages, so a sticky session, one IP for the whole visit, usually works better than a new IP per request. More on Playwright with proxies: Playwright proxy setup.

Microsoft's README also points coding agents to Playwright CLI with skills, which it describes as more token-efficient because it avoids "loading large tool schemas and verbose accessibility trees into the model context." MCP remains its recommendation for long-running agent loops that keep a browser open.

Choose Playwright MCP for new work. The Puppeteer server in the MCP reference collection has been archived, and Playwright MCP is maintained by Microsoft, supports Chromium, Firefox and WebKit, and works from accessibility snapshots, so the model does not need screenshots or a vision model. If you want Puppeteer's engine, Chrome DevTools MCP uses Puppeteer to automate Chrome and adds DevTools features such as network inspection and performance traces; it officially supports Google Chrome and Chrome for Testing only.

Any MCP client can use a scraping server. The MCP documentation names Claude and ChatGPT among AI assistants and Visual Studio Code and Cursor among development tools. Playwright MCP's README gives install steps for VS Code, Cursor, Windsurf, Claude Desktop, Claude Code, Codex, Gemini CLI, Goose and others. Local servers run as a command on your machine (stdio); remote servers are reached over HTTP, for example claude mcp add --transport http <name> <url> in Claude Code.

A hosted MCP server runs on a scraping-API vendor's infrastructure, so you add a URL instead of a command. In Claude Code that is claude mcp add --transport http <name> <url>; clients with a JSON config take the URL and a header such as Authorization: Bearer <api_key>. Keep the key in an environment variable or the client's secret store, never in the prompt.

The tools behind a hosted server are the vendor's scraping API, described for a model. Vendors and their per-request prices are compared on web scraping API prices. ProxyField's own scraper APIs have no MCP server or SDK in their first version; their design is in the API documentation (in development: Contact sales).

Where the proxy goes depends on the server:

• Playwright MCP: --proxy-server, or launchOptions.proxy in the config file for a username and password.
• Fetch server: --proxy-url; without it, the server uses the standard HTTP_PROXY, HTTPS_PROXY and ALL_PROXY variables. It obeys a site's robots.txt for requests that come from the model, unless you start it with --ignore-robots-txt; keep the default.
• Chrome DevTools MCP: set the proxy on the Chrome instance it controls; see its configuration guide.
• Crawl4AI: proxy_config in the crawler config takes the gateway, username and password; setup in Crawl4AI with proxies.
• Firecrawl MCP on a self-hosted Firecrawl: the MCP server only forwards calls, so the proxy goes into the Firecrawl instance: PROXY_SERVER=http://gate.proxyfield.io:7000, PROXY_USERNAME=<login>-country-us-city-new_york and PROXY_PASSWORD=<password> in its apps/api/.env; more in AI web scrapers compared.
• Hosted scraping-API servers: they use the vendor's IPs and bill per request, so you do not add your own proxy.

ProxyField gives a self-hosted MCP server one gateway, gate.proxyfield.io:7000, over HTTP, HTTPS or SOCKS5: a new IP on every request or a sticky session from 30 s to 36 h, targeting by country, region or city, and unlimited concurrent threads. You pay per GB with no subscription, and traffic never expires. All purchases are final.

A scraping tool works well for agents when:

If you already have a scraper, an MCP server makes it an agent tool in a few lines. With the official Python package for MCP, version 2 (pip install "mcp[cli]>=2"; 2.3.0 is the release of October 2, 2026), a fetch tool that goes through the ProxyField gateway:

# server.py: run with `mcp run server.py` (or `uv run mcp run server.py`)
import os
import requests
from mcp.server import MCPServer

PROXY = os.environ["SCRAPER_PROXY"]  # http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000
mcp = MCPServer("crawler")

@mcp.tool()
def fetch_page(url: str, max_chars: int = 20000) -> str:
    """Fetch a public web page through a rotating residential proxy and return its HTML."""
    resp = requests.get(url, proxies={"http": PROXY, "https": PROXY}, timeout=30)
    resp.raise_for_status()
    return resp.text[:max_chars]

Add it to Claude Code with claude mcp add --env SCRAPER_PROXY=... crawler -- mcp run server.py. On the 1.x releases the same class is FastMCP, imported with from mcp.server.fastmcp import FastMCP; version 2 renamed it MCPServer. To try the server in the MCP Inspector first, run mcp dev server.py. For crawling logic and rotation, see proxies for web scraping; for the AI assistants themselves, see web scraping with Claude and can ChatGPT scrape websites.

Get started: point your MCP server at residential proxies. Start with the Easy Trial: 1 GB for $2.95, no subscription.

• Model Context Protocol, What is MCP? (specification version 2026-07-28)
• Microsoft, Playwright MCP README
• Chrome DevTools, chrome-devtools-mcp README
• MCP reference servers and the Fetch server README
• MCP Python package README (main branch) and mcp on PyPI (2.3.0, released October 2, 2026)
• Chrome DevTools MCP tool reference
• Crawl4AI Docker server README (MCP endpoints and tools)
• Claude Code docs, MCP
• Firecrawl MCP server README (MIT; FIRECRAWL_API_URL for self-hosted instances)