How to Scrape a Website With OpenClaw

OpenClaw is a self-hosted gateway that connects chat apps such as Discord, Slack, Signal, Telegram and WhatsApp to AI agents: you run one Gateway process on your own machine and message your assistant from anywhere. Its documentation separates three things:

• Tools are "callable actions," such as exec, browser, web_search and web_fetch.
• Skills are "a SKILL.md instruction pack loaded into the agent prompt": they teach the agent a workflow with the tools it already has.
• Plugins add new capabilities, including extra tools and web fetch or web search providers.

For scraping, that gives you three methods, from the quickest to the most repeatable.

web_fetch "does a plain HTTP GET and extracts readable content (HTML to markdown or text). It does not execute JavaScript." It is on by default. Ask the agent something like "fetch https://example.com/blog and list every post title with its date," and it calls:

await web_fetch({ url: "https://example.com/blog", extractMode: "markdown" });

What to know before you rely on it:

• It runs Readability, which keeps the main content of a page and drops navigation, so it suits articles better than listing grids.
• Output defaults to 20,000 characters (tools.web.fetch.maxChars), and results are cached for 15 minutes.
• It blocks private and internal hostnames and re-checks every redirect.
• For JavaScript-heavy or login-protected pages, the documentation sends you to the browser.

The agent gets one browser tool with actions such as open, navigate, snapshot, act, text, requests and screenshot. A snapshot returns the page as a UI tree with element references, and act uses those references to click, type or select, so the agent can open a listing, scroll, click "next page" and read what loaded. browser text extracts visible text, up to 40,000 characters, and browser requests reads the page's network log, which often shows the JSON request behind a listing: a cleaner source than the rendered HTML.

Browser settings live in ~/.openclaw/openclaw.json. Local managed profiles run headless by default; set headless: false to watch the agent work.

For a job you will repeat, such as product listings from one store every morning, a skill is the dependable route: the agent follows the same steps each time and runs your tested script instead of improvising. Skills live in your workspace skills/ folder, one directory per skill with a SKILL.md file.

~/.openclaw/workspace/skills/product-listings/SKILL.md:

---
name: product-listings
description: Scrape product names, prices and URLs from a category page into a CSV with the bundled Playwright script.
---

# Product listings

When the user asks for product listings from a category URL:

1. Read the site's robots.txt and stop if the category path is disallowed.
2. Use the `exec` tool to run:
   `python {baseDir}/scripts/listings.py "<category URL>" --pages 5 > listings.csv`
   The script reads the proxy from SCRAPER_PROXY_USER and SCRAPER_PROXY_PASS.
3. If the script prints a 403 or 429 status, stop and tell the user; do not retry in a loop.
4. Reply with the row count and the first 5 rows, and say where listings.csv was saved.

~/.openclaw/workspace/skills/product-listings/scripts/listings.py, a Playwright scraper:

# pip install playwright && playwright install chromium
import csv, os, sys
from playwright.sync_api import sync_playwright

url = sys.argv[1]
pages = int(sys.argv[sys.argv.index("--pages") + 1]) if "--pages" in sys.argv else 1
proxy = {"server": "http://gate.proxyfield.io:7000",
         "username": os.environ["SCRAPER_PROXY_USER"],   # e.g. <login>-country-us-city-new_york
         "password": os.environ["SCRAPER_PROXY_PASS"]}

out = csv.writer(sys.stdout)
out.writerow(["name", "price", "url"])
with sync_playwright() as p:
    browser = p.chromium.launch(proxy=proxy)
    page = browser.new_page()
    for n in range(1, pages + 1):
        resp = page.goto(f"{url}?page={n}", wait_until="networkidle")
        if resp and resp.status in (403, 429):
            print("status", resp.status, file=sys.stderr)
            break
        for card in page.locator(".product-card").all():  # adjust selectors to the site
            out.writerow([card.locator(".title").inner_text(),
                          card.locator(".price").inner_text(),
                          card.locator("a").first.get_attribute("href")])
        page.wait_for_timeout(2000)
    browser.close()

Check that the skill loaded with openclaw skills list, then call it by name with /skill product-listings or simply ask for "product listings from" a URL. The selectors are examples: ask the agent to open one category page in the browser, take a snapshot and adjust .product-card, .title and .price to the real page before the first full run. More on Playwright itself: Playwright web scraping.

{baseDir} is resolved to the skill's own folder, so the skill keeps working wherever the workspace lives. To run the job on a schedule, OpenClaw's automations can start an agent turn at set times.

OpenClaw has three proxy settings, and only two of them are meant for scraping traffic:

1. The managed browser. The documentation is explicit: "Gateway/provider HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY environment variables do not automatically proxy the OpenClaw-managed browser." To proxy it, "pass explicit Chrome proxy flags through browser.extraArgs, such as --proxy-server=...." Chrome's flag takes an address but no username and password, so add your Gateway machine's IP to the ProxyField allowlist and use the plain gateway address, where <port> is the port you set up in the dashboard for that IP:

   Copy code

   {
     browser: {
       extraArgs: ["--proxy-server=http://gate.proxyfield.io:<port>"],
     },
   }

   The documentation adds that strict SSRF mode blocks explicit browser proxy routing unless private-network browser access is intentionally enabled, so check browser.ssrfPolicy if pages stop loading after the change.

2. Your scripts. A script run by a skill takes the proxy in its own code, with full username and password support, as in the Playwright example above. This is the most flexible place: you can set the targeting per job in the username, as in <login>-country-us-city-new_york for New York. Other targeting in the username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default) after the location, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600.

3. Not for scraping: proxy.proxyUrl. This setting routes OpenClaw's own HTTP and WebSocket traffic through "an operator-managed forward proxy" for egress control and SSRF protection, which would also send your model API calls through it. Likewise, web_fetch only uses an environment proxy with useTrustedEnvProxy, meant for operator-controlled proxies. Leave both for network security, not for scraping IPs.

ClawHub is the registry for OpenClaw skills and plugins. Search it from OpenClaw with openclaw skills search "scraper" and install a skill with openclaw skills install @owner/skill-name. A skill runs commands with your agent's permissions, so read its SKILL.md and scripts first, and check its ClawHub security audit: the audit shows the risk level, findings and the credentials a skill asks for, though ClawHub says audits "are not a guarantee that a release is risk-free." Never paste your proxy password into a skill file; keep it in environment variables or OpenClaw's secret store.

An agent that fetches a hundred pages in a minute from your home or server IP looks like what it is, and sites answer with rate limits, 429 Too Many Requests, 403 Forbidden or CAPTCHAs. Four fixes, in order:

ProxyField gives OpenClaw those IPs through one gateway, gate.proxyfield.io:7000: HTTP, HTTPS or SOCKS5, username and password (or an IP allowlist, on the port you set up in the dashboard), a new IP on every request or sticky sessions from 30 s to 36 h, and targeting by country, region or city. You pay per GB with no subscription, and traffic never expires. All purchases are final. More on the setup: proxies for web scraping.

An agent scrapes under the same rules as any scraper: public pages are generally fine; getting past logins or blocks, ignoring terms you accepted and collecting personal data carry legal risk. See is web scraping legal and, for AI-specific questions, is AI scraping legal.

Get started: set up residential proxies for your OpenClaw browser or skill. Start with the Easy Trial: 1 GB for $2.95, no subscription.

• OpenClaw docs: Tools overview, Web fetch, Browser agent tools, Browser configuration, Creating skills, Network proxy, ClawHub quickstart and ClawHub security audits
• OpenClaw documentation index (llms.txt)