Automated Web Scraping

To make a scraper run on its own, check its data and alert you, build these five pieces:

The script and the workflow below do all five.

Automated web scraping means the whole collection job runs without a person: deciding when to run, fetching, extracting, checking, storing and reporting. A scraper you start by hand each morning is a script; the same scraper started by a scheduler, retrying failures, validating its output and alerting on problems is an automated pipeline.

Manual scraping is a person copying data from pages. Semi-automated scraping is a tool you run by hand, such as a browser extension. Fully automated scraping is a scheduled job that delivers fresh data every hour, day or week, and tells you only when something needs attention.

This Python script fetches a page through the ProxyField gateway with automatic retries, validates the rows against five rules, saves the page that broke the parser, posts an alert to a webhook on any failure, and pings a heartbeat URL only after a good run. It targets books.toscrape.com, a public scraping sandbox, so you can try it before you point it at your own pages. Secrets come from environment variables, never from the code.

# pip install requests beautifulsoup4
import csv, datetime, json, os, pathlib, requests
from bs4 import BeautifulSoup
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

URL = "https://books.toscrape.com/"
PROXY = os.environ.get("PROXY_URL")          # http://<login>-country-us:<password>@gate.proxyfield.io:7000
WEBHOOK = os.environ.get("ALERT_WEBHOOK")    # a Slack incoming webhook URL
HEARTBEAT = os.environ.get("HEARTBEAT_URL")  # e.g. https://hc-ping.com/<your-check-uuid>
STATE = pathlib.Path("last_run.json")

session = requests.Session()
if PROXY:
    session.proxies = {"http": PROXY, "https": PROXY}
retry = Retry(total=3, backoff_factor=2, status_forcelist=[429, 500, 502, 503, 504])
session.mount("https://", HTTPAdapter(max_retries=retry))

def alert(message):
    if WEBHOOK:
        requests.post(WEBHOOK, json={"text": message}, timeout=10)  # Discord webhooks take "content"
    if HEARTBEAT:
        requests.post(HEARTBEAT + "/fail", data=message, timeout=10)

def parse(html):
    rows = []
    for card in BeautifulSoup(html, "html.parser").select("article.product_pod"):
        link, price = card.select_one("h3 a"), card.select_one("p.price_color")
        rows.append({"title": link.get("title") if link else None,
                     "price": float(price.get_text(strip=True).lstrip("£")) if price else None})
    return rows

def validate(rows):
    errors = []
    if len(rows) < 10:
        errors.append(f"only {len(rows)} rows, expected at least 10")
    bad = [r for r in rows if not r["title"] or r["price"] is None or not 0 < r["price"] < 1000]
    if bad:
        errors.append(f"{len(bad)} rows without a title or with a price outside 0-1000")
    if STATE.exists():
        last = json.loads(STATE.read_text())["rows"]
        if len(rows) < last * 0.5:
            errors.append(f"{len(rows)} rows against {last} last run: a drop of more than 50%")
    return errors

try:
    response = session.get(URL, timeout=30)
    response.raise_for_status()
    rows = parse(response.content)
    errors = validate(rows)
    if errors:
        pathlib.Path("failed.html").write_bytes(response.content)  # the page that broke the run
        raise ValueError("; ".join(errors))
except Exception as error:
    alert(f"scraper failed: {error}")
    raise SystemExit(1)  # a non-zero exit fails the cron job or the workflow run

today = datetime.date.today().isoformat()
with open("prices.csv", "a", newline="") as f:
    csv.writer(f).writerows([today, r["title"], r["price"]] for r in rows)
STATE.write_text(json.dumps({"date": today, "rows": len(rows)}))
if HEARTBEAT:
    requests.get(HEARTBEAT, timeout=10)  # "ran, and the data passed": silence after this means trouble

What each part does when something breaks:

Requests does not retry failed requests by default; the mounted adapter adds it. With a rotating gateway, each retry also leaves from a new IP, unless you set a sticky session in the username (-session-<id>).

GitHub Actions runs the script on GitHub's machines, with no server of your own. Put the three secrets (PROXY_URL, ALERT_WEBHOOK, HEARTBEAT_URL) in the repository's Actions secrets and add this file:

# .github/workflows/scrape.yml
name: scrape
on:
  schedule:
    - cron: "17 6 * * *"        # 06:17 UTC every day
  workflow_dispatch: {}         # a "Run workflow" button for tests
jobs:
  scrape:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with: { python-version: "3.12" }
      - run: pip install requests beautifulsoup4
      - run: python scrape.py
        env:
          PROXY_URL: ${{ secrets.PROXY_URL }}
          ALERT_WEBHOOK: ${{ secrets.ALERT_WEBHOOK }}
          HEARTBEAT_URL: ${{ secrets.HEARTBEAT_URL }}
      - if: failure()
        uses: actions/upload-artifact@v7
        with: { name: failed-page, path: failed.html, retention-days: 7 }
      - uses: actions/upload-artifact@v7
        with: { name: prices, path: prices.csv }

What to expect from the schedule, per GitHub's workflow syntax reference and events reference:

• The cron line has five fields (minute, hour, day of month, month, day of week) and runs in UTC unless you add a timezone, such as timezone: "America/New_York".
• The shortest interval is once every 5 minutes, and a run can start late when GitHub is busy, most often at the top of the hour; an off-the-hour minute such as 17 helps.
• Scheduled workflows run on the latest commit of the default branch.
• In a public repository, scheduled workflows are turned off after 60 days without repository activity.
• When the script exits with 1, the run fails and GitHub notifies the user who last changed the cron line, by email if their notification settings allow it.
• The runner is wiped after each run, so prices.csv and last_run.json must go to storage (an artifact, a bucket or a database) if the next run needs them.

cron on a Linux server does the same in one line, with a log:

17 6 * * * cd /home/me/scraper && /usr/bin/python3 scrape.py >> scrape.log 2>&1

More schedulers (systemd timers, AWS Lambda with EventBridge, Apache Airflow) and how to stop two runs from overlapping: scheduling web scrapers.

An automated scraper that saves wrong data silently is worse than one that fails. Write rules with numbers, not intentions:

Cleaning follows the same rules every run: one currency format, one date format, trimmed text, normalized units. Keep the raw values too, so you can rerun the cleaning if a rule was wrong.

Two kinds of alert cover every failure:

• Alert on failure. The script's except branch posts to a chat webhook. A Slack incoming webhook takes {"text": "..."}; a Discord webhook takes {"content": "..."}. Email works too, through the scheduler's own failure notice.
• Alert on silence. A heartbeat monitor expects a ping every day (or every hour) and alerts you when one is missing. Healthchecks.io gives each check a URL: a request to https://hc-ping.com/<uuid> marks a good run and .../fail a failed one. Sentry Crons and UptimeRobot offer the same kind of check. Silence catches what no failure alert can: a deleted schedule, an expired server, a workflow GitHub turned off.

Two habits make alerts useful: keep the raw HTML of failed pages for a few days (the workflow keeps failed.html for 7), and track the share of failed requests per site, because a rising share means the site is limiting you or changed.

Cloud servers and CI runners make automation easy, but their IP ranges are well known, and many sites score traffic from them as automated and block it or serve CAPTCHAs. A scraper that works from your laptop can fail from a GitHub runner for that reason alone. Route the cloud scraper's requests through rotating proxies: residential proxies for protected sites, rotating datacenter proxies where the site allows datacenter ranges.

ProxyField supports both a username and password and an IP allowlist. GitHub runners and serverless functions change IPs between runs, so they use the username and password from a secret; a fixed server can use the allowlist and keep no proxy password at all.

An automated scraper costs its compute, its storage and its proxy traffic, and the traffic grows with every run. Estimate it as runs × pages × page size: a daily run over 2,000 pages of 150 KB is about 300 MB a day, or about 9 GB a month. Plain HTML fetches keep it low; browser rendering raises it unless you block images and fonts. A failed run that retries uses traffic too, which is one more reason to stop early when validation fails.

ProxyField residential proxies give a new IP on every request or a sticky session from 30 s to 36 h, run unlimited concurrent threads, and target country, region and city, with a 99.6% success rate, according to our service data. They cost $2.95/GB for 1 GB, $2.79 from 10 GB and $2.65 from 50 GB, down to $1.95/GB from 1,000 GB, with no subscription; 9 GB a month at the 1–9 GB rate is $26.55. The traffic never expires, so a paused schedule wastes nothing; all purchases are final. What each job costs: proxies for web scraping. Whether a site may be scraped: is web scraping legal.