How to Schedule and Deploy Web Scrapers

Sources GitHub Actions events: schedule, AWS Lambda quotas, EventBridge Scheduler schedule types, Apache Airflow documentation (Airflow 3.3.2 on PyPI, 2026-10-23).

A rule of thumb: start with cron or GitHub Actions; move to Airflow when you have several scrapers whose outputs feed each other, or when someone needs to see why last Tuesday's run failed.

A scheduler can only start a script. These habits decide whether unattended runs work:

• One command, one run. python scrape.py --date today fetches, writes and exits. Exit code 0 means success; anything else means failure, so the scheduler can alert.
• Idempotent output. Write to a dated file or upsert into a table keyed by URL and date, so a rerun replaces the run instead of duplicating it.
• No overlap. If a run takes longer than the interval, two copies fight over the same files. Use a lock (flock with cron, max_active_runs=1 in Airflow, concurrency in GitHub Actions).
• Timeouts everywhere. A request with no timeout can hang a run forever. Set one per request and one for the whole job.
• Configuration from the environment. Proxy URL, tokens and output paths come from environment variables or a secrets store, never from the code.
• Logs that answer questions. Pages fetched, rows written, errors by status code, run time.

crontab -e and add a line. This one runs at 06:15 every day, skips the run if the previous one still holds the lock, and appends the output to a log:

# m  h  dom mon dow  command
15 6 * * * cd /opt/scraper && /usr/bin/flock -n /tmp/scraper.lock .venv/bin/python scrape.py >> logs/scrape.log 2>&1

Cron uses the server's time zone and a minimal environment, so use absolute paths and set the variables the script needs at the top of the crontab or in a wrapper script. A systemd timer does the same job with OnCalendar=*-*-* 06:15:00, keeps the output in the journal, and with Persistent=true runs a missed job after the server comes back.

A workflow file in .github/workflows/ runs the scraper on GitHub's runners. The cron expression is in UTC.

name: daily-scrape
on:
  schedule:
    - cron: "15 6 * * *"
  workflow_dispatch: {}          # also allows a manual run
concurrency: daily-scrape        # never two runs at once
jobs:
  scrape:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - run: python scrape.py
        env:
          PROXY_URL: ${{ secrets.PROXY_URL }}
      - uses: actions/upload-artifact@v7
        with: { name: data, path: out/ }

Store the proxy URL as a repository secret. Runner IPs belong to a cloud provider and change between runs, which is one reason scheduled scrapers on CI send their requests through a proxy.

Lambda suits short HTTP scrapers: no server to patch, and you pay for the seconds the function runs. A run can last at most 15 minutes (900 seconds), so split long jobs into batches, for example one function call per list page, fed from a queue.

# handler.py  (package with requests and beautifulsoup4, or use a Lambda layer)
import json, os, datetime
import boto3, requests
from bs4 import BeautifulSoup

s3 = boto3.client("s3")
PROXY = os.environ.get("PROXY_URL")

def handler(event, context):
    url = event.get("url", "https://www.scrapethissite.com/pages/simple/")
    r = requests.get(url, timeout=30, proxies={"http": PROXY, "https": PROXY} if PROXY else None)
    r.raise_for_status()
    names = [h.get_text(strip=True) for h in BeautifulSoup(r.text, "html.parser").select("h3.country-name")]
    key = f"countries/{datetime.date.today()}.json"
    s3.put_object(Bucket=os.environ["BUCKET"], Key=key, Body=json.dumps(names))
    return {"rows": len(names), "key": key}

Schedule it with EventBridge Scheduler, which takes rate(1 day) or a six-field cron expression such as cron(15 6 * * ? *) and a time zone. Writing each run to a dated S3 key keeps runs separate; Athena or Glue can then query the files.

Two Lambda specifics:

• Headless browsers (Playwright, Puppeteer) do not fit a zip package well; deploy the function as a container image, which may be up to 10 GB, and give it more memory.
• Scrapy on Lambda works for one crawl per invocation, but Scrapy's Twisted reactor cannot be restarted in the same process, so a warm container that runs a second crawl fails with ReactorNotRestartable. Run the crawl in a subprocess (subprocess.run(["scrapy", "crawl", "name"])) or start a fresh process per invocation.

Airflow runs DAGs: Python files that declare tasks, their order, their schedule and their retries. Each run is recorded, so you can see which task failed, rerun just that task, or backfill missed dates. In Airflow 3 a scraping DAG looks like this:

from datetime import datetime, timedelta
from airflow.sdk import dag, task

@dag(schedule="15 6 * * *", start_date=datetime(2026, 10, 1), catchup=False,
     max_active_runs=1, default_args={"retries": 3, "retry_delay": timedelta(minutes=10)})
def daily_prices():

    @task
    def scrape(ds=None):
        import subprocess
        subprocess.run(["python", "/opt/scraper/scrape.py", "--date", ds], check=True)
        return f"/data/prices/{ds}.csv"

    @task
    def load(path: str):
        ...  # load the CSV into the warehouse

    load(scrape())

daily_prices()

Give every scraping task a timeout (execution_timeout) and keep heavy work out of the DAG file itself, which the scheduler parses often. For browser scrapers, run each task in a container (Docker or Kubernetes operators) rather than on the Airflow worker.

As often as the data changes and you will act on it, and no more often than the site can comfortably serve. Typical examples: prices and stock daily or a few times a day; rankings daily; job and real-estate listings daily; reviews and catalog details weekly; a full crawl of a large site weekly or monthly, with daily runs for the pages that change. Fetch only what changed where you can: sitemaps with lastmod, "newest first" listings, and conditional requests (If-Modified-Since, ETag) that return 304 Not Modified when nothing changed.

Silent failures are the main risk of scheduled scraping: the job "succeeds" and writes 0 rows because the layout changed.

• Alert on failure: the scheduler's own alert, or a dead-man's switch that expects a ping at the end of each run.
• Alert on volume: compare rows per run with the last week; for example, look into any drop of more than a third.
• Alert on errors by type: a rise in 403 or 429 responses means blocking; a rise in parse errors means a layout change.
• Keep the raw HTML of a sample of pages from each run, so you can fix the parser against the page that broke it.

• Build the scraper first: how to build a web scraper.
• Make each run shorter: how to make web scraping faster.
• Project ideas that need a schedule: web scraping projects.
• More automation options: automated web scraping.

Scraping on a schedule? ProxyField residential proxies: pay per GB from $1.95/GB, no subscription, and traffic that never expires between runs. Start with the Easy Trial (1 GB); all purchases are final.