Web Scraping Projects With Code

A project teaches you most when the data is public, the site is stable and you will use the result. Four questions narrow the list:

How-tos for the harder parts: how to build a web crawler (project 12), scraping dynamic content (project 11), scheduling and deploying scrapers (projects 6 and 14), and web scraping for AI training (project 15).

Beginner projects

Start on pages built for practice or published as open data. The practice sandbox at scrapethissite.com has a single-page list of 250 countries and a paginated table of hockey teams; its robots.txt blocks only its lesson and FAQ pages. Wikipedia tables load straight into pandas, and project 5 on your own site teaches crawling with no permission question at all. The step-by-step script for project 1 is in how to build a web scraper.

Intermediate projects

These run more than once, so they need storage and a schedule. A price tracker is the classic one: scrape the price, compare with yesterday's, alert on a drop. The starter code below is that project's core. Projects 9 and 13 add location: the same page shows different results or prices by country or city, so the scraper needs an IP in each place.

Advanced projects

Advanced projects combine the hard parts: JavaScript pages, many sites, scale and reliability. Expect to spend more time on queues, retries and monitoring than on selectors, and decide early whether you will run browsers, call hidden APIs, or use a scraping API such as the ProxyField Web Scraping API for the pages that need rendering.

The core of project 6: fetch a product page, read the price, store it with a timestamp, and report a drop. Replace the URL and the selector with a page and element you are allowed to scrape (find the selector with right-click, Inspect).

# pip install requests beautifulsoup4
import datetime, re, sqlite3
import requests
from bs4 import BeautifulSoup

URL = "https://example.com/product/123"
SELECTOR = "span.price"                       # from DevTools
PROXY = None                                  # e.g. "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"

db = sqlite3.connect("prices.db")
db.execute("CREATE TABLE IF NOT EXISTS prices (url TEXT, price REAL, seen_at TEXT)")

r = requests.get(URL, headers={"User-Agent": "Mozilla/5.0"}, timeout=30,
                 proxies={"http": PROXY, "https": PROXY} if PROXY else None)
r.raise_for_status()
text = BeautifulSoup(r.text, "html.parser").select_one(SELECTOR).get_text()
price = float(re.sub(r"[^\d.]", "", text))

last = db.execute("SELECT price FROM prices WHERE url=? ORDER BY seen_at DESC LIMIT 1", (URL,)).fetchone()
db.execute("INSERT INTO prices VALUES (?, ?, ?)", (URL, price, datetime.datetime.now().isoformat()))
db.commit()
if last and price < last[0]:
    print(f"Price drop: {last[0]} -> {price}")

Run it once a day with cron or a scheduled GitHub Actions workflow (how to schedule web scrapers), and swap print for an email or chat message. Prices written as 1.299,00 need a locale-aware parser.

GitHub is the largest library of open scrapers. To find good ones, browse the web-scraping topic and the scraper and crawler topics, then filter by language and sort by recently updated. Before you reuse a repository, check four things:

• License. No license file means no permission to reuse the code. MIT, BSD and Apache-2.0 allow reuse with attribution.
• Last commit. Scrapers break when sites change. A target-specific scraper untouched for a year probably no longer works.
• Issues. Open issues titled "not working" or "blocked" tell you more than the star count.
• Secrets. Never run code that asks for your passwords or tokens in a way you cannot read.

When you publish your own project:

1. Write a README with what it scrapes, how to run it and an example of the output.
2. Keep credentials out of the repository: read the proxy URL and any tokens from environment variables, and list .env in .gitignore. If a key was ever committed, revoke it; deleting the file does not remove it from the history.
3. Pin your dependencies (requirements.txt with versions, or pyproject.toml).
4. Add a small test that parses a saved copy of the page, so you can tell a layout change from a bug.

import os
PROXY = os.environ.get("PROXY_URL")   # set it in your shell or in your CI's secret settings

For repositories, issues, stars and users, GitHub's REST and GraphQL APIs return clean JSON and are the documented way in. Per GitHub's REST API rate limits, unauthenticated requests are limited to 60 per hour and requests with a personal access token to 5,000 per hour. Creating a token takes a minute and multiplies what a project can fetch, so use one. Scrape GitHub's HTML only for what the API does not return, at a gentle pace, and read GitHub's Acceptable Use Policies first.

Most beginner projects never do. A project needs proxies when:

• it fetches thousands of pages a day from one site, and the site starts answering with 429 or 403;
• it needs the page as seen from another country or city (projects 9 and 13);
• it runs from a cloud server whose IP range the target site already treats with suspicion.

For a project, the cost is the traffic: plain HTML pages are small, while browser rendering and images use much more per page. ProxyField residential proxies cost $2.95/GB for 1-9 GB down to $1.95/GB from 1,000 GB, the traffic never expires, and you can buy just 1 GB to start; see proxies for web scraping for sizing a job. The gateway is gate.proxyfield.io:7000; <login>-country-us-city-new_york asks for an exit IP in New York. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.

A project that needs other IPs? ProxyField residential proxies: 412M+ residential IPs in 195+ countries with country, region and city targeting. Pay per GB, no subscription; start with the Easy Trial (1 GB). Traffic never expires, and all purchases are final.