Web Scraping Best Practices

The basics protect both the site and your scraper:

A request can fail for reasons a retry fixes (a timeout, a dropped connection, a busy server) and reasons it cannot (a page that does not exist, a request the site refuses). Retry only the first kind, with growing waits.

Which status codes to retry:

How to wait between attempts. Use exponential backoff with jitter: wait about 1 s, then 2, 4, 8 and so on up to a cap, and randomize each wait so many workers do not retry in step. If the response has a Retry-After header, wait that long instead (RFC 9110, Retry-After). Stop after 3-5 attempts and record the URL as failed.

Retry only safe requests. GET and HEAD can be repeated without side effects. Do not retry a POST blindly: if the first one reached the server, a retry may submit a form twice.

In Python, Requests does not retry failed requests by default, but urllib3's Retry adds backoff, status-code retries and Retry-After support to a session:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retry = Retry(
    total=5,                                   # at most 5 retries
    backoff_factor=1,                          # exponential waits between attempts
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],           # never POST
    respect_retry_after_header=True,
)
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry))
session.mount("http://", HTTPAdapter(max_retries=retry))

resp = session.get("https://books.toscrape.com/", timeout=(10, 30))   # connect, read timeouts

When retries run out, Requests raises requests.exceptions.RetryError (the "Max retries exceeded" message). Catch it, log the URL and move on: one failed page should never stop the run. Always set a timeout; without one, a request can hang for a very long time and a retry never starts.

Design the run so a failure costs little:

• Checkpoint. Store which URLs are done (in a database table, a file, or the framework's job directory) and skip them on restart.
• Separate the failures. Put URLs that failed after all retries in a "failed" list and re-run them later, at a slower rate.
• Make writes idempotent. Upsert by a stable key (the URL or the product ID) so re-running does not create duplicates.
• Alert on the numbers. Alert when the success rate drops, records per page fall to zero, or the run takes twice as long as usual.
• Keep evidence. Save the HTML (or a screenshot) of a page that failed to parse; it shows at once whether the layout changed or you got a block page.

Then read the failure: a wave of 429 or 403 means you are going too fast or from too few IPs; timeouts point to the network or the proxy; parse errors with 200 responses mean the page changed or you received a block page with a normal status.

Layouts change without notice, and a scraper that silently writes empty prices is worse than one that crashes. Build for it:

Basic scraping is one script that fetches and parses; advanced scraping adds the rest of the table: queues, retries, sessions, browsers where needed, proxies and monitoring.

Names, email addresses and phone numbers of people are personal data in the EU and the UK even when they are published on a public page, so the GDPR and the UK GDPR apply to collecting them (Regulation (EU) 2016/679). Practices that keep a contact-data project defensible:

• Prefer business contact points. Generic company addresses and phone numbers published for inquiries (sales@, info@) carry less risk than personal ones.
• Define the purpose and the fields before you start, and collect nothing else (data minimization).
• Have a lawful basis. For B2B outreach this is usually legitimate interests, which needs a documented balancing test; France's regulator, the CNIL, lists measures for scraping under that basis, such as defining the data in advance and excluding sites that object to scraping (CNIL focus sheet).
• Tell people. When you collect personal data not from the person, Article 14 GDPR requires you to inform them, generally within one month.
• Follow the outreach rules. Email marketing has its own rules: the CAN-SPAM Act in the US (FTC compliance guide) and the ePrivacy rules in the EU, which the UK applies through PECR.
• Secure and delete. Store contact data securely, honor objections and deletion requests, and delete data you no longer need.

ProxyField's acceptable use allows collecting public web data; it does not allow data behind logins you are not entitled to, or fraud. Collecting personal contact data (emails, phone numbers) needs a lawful basis, and that is the customer's responsibility. Public company data is fine. Spam is forbidden in any case.

• Match rotation to the job. Rotate per request for independent pages (product pages, search results); keep a sticky session for flows that need one IP (pagination tied to cookies, multi-step forms). ProxyField sessions can stay on one IP from 30 s to 36 h.
• Target the location you are measuring. Prices, search results and ads differ by country and city; request from there.
• Do not use IPs to hide a too-fast scraper. More IPs at the same rate still overload the site; pace first, then rotate.
• Budget traffic. Residential proxies are billed per GB: measure the bytes per page, block images and fonts in browsers, and avoid re-downloading unchanged pages.

ProxyField residential proxies cost $2.95/GB for 1 GB down to $1.95/GB from 1,000 GB, with no subscription; the traffic never expires, and all purchases are final. See rotating proxies and residential and mobile proxy pricing.

This article describes general practice and is not legal advice.