How to Scrape a Website: Methods, Languages and Libraries

Every scraper, from a 10-line script to a crawler of millions of pages, follows the same 6 steps.

A complete example in Python on Books to Scrape, a site built for practice, through a ProxyField proxy:

import csv
import time
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

PROXY = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
# copy the User-Agent string of your own, current browser
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                         "(KHTML, like Gecko) Chrome/<version>.0.0.0 Safari/537.36"}

url = "https://books.toscrape.com/"
rows = []
while url:
    resp = requests.get(url, headers=HEADERS, proxies={"http": PROXY, "https": PROXY}, timeout=30)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")
    for book in soup.select("article.product_pod"):
        rows.append({
            "title": book.select_one("h3 a")["title"],
            "price": book.select_one("p.price_color").get_text(strip=True),
            "url": urljoin(url, book.select_one("h3 a")["href"]),
        })
    next_link = soup.select_one("li.next a")
    url = urljoin(url, next_link["href"]) if next_link else None
    time.sleep(2)  # be gentle

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["title", "price", "url"])
    writer.writeheader()
    writer.writerows(rows)
print(len(rows), "books saved")

Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional. The same scraper in more detail, with error handling and other libraries: web scraping with Python.

Selectors that do not crash. A selector that matches nothing returns nothing, and the next line fails. Test before you read:

el = book.select_one("p.price_color")          # BeautifulSoup: None when nothing matches
price = el.get_text(strip=True) if el else ""

# Scrapy: .get() returns None (or your default) instead of raising
price = response.css("p.price_color::text").get(default="")
link = response.css("h3 a::attr(href)").get()

In Scrapy, css() and xpath() return a SelectorList, ::text and ::attr(name) pick text and attributes, and //li[1] (the first li of each parent) is not (//li)[1] (the first li on the page): see the Scrapy selectors documentation.

Choose by how the page is built and how often you need the data, not by habit.

Test in that order: if curl (below) returns the data, write a script; move up to a browser or an API only when the cheaper step fails. Guides: no-code web scrapers, what is a web scraping API, scraper API vs proxies.

There are 8 common ways to scrape a website; pick by how the page is built and how much data you need.

Why curl? curl is on almost every machine, shows you exactly what the server returns to a plain request, and lets you test headers, cookies and a proxy in one line before you write a scraper:

curl -s -x gate.proxyfield.io:7000 -U "<login>-country-us-city-new_york:<password>" -A "Mozilla/5.0" https://books.toscrape.com/ | head -c 500

If the data is in that output, a simple HTTP scraper will work; if not, the page needs JavaScript or a different request.

Any language that can send HTTP requests and parse HTML can scrape; Python and JavaScript have the most tools and examples.

C and C++ work well for speed-critical fetching: libcurl sends the requests (with CURLOPT_PROXY for a proxy) and libxml2's HTML parser with XPath, or Google's Gumbo, reads the pages. The code is longer and memory handling is on you, so most teams use C or C++ only inside a larger system. The choice between languages: best language for web scraping.

A static (server-rendered) page carries its data in the HTML the server sends; a dynamic page builds it in the browser with JavaScript. The difference decides your tools.

• Test it: if the value appears in View page source or in the curl output, the page is static for your purposes.
• Dynamic pages: first look for the JSON request in the Network tab; calling it directly is faster and lighter than rendering. If there is none, use a headless browser. Guides: dynamic pages, dynamic JavaScript sites with Python, headless browser scraping in Python and Go.
• Browsers cost more: a headless browser downloads scripts, images and fonts, so it uses several times the traffic and CPU of a plain request; block images and fonts when you do not need them.

"Database scraping" means collecting a site's records (products, listings, filings, people in a directory) automatically into a spreadsheet, CSV file or your own database; web scraping itself is "the process of programmatically and systematically collecting information on the web" (NNLM data glossary). There are two ways to get the records, and they are not the same job.

A site's search results are not a complete copy of its database. The search route is lighter but has limits:

• It is a view, not a copy. A search returns what matches and what the site chooses to show, ranked by relevance; records that are not indexed, or that sit past a result cap, never appear.
• Fields are thinner. Results often carry the name, price and link, not the full description, so many jobs still fetch the detail pages for some fields.
• It changes without notice. An internal search request is not a public API, and its parameters can change.

When the site offers an official API or a data export, use it: it beats both routes on stability. How to find and call a page's own requests: scrape hidden APIs, pagination.

Copying a database has its own legal angle. In the EU, the Database Directive (96/9/EC) gives a database's maker a right against extracting a substantial part of its contents, even when each record is public; the UK keeps its own database right on the same model, and a site's terms may forbid systematic extraction. robots.txt is a crawler request, not access control: RFC 9309 says its rules "are not a form of access authorization" (RFC 9309).

No. Most publicly viewable pages can be collected, including pages built by JavaScript, but no tool guarantees every site: login-only, paid, never-sent and challenge-protected content cannot be reliably scraped, and some content should not be scraped at all.

Cloudflare's documentation lists which of its products issue which challenge: the WAF (custom rules, rate limiting rules, IP access rules) and Bot Fight Mode show the Interstitial Challenge Page, Bot Management runs JavaScript Detections, Under Attack Mode issues a Managed Challenge, and Turnstile is a widget embedded in a site's form. It also notes a limit that catches scrapers: a Managed Challenge solved from a different IP than the one it was issued to is not valid, and you may land in a challenge loop (how challenges work). Keep one IP for a whole session, a sticky session on a proxy, so a page you are allowed to read does not loop. The same page (last updated October 16, 2026) lists two more limits: challenges do not support "browser extensions that modify the browser's User-Agent value or Web APIs such as Canvas and WebGL", and "Challenge Pages cannot be embedded in cross-origin iframes". Why sites flag a visitor as a bot: why does Cloudflare think I'm a bot; the challenge types: types of CAPTCHA.

Many public pages that refuse a bare script load in a real browser on one consistent residential IP, but no tool, ours included, is guaranteed to reach every site. Where a site has put data behind a login or a deliberate barrier, the route is its official API, an export or the owner's permission, not a way around the barrier.

One page is a script; thousands of pages are a system with a queue, retries and a schedule.

• Pagination and crawling: follow "next" links, page numbers or infinite-scroll API calls; for whole sites, crawl from the sitemap. How to crawl a website, build a web crawler, Python web crawler.
• Speed: run requests concurrently with async clients or a framework, reuse connections, and fetch only what you need: make web scraping faster.
• Scheduling: cron, a cloud function or a workflow tool such as n8n reruns the scraper and stores the results.

At volume, a website notices a scraper by its IP, its request rate, its headers and, in browsers, its fingerprint. Rotate residential IPs, send real browser headers, pause between requests and keep sessions consistent; the full checklist is in how to avoid getting blocked while scraping, and the habits that keep scrapers healthy in web scraping best practices.

ProxyField residential proxies cost $2.95/GB for 1 GB down to $1.95/GB from 1,000 GB, rotate on every request or hold a sticky IP for 30 seconds to 36 hours, and target country, region and city in 195+ countries; the traffic never expires. See proxies for web scraping.

Pick storage by how you will use the data, not by habit.

Keep the raw HTML of each page for a while: when a selector breaks, you can fix it and re-parse without fetching the pages again.

Scraping public data is lawful in many cases, but the answer depends on the data, the site's terms, the method and the country: personal data, copyrighted content and data behind logins carry the most risk.

What you cannot or should not scrape, with the sources:

• robots.txt is a request, not a lock. RFC 9309, the Robots Exclusion Protocol, says its rules "are not a form of access authorization" (RFC 9309). Respecting it is good practice; ignoring it does not by itself unlock anything.
• Logged-in and non-public areas are where US computer-fraud law bites. In Van Buren v. United States (2021) the Supreme Court read the Computer Fraud and Abuse Act as being about areas of a computer you are not entitled to access (opinion); the Department of Justice's charging policy says that breaking a public website's terms of service is not, by itself, grounds for a CFAA prosecution (Justice Manual 9-48.000). In hiQ Labs v. LinkedIn (9th Cir. 2022), a ruling on a preliminary injunction, the court found serious questions as to whether scraping public profiles is "without authorization" under the CFAA (opinion); the case later ended in LinkedIn's favor on its contract claims (November 2022 ruling, December 2022 consent judgment).
• A deliberate barrier means ask. When a site puts data behind a login, a paywall or a challenge, use its official API, an export or the owner's permission.

The overview is in is web scraping legal, and the series in what data is legal and safe to scrape. This is not legal advice.

• Start here: how to build a web scraper, HTML scraping with CSS selectors and XPath, web scraping projects with code, web scraping best practices.
• Python: web scraping with Python, Python libraries for web scraping, BeautifulSoup tutorial, what is Scrapy, Scrapy tutorial, Python web crawler, dynamic JavaScript sites with Python.
• Browsers: what is a headless browser, headless browsers compared, headless browser scraping in Python and Go, what is Puppeteer, scraping with Puppeteer, what is Playwright, scraping with Playwright, what is Selenium, Selenium with Python, Puppeteer vs Playwright, Puppeteer alternatives.
• Other languages: JavaScript and Node.js, Java, C#, PHP, Ruby, Go, R, Rust, which language to choose.
• Techniques: dynamic web pages, how to crawl a website, build a web crawler, scrape images from a website, make web scraping faster, schedule and deploy scrapers.