What Is a Web Scraper?

A scraper, in computing, is software that extracts data from output meant for people. A web scraper does it on websites: it reads pages the way a browser loads them, but instead of displaying them, it keeps only the fields you asked for.

"Web scraper", "website scraper", "web data scraper" and "scraper bot" are used for the same kind of tool. The activity is called web scraping. Outside computing, a scraper is a hand tool or a machine for scraping surfaces, which is where the word comes from: the program scrapes the data off the surface of the page.

Every web scraper runs the same loop:

This Python scraper collects the titles and prices from books.toscrape.com, a public sandbox for scraping practice; it returned 20 books per page when we ran it on October 23, 2026 (Requests 2.34, Beautiful Soup 4.15).

# pip install requests beautifulsoup4
import requests
from bs4 import BeautifulSoup

proxy = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"   # optional for one page
html = requests.get("https://books.toscrape.com/", proxies={"http": proxy, "https": proxy}, timeout=30).content
soup = BeautifulSoup(html, "html.parser")

for book in soup.select("article.product_pod"):
    print(book.select_one("h3 a")["title"], book.select_one("p.price_color").get_text(strip=True))

Passing the raw bytes (.content) lets Beautiful Soup detect the page's encoding, so the pound sign comes out right. The step-by-step project, with pagination, errors and storage, is in how to build a web scraper.

Web scrapers differ by how they are built, how they load pages, where they run and how much they cover.

Two more distinctions matter in practice:

• Local vs cloud. A local scraper runs on your computer with your IP; a cloud scraper runs on servers, on a schedule, usually with proxies.
• General-purpose vs specialized. A general scraper takes selectors for any site; a specialized one knows one site's pages and returns its fields ready-made.

The tools for each type are compared in best web scraping tools; the no-code ones in our guide to no-code web scrapers.

• A crawler discovers pages by following links; a scraper extracts data from pages. Many scrapers include a small crawler. See web crawling vs web scraping.
• A parser turns raw HTML into fields; it is one part of a scraper. See data parsing vs web scraping.
• An official API returns data the site chooses to share; a scraper reads what the page shows. See web scraping vs API.

Web scrapers collect public data that changes often or spreads across many sites: competitors' prices, search rankings, product catalogs, reviews, job ads, property listings, news and training data for AI. Examples by industry are in our guide on what web scraping is used for.

A scraper that worked on day one breaks for three reasons: the site changes its layout, the site limits requests from your IP, or the page starts loading its data with JavaScript. Keep it working with:

• Monitoring: alert when a field comes back empty or the row count drops.
• Polite pacing: a delay between requests, and respect for robots.txt and the site's limits.
• Rotating IPs at volume: rotating proxies give each request a new IP, and a sticky session keeps one IP for flows that span pages.
• A browser only where needed: render the pages that need it and fetch the rest as plain HTML.

ProxyField residential proxies work with Python requests, Scrapy, Playwright, Puppeteer and Selenium over HTTP, HTTPS and SOCKS5, with country, region and city targeting and unlimited concurrent threads. You pay per GB with no subscription; the traffic never expires, and all purchases are final. What a job costs: proxies for web scraping. Whether a site may be scraped: is web scraping legal.