What Is Web Scraping? How It Works, Uses and Tools

Web scraping is software that collects data from web pages automatically and saves it in a structured form you can sort, filter and analyze. The US Network of the National Library of Medicine defines it as "the process of programmatically and systematically collecting information on the web" (NNLM data glossary). Website scraping, internet scraping and web data extraction mean the same thing.

What counts as web scraping is the automation: a program fetches the pages and picks out the data without a person copying it. Copying a table into a spreadsheet by hand is manual data collection, even though the result looks the same. A script that does the same thing every morning across every product page in a category is scraping.

An example. An online store lists laptops on a category page, each with a name, a price and a stock status. A scraper opens every page of the category, finds those three fields in the HTML and writes one row per laptop to a CSV file. Run it daily and you have a price history the store never published as a file.

The data is usually public: product listings, prices, search results, reviews, job ads, news articles and business directories. A scraper that behaves well reads robots.txt (the Robots Exclusion Protocol, RFC 9309, September 2022), keeps to a request rate the site can handle and checks the site's terms. What you may collect, and how, is covered in is web scraping legal and how to check if scraping is allowed. The program itself is explained in what is a web scraper.

A scraper works in five steps: it requests the page, renders it if the content needs JavaScript, parses out the fields, cleans them and stores them.

A web crawler often feeds step 1: it follows links or reads a sitemap to build the list of URLs the scraper visits.

Here is a short example in Python with Requests and Beautiful Soup that sends the request through the ProxyField gateway. Replace <login> and <password> with your credentials; <login>-country-us-city-new_york asks for an exit IP in New York, United States.

# Example: fetch one page through a proxy and parse two fields
import requests
from bs4 import BeautifulSoup

proxy = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
proxies = {"http": proxy, "https": proxy}

resp = requests.get("https://example.com", proxies=proxies, timeout=30)
resp.raise_for_status()  # raises on 403, 429 and other error codes
soup = BeautifulSoup(resp.text, "html.parser")

title = soup.select_one("h1").get_text(strip=True)
links = [a["href"] for a in soup.select("a[href]")]
print({"title": title, "links": links})

Install the two libraries with pip install requests beautifulsoup4. The gateway works over HTTP, HTTPS and SOCKS5, and you can authenticate with an IP allowlist instead of a username and password. Region-level targeting and sticky sessions take further username settings: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default) after the location, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600. The full tutorial, with pagination, errors and storage, is in how to scrape a website; the same job in Python step by step is in Python web scraping.

Yes, but the skill changed: from writing selectors to running browsers, IPs and request rates. Businesses still need public data from sites that have no API, for price monitoring, search monitoring, market research and AI training data. What changed is the environment around the scraper:

What to learn now, in this order:

Use AI as a coding assistant on top of these skills, not instead of them: AI web scraping covers the new tools. Avoid building a skill set around defeating a site's defenses; it is fragile and often breaks the site's terms. How long it takes, and where beginners get stuck, is in how to build a web scraper; the language choice is in best language for web scraping.

Scraping one web page takes one request and one parse; scraping a whole site adds a crawl that finds the pages first. For a single page, the scraper fetches the URL, extracts the fields and stops, as the Python example above does. For a site, it builds the URL list from the sitemap or by following links, then runs the same extraction on every page, at a pace the site can handle; how to crawl a website shows how. Online web scrapers and no-code apps run the same steps behind a form.

Web scraping comes in eight common forms, set apart by where the data sits and how the scraper reaches it. Start with the lightest method that works: official API, then the site's internal JSON, then plain HTML, then a headless browser.

Manual vs automated scraping

Manual scraping is a person copying data from pages by hand; automated scraping is a program doing it. Manual work suits a one-off list of a few dozen rows. Anything that repeats, or runs to thousands of pages, is a job for a script, and that is what "web scraping" normally means; automated web scraping covers scheduling.

HTML and DOM scraping

HTML scraping reads the page's source code as the server sends it and extracts fields with CSS selectors or XPath. DOM scraping does the same on the DOM, the tree of elements the browser builds from that HTML and that JavaScript can change. It works whenever the data is in the HTML the server returns.

Client-side (JavaScript-rendered) scraping

Client-side scraping handles pages that build their content in the browser with JavaScript. The scraper drives a headless browser, a real browser engine without a window, lets the page's scripts run and then reads the finished page. Use it only for the pages that need it.

Official APIs and internal JSON endpoints

An official API is published by the site, with documentation, a key and a rate limit, and its format stays stable. Internal JSON endpoints are the requests a site's own pages make to load their data; you see them in the browser's developer tools, on the Network tab. They arrive already structured, but they are not a published API and can change without notice. Calling them is often called API scraping.

Deep scraping

Deep scraping reaches data that sits several levels down: behind search forms and filters, or across many levels of pagination. The scraper fills in the form or walks every page of results to collect the full set; web scraping pagination shows the patterns. It sends more requests per record, so it needs more care with request rates.

Data scraping is the broad term for any automated extraction; web scraping is data scraping from web pages; screen scraping reads what an application shows on screen.

Web scraping extracts data from pages; the neighboring terms name the step before it, an alternative to it, a part of it, or what comes after it.

• Web crawling discovers pages: a crawler follows links to find and list URLs, as search engines do. Scraping extracts data from the pages. See web crawling vs web scraping.
• An official API gives you the data a site chooses to share, in a stable format. Scraping takes what the page shows. Use the API when it has the fields you need. See web scraping vs API.
• Parsing is one step inside scraping: turning raw HTML or JSON into fields. See data parsing vs web scraping.
• Data mining analyzes data you already have to find patterns; scraping collects it first. See web scraping vs data mining.

Web scraping is used to collect public data that would take too long to copy by hand: prices, search results, product catalogs, market signals, business listings and training data for AI.

• Price monitoring. Track competitors' prices and stock across stores and regions; our price scraper page covers the setup.
• SEO and SERP tracking. Check rankings, ads and search features by country, region and city with proxies for SEO, or get the Google results page as JSON, with the AI Overview, the People Also Ask block and the local pack parsed, from the Google SERP API.
• E-commerce data. Product details, reviews and seller information across marketplaces: e-commerce scraper and the Amazon scraper API.
• Market and alternative data. Job ads, listings, news and company pages as signals for research and investment.
• Lead generation. Lists of businesses from public directories; see lead scraper. Business contact details are often personal data under privacy law, so check the rules first.
• AI training data. Public text and documents to train or ground AI models; see web scraping for AI training. In the EU, respect a site's machine-readable opt-out from text and data mining.

The advantages over manual collection: speed (minutes instead of days), scale (the same code handles ten pages or a million), freshness (scheduled runs) and cost (each extra page costs compute and traffic, not hours of work). More examples by industry are in web scraping use cases; turning scraping into a business is in web scraping business ideas.

Web scraping tools come in four kinds: no-code apps and extensions, scraping APIs, code libraries and frameworks, and ready-made datasets. They differ in the skill they need and in who keeps the scraper working when the site changes.

For a ranked list by skill and scale, see the best web scraping tools; open-source options are in free web scraping tools, and crawlers in best web crawlers. Proxy setup for the browser tools is in our guides for Playwright, Puppeteer and Selenium.

Yes: browser extensions, point-and-click apps and spreadsheet functions all collect web data without code. An extension or app lets you click the fields, repeats the selection across similar pages and exports a CSV file; see web scraper Chrome extensions. In Google Sheets, =IMPORTXML(url, xpath) pulls an element from a page into a cell; see web scraping in Google Sheets and scraping into Excel. These tools suit small, regular jobs; large or JavaScript-heavy sites soon reach their limits.

Each option is priced on a different unit: a scraper API per request or per successful result, your own scraper per GB of proxy traffic plus your servers, and a dataset per dataset or per record.

• Scraper API. You send a URL and get back the page or structured data; the provider runs the browsers, the IPs and the retries. The cost follows the number of pages, not their size. It wins when you have little engineering time or the targets are hard to fetch.
• Your own scraper with proxies. You write and run the code and route it through a proxy pool. A scraper that downloads N pages of X MB each uses about N × X MB of proxy traffic: 10,000 pages of 200 KB are about 2 GB, or $5.90 at $2.95/GB. Browser rendering pushes the size up, because the browser also loads scripts, images and fonts unless you block them. It wins on high volumes and custom logic.
• Datasets. You buy data someone else collected, priced per dataset or per record. It wins when the data is standard and you do not need it fresher than the seller's refresh schedule.

The trade-offs are worked through in scraper API vs proxies; the cost of a scraping job with proxies, page by page, in proxies for web scraping.

ProxyField sells both sides:

Traffic never expires, and all purchases are final. To test on your own targets first, start with the Easy Trial: 1 GB for $2.95.

Sites block scrapers that send too many requests from one IP, come from IP ranges with a poor reputation, or do not look like a real browser. The usual signals are rate limits per IP (answered with 429 Too Many Requests), IP reputation (datacenter ranges are often scored as automated traffic), browser and TLS fingerprints, headers that do not match a real browser, and CAPTCHAs shown when traffic looks suspicious.

Rotating residential IPs spread normal load across many home connections, and each request comes from the kind of address real visitors use. Keep within each site's request limits; rotation is not a way around them. Rotating proxies give you a new IP on every request, or keep one IP for a sticky session when a task needs the same IP across several pages. Proxies change only the IP: headers, fingerprints, cookies and request pacing still matter; see what proxies can and cannot fix. Scraping Google? The pool is clean for Google, checked and segmented, and the Google SERP API returns the results page as JSON if you want data instead.

ProxyField's own rule for scraping: "Configure location targeting and connection rotation in your scraper, respecting website rules and request limits." Headers, fingerprints and CAPTCHAs are covered in our anti-bot guides. New to proxies? Start with what a proxy server is.

Scraping public data is not banned as such in the US, the EU or the UK, but each case turns on the details: personal data (GDPR in the EU and UK), copyrighted content, data behind a login, and a site's terms each bring their own rules. Our guide is web scraping legal goes through the US rules, and web scraping law in the UK and EU the European ones. This page is not legal advice.

Basics: What is a web scraper? · What is data scraping? · What is screen scraping? · What is web data extraction? · What is a web crawler? · What is a scraping bot? · Web scraping use cases

Comparisons: Web crawling vs web scraping · Web scraping vs API · Data parsing vs web scraping · Web scraping vs data mining · Scraper API vs proxies vs datasets · What is a web scraping API?

Tools: Best web scraping tools · Free and open-source web scraping tools · Best no-code web scrapers · Online web scrapers · Web scraper Chrome extensions · Best web crawlers

Without code and at work: Scraping into Excel · Web scraping in Google Sheets · Automated web scraping · Web scraping business ideas · Is web scraping legal?

Running your own scraper? ProxyField residential proxies and mobile proxies rotate the IP on every request or hold a sticky session, with country, region and city targeting. Pay per GB from $1.95/GB, no subscription; start with the Easy Trial (1 GB for $2.95). Traffic never expires, and all purchases are final.