Is Web Scraping Legal? US, EU and UK Rules (2026)

Scraping public data is generally legal; the risk rises when any of these five things is not true:

Screen scraping, data scraping and web crawling are judged the same way, and so is every tool: a Python script, cURL, a Selenium-driven browser or a scraping API. Courts and regulators look at what you accessed, how, and what you did with it. If you are new to the technique, start with what web scraping is. More legal guides: What data is legal and safe to scrape?, scraping Amazon, scraping Google, scraping social media, scraping ecommerce sites, AI scraping and using proxies.

Usually yes: pages anyone can open without logging in are the lowest-risk kind in the US, the EU and the UK. The Ninth Circuit put the CFAA test in one sentence: "authorization is only required for password-protected sites or sites that otherwise prevent the general public from viewing the information" (hiQ v. LinkedIn, 2022). That ruling has limits: it affirmed a preliminary injunction on "serious questions", it binds only courts in the Ninth Circuit, it covers the CFAA only, and hiQ later lost on contract (November 2022 ruling, December 2022 consent judgment). "Public" settles the access question, not the rest:

No law makes robots.txt binding on its own: RFC 9309, the IETF standard published in September 2022, says its rules "are not a form of access authorization". Ignoring it still costs you: it shows what the site owner asks of crawlers, ignoring it weakens a good-faith position in a contract or access dispute, and in the EU the CNIL lists excluding sites that object to scraping as one of the measures for scraping under legitimate interests (CNIL focus sheet).

A 3-line file that asks every crawler to stay out of /checkout/:

User-agent: *
Disallow: /checkout/
Allow: /

To test a URL from Python before you fetch it, use the standard library: RobotFileParser("https://example.com/robots.txt"), then .read() and .can_fetch(user_agent, url); the Python web scraping tutorial runs this check before every page. Terms, robots.txt and APIs together: how to check if scraping is allowed.

Sources 18 U.S.C. § 1030; Van Buren v. United States; hiQ Labs v. LinkedIn, 9th Cir. 2022; GDPR, Regulation (EU) 2016/679; Database Directive 96/9/EC.

In the US, scraping public pages is hard to fit under the federal anti-hacking law after Van Buren and hiQ, but contract and copyright claims remain. The CFAA question is whether you passed a gate; the contract question is whether you agreed not to scrape.

What the CFAA says

The Computer Fraud and Abuse Act, 18 U.S.C. § 1030, punishes accessing a computer "without authorization" or in a way that "exceeds authorized access" and obtaining information. It is a criminal statute, and subsection (g) also lets anyone who suffers damage or loss from a violation bring a civil suit.

Van Buren v. United States (2021): gates up or down

On June 3, 2021, the Supreme Court read the CFAA narrowly in Van Buren v. United States, decided 6-3 in an opinion by Justice Barrett. A person "exceeds authorized access" when they enter parts of a computer, such as files, folders or databases, that are off limits to them. Misusing information you were entitled to reach is not enough. The Court described this as a "gates-up-or-down inquiry": either you can enter an area of the system or you cannot. It did not decide whether a gate has to be technical, such as a password, or can also be set by a contract or policy.

hiQ Labs v. LinkedIn (9th Cir. 2022)

On April 18, 2022, after the Supreme Court sent the case back in light of Van Buren, the Ninth Circuit again upheld the preliminary injunction that stopped LinkedIn from blocking hiQ (opinion, No. 17-16783). hiQ scraped LinkedIn profiles that anyone could view without logging in. The court held that hiQ had raised serious questions that this was not access "without authorization" under the CFAA, even after LinkedIn sent a cease-and-desist letter and blocked it, reasoning that "authorization is only required for password-protected sites or sites that otherwise prevent the general public from viewing the information."

The ruling has two limits. First, it is a ruling on a preliminary injunction, and it binds courts in the Ninth Circuit only. Second, hiQ later lost on contract: in November 2022 the district court ruled that hiQ had breached LinkedIn's User Agreement, through its scraping and through fake profiles, and in December 2022 the case ended in a consent judgment against hiQ (district court docket, No. 3:17-cv-03301). The CFAA win did not make the scraping risk-free.

DOJ charging policy (Justice Manual 9-48.000)

Federal prosecutors follow the Justice Department's CFAA charging policy, Justice Manual 9-48.000, revised in May 2022 after Van Buren. It says good-faith security research should not be charged, and that breaking an access restriction found only in a contract or terms of service is not, by itself, enough for federal criminal charges. It is internal policy: it does not bind civil suits or state law.

Contract claims and a 2024 ruling on platform terms

Contract is now the main US risk for scraping public pages, and it depends on whether you are bound by the terms. In January 2024, a federal court in the Northern District of California held that a social media company's terms did not stop a data company from scraping public data while logged out (order granting summary judgment, No. 3:23-cv-00077). The court read the terms as governing account holders using the platform while logged in.

The lesson cuts both ways. A scraper that logs in has accepted the terms, as hiQ's contract loss shows. New rulings as they come: web scraping legal news.

Sources 18 U.S.C. § 1030; Van Buren v. United States; hiQ Labs v. LinkedIn, 9th Cir. 2022; hiQ district court docket; Justice Manual 9-48.000; N.D. Cal. No. 3:23-cv-00077, Dkt. 181.

In the EU, scraping public pages is not illegal as such, but the GDPR applies to any personal data you collect, and the database right protects investment in databases.

Member states add their own rules on top; country by country in web scraping law in the UK and EU.

Sources GDPR; EDPB basic principles; CNIL focus sheet on web scraping; Directive 96/9/EC; C-203/02; C-30/14.

In the UK, scraping public pages is generally legal, but unauthorized access is a crime, personal data falls under the UK GDPR, and the database right still applies.

Sources Computer Misuse Act 1990, s.1; ICO joint statement on data scraping; GOV.UK database rights; CDPA 1988, Part I.

No single global rule decides whether web scraping is legal. Most countries apply the same four bodies of law: contract, copyright, privacy and computer-misuse law, in the same way as the US, the EU and the UK.

Sources ICO joint statement.

A commercial purpose does not by itself make scraping illegal; it raises the stakes on contract, copyright, the database right and privacy law.

Building a product on scraped public data is generally lawful when what you use is facts; the risk comes from the source's terms, copied expression, a substantial part of a database and personal data.

Facts versus expression. Prices, dates, specifications and other facts are not protected by copyright. The US Copyright Office says "copyright does not protect facts, ideas, systems, or methods of operation, although it may protect the way these things are expressed", and that "the original authorship appearing on a website may be protected by copyright", which includes writings, artwork and photographs (Copyright Office FAQ). Product descriptions, reviews, articles and photos are expression; republishing them is where copyright risk sits.

Worked example: hiQ. hiQ built a workforce-analytics product on public LinkedIn profiles. It won the CFAA round in the Ninth Circuit in April 2022, then lost on LinkedIn's User Agreement in November 2022 and ended in a consent judgment in December 2022 (docket). The access question was settled in its favor; the contract and the people in the data were not.

What to check before you build a product on scraped data:

• Source terms. Did you accept the site's terms, for example by logging in, and do they forbid scraping or commercial reuse?
• Personal data. If the data includes people, the GDPR and the UK GDPR require a lawful basis.
• Redistribution. Deriving facts is lower risk than republishing text and images.
• Database extraction. In the EU and UK, are you taking a substantial part of one site's database?
• Official API. An API or a data license removes most of these questions.

If the checks come out clean, see proxies for web scraping for the infrastructure side.

Sources US Copyright Office FAQ; GDPR; Directive 96/9/EC; hiQ docket.

Reading published prices, fares, ticket prices and job ads is generally lawful; the risk comes from site terms, the EU and UK database right, personal data and, for tickets, getting around purchase controls.

Price scraping and price monitoring

Price scraping is usually legal, because prices are facts. The risk sits in the site's terms, if you accepted them, and in the EU and UK in taking a substantial part of a retailer's database (BHB v William Hill). Monitoring a set of products is a different act from copying a whole catalog. For the product side, see price scraping with the Web Scraping API.

Airfares and travel sites

Airline fares are facts too, but airlines enforce their website terms. In Ryanair v PR Aviation, the EU's top court held that the Database Directive does not stop an airline from restricting the use of its flight data by contract where neither copyright nor the database right protects it. Reading published fares for comparison is lower risk than automating bookings.

Ticket prices

The US BOTS Act targets getting around ticket-purchase controls, not reading prices. Per the FTC's page on the Better Online Ticket Sales Act, the 2016 law prohibits circumventing a security measure, access control system or other technological control that a ticket issuer uses to enforce posted purchase limits or to keep its online purchasing order rules intact. Reading published prices is a different act.

Job sites

Job-data scraping is the context of the best-known US case: hiQ scraped public LinkedIn profiles for workforce analytics (Ninth Circuit opinion). Public job ads are generally lower risk. Candidate profiles are personal data, so the GDPR and the UK GDPR apply to them in Europe, and hiQ's later contract loss shows what accepted terms can do.

Sources C-203/02; C-30/14; FTC, BOTS Act; hiQ v. LinkedIn.

Getting past a CAPTCHA, Cloudflare or another anti-bot system is legally risky: a block is a signal that you may not have permission, and circumventing it can bring computer-misuse, contract and copyright claims.

• Computer-misuse risk. Van Buren framed the CFAA around gates (opinion). A barrier the site puts up against you looks more like a gate than a public page does. In the UK, the Computer Misuse Act turns on whether access is unauthorized and whether you know it.
• Contract risk. Site owners write rules for bots into their terms. Cloudflare publishes sample terms that sites can add to their terms of use to forbid bots from scraping content for AI training.
• Anti-circumvention risk. 17 U.S.C. § 1201 forbids circumventing a technological measure that effectively controls access to a copyrighted work. Where an anti-bot barrier protects copyrighted content, this adds a separate claim.

A block is not a use case. If a site blocks your access, the lawful routes are its official API, a data license or the owner's permission. What a Cloudflare block looks like from the scraper's side: Cloudflare 403 Forbidden.

Sources Van Buren; 18 U.S.C. § 1030; Computer Misuse Act 1990, s.1; Cloudflare sample terms; 17 U.S.C. § 1201.

Yes, you can be sued for web scraping, most often for breach of contract or copyright, and in Europe you can be fined for misusing personal data. The claims that come up most:

• Breach of contract or terms of service. hiQ won on the CFAA and lost on LinkedIn's User Agreement (docket).
• Computer-misuse claims. The CFAA has a civil action in subsection (g) (18 U.S.C. § 1030); after Van Buren it is hard to use against public pages, not against logins or barriers.
• Copyright and anti-circumvention. Copying expression, or breaking a measure that protects a work (17 U.S.C. § 1201).
• Database right. Taking a substantial part of a European database (C-203/02).
• GDPR enforcement. France's CNIL fined a company whose browser extension collected the contact details of people whose profiles its customers visited on a professional social network; the findings included the lack of a lawful basis under Article 6 (EDPB news item).

In practice, the first step is often an IP block or a cease-and-desist letter, not a lawsuit. In hiQ, the dispute began with a cease-and-desist letter and a block (Ninth Circuit opinion).

Sources as linked in the list above.

Yes, attackers use scraping to collect names, emails and phone numbers for phishing, identity fraud and unwanted marketing. The ICO's joint statement lists these harms, from targeted cyberattacks and identity fraud to monitoring, profiling and surveillance of individuals. The same technique, used to collect public prices or for research, is lawful. What separates the two is the type of data and the purpose, not the code.

Sources ICO joint statement on data scraping.

These seven practices follow from the sources above:

1. Collect public pages only. Stay out of areas behind logins and barriers.
2. Read the terms and robots.txt. RFC 9309 standardizes robots.txt and says its rules are not a form of access authorization, so robots.txt neither grants permission nor replaces the terms. It does show what the site owner asks of crawlers, and the CNIL lists excluding sites that object to scraping as one possible measure. How to read both: how to check if scraping is allowed.
3. Prefer an official API or a data license where the target offers one.
4. Collect no personal data without a lawful basis, and only the fields you need (data minimization).
5. Rate-limit your requests so the site's service is not harmed.
6. Identify your bot where possible, with a user agent and a contact.
7. Keep records of what you collected, from where, when and on what basis.

Whether scraping is ethical, beyond legal, is the subject of our ethical web scraping guide.

Sources RFC 9309; CNIL focus sheet; EDPB basic principles; ICO joint statement.