How to Check If a Website Allows Scraping

An official API is the clearest "yes": it tells you which data the site is willing to share and on what terms. Search the footer for "Developers", "API" or "Partners", or search the web for "[site] API". Read the API license too: many allow only some uses, for example eBay points automated integrations to its API License Agreement, and TikTok's research access is limited to academic and not-for-profit researchers.

The terms of use decide what you agree to, so they carry the most weight. Open the "Terms", "Terms of Service" or "Legal" link, press Ctrl+F (Cmd+F on a Mac) and search for these words:

• robot, robots, bot
• scrape, scraper, scraping
• crawl, crawler, spider
• automated, automatic, script
• data mining, extract, harvest

Typical clauses:

• Tinder (Terms of Use, effective March 5, 2026): you may not "use any robot, crawler, site search/retrieval application, proxy or other manual or automatic device, method or process to access, retrieve, index, 'data mine,' or in any way reproduce or circumvent the navigational structure or presentation of our Services or its contents". That covers its private API as much as its website.
• eBay (User Agreement): no "robot, spider, scraper, data mining tools, data gathering and extraction tools, or other automated means" for any purpose "except with the prior express permission of eBay".

Two questions decide how much the terms bind you. Did you accept them, for example by creating an account or clicking "I agree"? And do they restrict your specific act? Scraping while logged in is where terms bind most clearly, as hiQ found when it lost to LinkedIn on contract (docket). The legal background is in is web scraping legal.

robots.txt is a plain text file at the root of the site, for example https://shopee.sg/robots.txt. RFC 9309 (2022) standardizes it as rules that "crawlers are requested to honor", and says the rules "are not a form of access authorization". It is the site owner's stated wish, not a contract, but European regulators treat it as a signal: the EDPB's July 2026 draft guidelines list robots.txt among the measures by which websites "clearly oppose the scraping of their content".

How to read it:

Example: Shopee Singapore. On October 23, 2026, shopee.sg's robots.txt gave Googlebot and Bingbot a Crawl-delay of 0.1 and all other agents a Crawl-delay of 1, and disallowed /cart/, /checkout/, /user/, /me/, /order/ and several search and recommendation paths for everyone. Product pages were not disallowed. That is a "yes, at one request per second, outside account areas" from robots.txt, which still leaves the terms of use to check.

Example: a full stop. Reddit's robots.txt, read the same day, was User-agent: * followed by Disallow: /, with a comment pointing to its Public Content Policy.

Check robots.txt in code with Python's standard library:

from urllib import robotparser

rp = robotparser.RobotFileParser()
rp.set_url("https://shopee.sg/robots.txt")
rp.read()

agent = "ExamplePriceBot"
for url in ["https://shopee.sg/cart/", "https://shopee.sg/user/account"]:
    print(url, "allowed" if rp.can_fetch(agent, url) else "disallowed")
print("crawl delay:", rp.crawl_delay(agent))

Python's parser matches path prefixes and does not implement every wildcard rule of RFC 9309, so check rules with * or $ by hand. For a crawler that applies robots.txt on every request, see the Python web crawler guide.

The site's behavior is the last check:

• A login wall means the data is not public; logging in binds you to the terms.
• A CAPTCHA or "verify you are human" page means the site is filtering bots on that path.
• 403 Forbidden, 429 Too Many Requests or an IP block means stop or slow down; 429 responses often carry a Retry-After header. See 429 Too Many Requests and 403 Forbidden.
• A cease-and-desist letter is the clearest "no". In Craigslist v. 3Taps (2013), scraping on after a letter and an IP block was held to be possible unauthorized access under the CFAA (opinion).

Even where a site allows scraping, the data can carry its own rules. Prices, specifications and public statistics are facts. Names, profiles, emails and photos of people are personal data under the GDPR in Europe, whoever published them. Articles, reviews and images are copyrighted. AI training adds text-and-data-mining opt-outs. The overview is in what data is legal and safe to scrape.

If the terms are silent and robots.txt allows your paths, keep a record of what you checked and when. If the terms forbid automation but you need the data, email the site: a short note with who you are, what you want, how often and from which user agent often gets a license or an API key. Written permission settles the question.

Ready to collect from sites that pass these checks: proxies for web scraping and residential proxies.

• RFC 9309, Robots Exclusion Protocol (2022)
• EDPB, Guidelines 03/2026 on web scraping in the context of generative AI (adopted July 7, 2026, public consultation)
• Tinder Terms of Use (effective March 5, 2026)
• eBay User Agreement
• eBay API License Agreement
• shopee.sg/robots.txt
• reddit.com/robots.txt
• Python documentation, urllib.robotparser
• hiQ Labs v. LinkedIn, N.D. Cal. No. 3:17-cv-03301, docket
• Craigslist v. 3Taps, 964 F. Supp. 2d 1178 (N.D. Cal. 2013)

This is not legal advice. Laws, court decisions and sites' terms change, and the primary sources linked on this page are the authority.