Reddit Scraper: Posts and Comments from Public Pages
To pull posts and comments from a handful of subreddits without paying for Reddit's API, fetch the public pages: ProxyField's Web Scraping API reads 5 subreddits with their top 25 threads every day (3,900 pages a month) for $1.95, at $0.50 per 1,000 successful results, with no Reddit login or API key. Prefer your own script? The Python on this page runs the same job through ProxyField residential proxies on about 0.6 GB a month, inside the Easy Trial: 1 GB for $2.95. The third route is Reddit's free Data API tier with PRAW, if Reddit approves your app for non-commercial use (100 queries per minute per client ID); for brand monitoring, which is commercial use, ask Reddit for Data API access first.
Through the sales team on our 24/7 live chat
| Fetch | Entry | Volume, from |
|---|---|---|
| HTTP | $0.50 | $0.20 |
| JS rendering | $2.00 | $1.00 |
Billed on successful results only.
What one request returns
One request is one public Reddit page, as a logged-out visitor sees it. The Web Scraping API fetches the URL and returns the page; your code pulls the fields out of it. ProxyField sells no separate Reddit API with pre-parsed fields.
Read moreShow less
| Reddit page (one request) | What it holds |
|---|---|
| Subreddit listing (hot, new, top) | About 25 posts: title, author, score, comment count, flair, post time, permalink, plus the "next" link to the following 25 |
| Post with its comment thread | The post text or link, and every visible comment with author, score, time and nesting level; very long threads hide some replies behind "load more" links, which are further pages |
| Search results | Posts and communities that match a keyword |
| User profile (public) | Public posts and comments, karma, account age |
Fields you can parse from Reddit's old interface (old.reddit.com): every post and comment is a div.thing whose attributes carry its data, so no field is guessed from layout. Reddit changes its markup: check these on one page before a long run.
| Attribute | Field | On |
|---|---|---|
data-fullname | unique ID (t3_… post, t1_… comment): your dedupe key | posts, comments |
data-subreddit | subreddit name | posts |
data-author | username | posts, comments |
data-timestamp | creation time, Unix milliseconds | posts, comments |
data-score | score | posts |
data-comments-count | number of comments | posts |
data-permalink | link to the thread or comment | posts, comments |
data-url, data-domain | the linked URL and its domain | link posts |
data-nsfw, data-promoted | NSFW flag, ad flag (skip promoted posts) | posts |
a.title, div.md | title, post or comment text | posts, comments |
span.score (title attribute) | comment score | comments |
nesting of div.child | reply depth | comments |
Private communities, chats and anything behind a login are out of scope: ProxyField's acceptable use policy forbids accessing data behind logins you are not entitled to. The API returns the page itself as HTML, Markdown or plain text, or a JSON envelope with the status code, the final URL, the headers, the cookies and the content; it has no parsed Reddit fields, so your parser reads them.
Posts and comments from a handful of subreddits, without paying for Reddit's API
For a few subreddits, fetch the public pages: two of the three routes need no Reddit account or API key.
| Route | 5 subreddits, top 25 threads daily (3,900 pages a month) | What it needs | Fits |
|---|---|---|---|
| ProxyField Web Scraping API | $1.95 at $0.50 per 1,000 successful results | A ProxyField account | No scripts or IPs to run |
| Your own script on ProxyField residential proxies | About 0.6 GB at an example 150 KB a page: inside the Easy Trial, 1 GB for $2.95 | The two Python scripts below (posts, then comments), one request every 3–5 seconds | A fixed handful of public pages, from the country you choose |
| Reddit's free Data API tier, e.g. with PRAW (the Python Reddit API Wrapper) | No charge from Reddit | An OAuth app that Reddit approves; non-commercial use; Reddit's rate limit, published in its Data API Wiki as 100 queries per minute per OAuth client ID (check the current figure there) | Personal and research projects Reddit approves |
Read moreShow less
What 1,000 posts cost on the Web Scraping API (one page fetched = one result, at $0.50 per 1,000 over plain HTTP):
| You collect | Pages | Cost |
|---|---|---|
| 1,000 posts, listing rows only (title, author, score, comment count, time, permalink) | 40 listing pages of about 25 posts | $0.02 |
| 1,000 posts with their comment threads | 40 listing pages + 1,000 thread pages | $0.52 |
| Each "load more comments" page of a long thread | 1 page | $0.0005 |
Limits to plan for:
- A listing page holds about 25 posts; Reddit's listings stop paging back after about 1,000 posts, so neither route is a full archive.
- Very long threads hide replies behind "load more comments" links, which are further pages.
- Only what a logged-out visitor sees: no private communities, chats or anything behind a login.
New posts only, every day: fetch /r/name/new/, read each post's data-fullname, and stop at the first one you already stored; only the new posts then need their thread pages fetched. A subreddit with 100 new posts a day is about 104 pages, $0.05 a day.
The page size is an example input, not a measurement: fetch one listing and one thread through the proxy and check the dashboard's real-time usage before you plan. Residential traffic never expires, so what a small job does not use stays on your balance. All purchases are final.
Whichever route you take, Reddit's User Agreement and Data API Terms apply to how you collect and use the data.
For brand monitoring: Reddit's Data API or a scraper?
Use Reddit's Data API for brand monitoring, because it is commercial use; use a scraper only for a modest set of public searches and threads. Reddit's Developer Terms, Data API Terms and Responsible Builder Policy set the rules for apps that collect Reddit content: the Data API returns posts and comments as JSON to an OAuth app that Reddit approves, the free tier is rate-limited and meant for non-commercial use, and commercial use needs an agreement with Reddit. A scraper reads the public pages instead, with no Reddit account.
Read moreShow less
| Item | Reddit Data API | Web Scraping API |
|---|---|---|
| Access | Approved OAuth app; commercial use by agreement (Data API Terms) | ProxyField account, no Reddit login |
| Output | Posts and comments as JSON | The public page, for your parser |
| Limits | 100 queries per minute per OAuth client ID on the free tier (Data API Wiki) | Billed per 1,000 successful results: $0.50 over plain HTTP |
| Data | What the API exposes | What a logged-out visitor sees |
| Deleted posts and comments | Reddit's terms tie API use to honoring deletions (Developer Terms) | You see only what is public at fetch time; re-check stored items and drop what was deleted yourself |
| History | Listings page back only so far; neither route is a full archive | The same: about 1,000 posts per listing |
| Fits brand monitoring | Yes, once Reddit approves the commercial use | A modest set of public searches: 1,000 keyword searches a day = 30,000 pages = $15 a month |
Brand monitoring on public pages: a daily new-mentions recipe.
-
Build one search URL per brand term, sorted by new:
https://old.reddit.com/search?q=yourbrand&sort=new. -
Fetch it once a day and read each result's
data-fullnameanddata-permalink. -
Stop at the first
data-fullnameyou already stored: everything above it is new. -
Fetch only the new threads for their comments, and store each permalink so you never fetch it twice.
1,000 keyword searches a day is 30,000 pages a month: $15 over plain HTTP on the Web Scraping API.
Do you need proxies to scrape Reddit?
For a few pages, no. For a sustained job, one IP sending too much gets HTTP 429 or 403, so use rotating IPs or an API; a proxy never grants permission. In one line: a few pages cost nothing from your own connection; a paced recurring job runs on residential proxies from 1 GB for $2.95; with no scripts at all, the Web Scraping API costs $0.50 per 1,000 successful results.
What is paid in Reddit's API is commercial and large-scale access; the free tier still allows 100 queries per minute per OAuth client ID for approved non-commercial apps, per Reddit's Data API Wiki. If Reddit keeps refusing your requests, the supported route is its Data API.
| Your job | Route | Cost |
|---|---|---|
| A few pages, now and then | Your own connection | No charge |
| A paced, recurring job on public pages | Your script on residential proxies: a new IP on every request, or sticky 30 s to 36 h | 1 GB for $2.95, from $1.95/GB at volume |
| Pages without running scripts or IPs | Web Scraping API | $0.50 per 1,000 successful results |
Pacing that keeps a job polite: one request every 3–5 seconds; on a 429, wait and double the wait on each repeat (for example 30 s, then 60 s, then 120 s), and honor a Retry-After header when Reddit sends one; stop the run after three refusals in a row.
Use proxies for location and for spreading a modest job, not to get around limits Reddit sets: slow down, send an honest User-Agent, and read Reddit's Don't break the site rules. More on proxies for Reddit itself: Reddit proxies.
How a request works
You send a Reddit URL, such as https://old.reddit.com/r/webscraping/top/?t=week, and the Web Scraping API returns the page, over plain HTTP or with JS rendering. The API takes a POST to https://api.proxyfield.io/v1/scrape with your API key in an Authorization: Bearer <api_key> header and a JSON body: url, render (false for plain HTTP, true for JS rendering), country and output (html, markdown, text or json). For the new interface, which builds more of the page with scripts, wait_for takes a CSS selector to wait for before the rendered page is returned. Parameter reference: API documentation.
# The request the Web Scraping API takes (access through Contact sales)
curl https://api.proxyfield.io/v1/scrape \
-H "Authorization: Bearer <api_key>" \
-H "Content-Type: application/json" \
-d '{"url": "https://old.reddit.com/r/webscraping/top/?t=week", "render": false, "country": "us", "output": "json"}'Example response (shortened; content is the page's HTML):
{
"metadata": {"id": "req_c39a0f5e72b1d846", "api": "web_scrape", "status": "success", "billed": true},
"status_code": 200,
"final_url": "https://old.reddit.com/r/webscraping/top/?t=week",
"content_type": "text/html; charset=utf-8",
"content": "<!doctype html><html xmlns=\"http://www.w3.org/1999/xhtml\" lang=\"en\">…</html>"
}Pricing: $0.50 per 1,000 pages, from $0.20
Reddit's old interface (old.reddit.com), where Reddit still serves it, is plain HTML, so plain HTTP at $0.50 per 1,000 successful results covers most jobs; the new interface builds more of the page with scripts, and JS rendering costs $2.00 per 1,000, from $1.00 at volume. The volume price of $0.20 starts at 10,000 successful results in one purchase.
You pay only for successful results. Billed: a page Reddit answered with a 2xx status. Not billed: blocked requests, timeouts and errors on our side; a blocked request is retried on new IPs, and the retries are not billed. Every response says whether it was billed. A target's 404 or 410, and a sign-in page the site serves with a 200 status, count as billed results. For availability, contact sales through the sales team on our 24/7 live chat.
Each Reddit page you fetch is one request, so a long thread that Reddit splits over several pages takes one request per page. Payment and balance for the API: the same crypto balance as proxies, which never expires; there is no subscription or monthly fee. All prices: pricing.
Through the sales team on our 24/7 live chat
| Workload (30-day month) | Pages a month | Plain HTTP at $0.50 | At $0.20 (volume) |
|---|---|---|---|
| 5 subreddits, top 25 posts and their threads daily | 3,900 | $1.95 | – |
| 50 subreddits, top 25 posts and their threads daily | 39,000 | $19.50 | $7.80 |
| 1,000 keyword searches a day | 30,000 | $15 | $6 |
Scrape Reddit with Python and ProxyField proxies
This script reads one subreddit's top posts from the old interface through a rotating residential IP. ProxyField gives you a new IP on every request or a sticky session from 30 s to 36 h, and targeting by country, region (state) and city; the pool has 412M+ residential IPs in 195+ countries.
Read moreShow less
# pip install requests beautifulsoup4
import json, requests
from bs4 import BeautifulSoup
proxy = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
url = "https://old.reddit.com/r/webscraping/top/?t=week"
resp = requests.get(url, proxies={"http": proxy, "https": proxy},
headers={"User-Agent": "research-script/1.0 (contact: [email protected])"},
timeout=30)
resp.raise_for_status() # stops on 403 or 429: slow down instead of retrying at once
soup = BeautifulSoup(resp.text, "html.parser")
# Reddit changes its markup: check these selectors before a long run.
for post in soup.select("div.thing[data-permalink]"):
if post.get("data-promoted") == "true":
continue # skip ads
title = post.select_one("a.title")
print(json.dumps({
"id": post.get("data-fullname"),
"subreddit": post.get("data-subreddit"),
"author": post.get("data-author"),
"created_ms": post.get("data-timestamp"),
"score": post.get("data-score"),
"num_comments": post.get("data-comments-count"),
"title": title.get_text(strip=True) if title else "",
"permalink": post.get("data-permalink"),
}))Output shape: one JSON line per post (the values below are placeholders, not real data):
{"id": "t3_xxxxxx", "subreddit": "webscraping", "author": "example_user", "created_ms": "1700000000000", "score": "128", "num_comments": "42", "title": "Example post title", "permalink": "/r/webscraping/comments/xxxxxx/example_post_title/"}Comments from one thread. Fetch a permalink from the listing and walk the comment tree; each comment prints as one JSON line with its ID, depth, author, score and text.
import json, time, requests
from bs4 import BeautifulSoup
proxy = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
headers = {"User-Agent": "research-script/1.0 (contact: [email protected])"}
url = "https://old.reddit.com/r/webscraping/comments/POST_ID/" # a data-permalink from the listing
time.sleep(4) # pace: one request every 3-5 seconds
resp = requests.get(url, proxies={"http": proxy, "https": proxy}, headers=headers, timeout=30)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")
def walk(listing, depth=0):
for c in listing.find_all("div", class_="comment", recursive=False):
score = c.select_one("span.score.unvoted")
text = c.select_one("div.entry div.md")
print(json.dumps({"id": c.get("data-fullname"), "depth": depth,
"author": c.get("data-author"),
"score": score.get("title") if score else None,
"body": text.get_text(" ", strip=True) if text else "[deleted]"}))
replies = c.select_one(":scope > div.child > div.sitetable")
if replies:
walk(replies, depth + 1)
walk(soup.select_one("div.commentarea > div.sitetable"))Output shape: one JSON line per comment, with its reply depth (placeholders):
{"id": "t1_yyyyyy", "depth": 0, "author": "example_user", "score": "15", "body": "Example comment text"}
{"id": "t1_zzzzzz", "depth": 1, "author": "another_user", "score": "4", "body": "Example reply text"}Replies hidden behind "load more comments" are further pages; fetch them the same way, a few seconds apart.
Access opens once the payment is confirmed; the gateway, credentials or allowlist are in your account. Username syntax beyond country and city: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default) after the location, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600.
-
Pick the listing:
/top/?t=week,/new/or a search URL. -
Fetch it through
gate.proxyfield.io:7000, one request every 3–5 seconds. -
Read the score, comment count, title and permalink of each post.
-
Fetch each permalink with the comments script, and store what you collect keyed on
data-fullname.
Residential proxies cost $2.95/GB for 1 GB and from $1.95/GB at volume; the traffic never expires, and all purchases are final. To drive a real browser, use Playwright or Selenium with the same gateway.
What teams use Reddit data for
- Brand and product monitoring: what people say about you and your category.
- Market research: recurring questions, complaints and feature requests in a niche.
- Sentiment analysis on public threads.
- Content research: which topics and formats a community upvotes.
Posting, voting or messaging through a scraper is not data collection; fake accounts and fake engagement are forbidden under ProxyField's acceptable use policy.
Legal questions
Reddit's User Agreement restricts scraping without Reddit's written permission, its Data API Terms govern API access. Collect public pages only, never personal data you do not need, and ask Reddit for API access for commercial projects; more in Is It Legal to Scrape LinkedIn, Instagram, X and Other Social Media? For the wider question, read Is Web Scraping Legal? US, EU and UK Rules (2026).
ProxyField's rule for every site: public pages only, and follow the site's terms.
This is not legal advice.
More scrapers
- Pages for other sites on the same Web Scraping API, not separate APIs: YouTube, social media and news pages.
- Google SERP API: Google results, including Reddit threads that rank, $1.20 per 1,000 successful results, from $0.80.
- ChatGPT scraper API: the ChatGPT answer, $1.50 per 1,000 successful results, from $1.00.
- All scrapers: scrapers.
Reddit Scraper FAQ
24/7 support by live chat, including help with setup and integration.
How much does it cost to scrape Reddit?
ProxyField's Web Scraping API fetches public Reddit pages for $0.50 per 1,000 successful results over plain HTTP, from $0.20 at volume, and $2.00 per 1,000 with JS rendering. Reading 5 subreddits and their top 25 threads every day is 3,900 pages a month, $1.95; 50 subreddits is 39,000 pages, $19.50. Your own scraper on residential proxies starts at 1 GB for $2.95.
What's a good Reddit scraper for pulling posts and comments from a handful of subreddits without paying for the Reddit API?
ProxyField offers two routes without a Reddit key: the Web Scraping API fetches the public pages of 5 subreddits with their top 25 threads daily (3,900 pages a month) for $1.95, at $0.50 per 1,000 successful results, and 1,000 posts with their comment threads cost about $0.52. Or run your own script on ProxyField residential proxies: about 0.6 GB a month at an example 150 KB a page, inside the 1 GB Easy Trial for $2.95. If Reddit approves your app for non-commercial use, its free Data API tier through PRAW allows 100 queries per minute per client ID.
For brand monitoring, should I use Reddit's official Data API or a scraper to collect posts that mention us?
Use Reddit's Data API: brand monitoring is commercial use, and commercial use of Reddit data needs Reddit's agreement. A scraper reads only public pages and does not replace that agreement. It suits a modest set of public searches and threads (1,000 keyword searches a day is 30,000 pages, $15 a month on the Web Scraping API) and checks of how a thread looks from a given country.
How many posts can I scrape from one subreddit?
About 1,000 per listing: a listing page holds about 25 posts, and Reddit's listings stop paging back after about 1,000. On ProxyField's Web Scraping API, those 40 listing pages cost $0.02 and 1,000 posts with their comment threads about $0.52, at $0.50 per 1,000 successful results; sorting by top, new or a time window gives you other sets of posts.
How do I collect only new Reddit posts each day?
Fetch the subreddit's /new/ listing, read each post's data-fullname ID and stop at the first one you already stored. Only the new posts then need their thread pages: 100 new posts a day is about 104 pages, $0.05 a day on the Web Scraping API.
What is a Reddit scraper?
A tool or script that collects public posts, comments, scores and community data from Reddit pages. It reads what a logged-out visitor sees, where Reddit's Data API returns JSON to approved apps.
How do I scrape Reddit?
Fetch the public listing or thread pages, parse the posts and comments, and pace your requests. Send the URLs to the Web Scraping API, or run the Python on this page through rotating residential proxies.
How do I scrape Reddit with Python?
Request an old.reddit.com listing with requests through a proxy and read each post's score, comment count, title and permalink with Beautiful Soup, as in the code on this page. Then fetch each permalink for its comments, a few seconds apart.
How do I scrape Reddit comments?
Fetch the post's permalink: the thread page carries every visible comment with its author, score, time and reply level. Very long threads hide some replies behind "load more" links, which are further pages to fetch.
How do I scrape posts from a subreddit?
Request the subreddit's listing, such as /r/name/new/ or /r/name/top/?t=month, and follow the "next" link to page back through it. Each listing page holds about 25 posts.
Show 13 more questionsShow fewer questions
What data can you scrape from Reddit?
Public Reddit pages show post titles and text, authors, scores, comment counts, flairs, timestamps, comment threads and public profiles. Private communities and anything behind a login are out of scope.
Should I use the Reddit API or a Reddit scraper?
Use Reddit's Data API when Reddit approves your app and its terms fit your use, because it returns clean JSON; commercial use needs Reddit's agreement. A scraper reads the public pages with no Reddit account, which suits smaller research jobs and checking what visitors in a given country see.
Does a Reddit scraper need a Reddit login or API key?
No: a scraper reads public pages as a logged-out visitor. Reddit's Data API is the route that needs an OAuth app and credentials.
Do I need proxies to scrape Reddit now that its API is paid?
For a few pages, no; Reddit's free API tier also still allows 100 queries per minute per OAuth client ID for approved non-commercial apps. For a sustained scrape, one IP sending too much usually starts getting 429 or 403 answers, so use residential proxies for a paced job (a new IP per request, or sticky 30 s to 36 h; 1 GB for $2.95) or let the Web Scraping API fetch pages at $0.50 per 1,000 successful results. A proxy never grants permission: Reddit's terms still apply, and for commercial or large-scale use the supported route is its Data API.
How can I scrape Reddit for free?
For a few pages, a script on your own connection works until Reddit rate-limits it, and Reddit's free Data API tier serves approved non-commercial apps. At any scale you pay for IPs or for an API; the smallest ProxyField purchase is 1 GB of residential traffic for $2.95, with no subscription.
How do I choose a Reddit scraper?
Check what it collects (posts, comments, profiles), whether it needs your Reddit login, how it is billed and whether failed pages cost money. Check too that its use fits Reddit's terms and your own purpose.
Why do teams collect Reddit data?
Reddit threads are candid, so teams use them for brand monitoring, product and market research, sentiment analysis and content ideas. The questions people ask in a niche show demand that keyword tools miss.
What does a Reddit scraper return?
The Web Scraping API returns the Reddit page; your parser turns it into rows such as title, author, score, comment count, time and permalink. A listing page holds about 25 posts; a thread page holds the post and its visible comments.
Can I scrape Reddit without the API?
Yes: public listing and thread pages can be read without the Data API, as in the code on this page. Reddit's terms still apply to how you collect and use the data.
Is there a Reddit scraper on GitHub?
Yes, many open-source Reddit scrapers and API wrappers are on GitHub, PRAW among the wrappers. Wrappers need Reddit API credentials and scrapers need IPs to run at scale, and Reddit changes its pages, so check when a project was last updated.
Can I scrape Reddit with a Chrome extension?
Extensions collect what your own browser shows, which works for a few pages in your logged-in session. For scheduled or large jobs, a script or an API that runs without your browser is the better fit.
Can I scrape Reddit with AI tools like Claude or n8n?
Yes: an AI agent or an n8n workflow can call the Web Scraping API with a Reddit URL and pass the page to a model for summaries. Point it at public pages and pace the calls as you would for any scraper.
What challenges come with scraping Reddit?
Rate limits on logged-out traffic, threads that hide replies behind extra pages, and two interfaces with different markup. Reddit's terms on automated collection matter most, so keep jobs small and public.
Related pages
Collect Reddit pages with the Web Scraping API
Billed on successful results only. Contact sales for the Web Scraping API and the volume you need, or run your own scraper on residential proxies. 24/7 live chat helps with setup and integration.