Twitter (X) Scraper: Public X Pages via API
ProxyField's Web Scraping API is a Twitter scraper for public X posts, with no X API key and no login: send the post links you have and pay $0.50 per 1,000 successful results over plain HTTP (from $0.20 at volume), or $2.00 per 1,000 as rendered pages (from $1.00). Through the JSON behind X's embedded posts, each result gives the text, date, language, author, like count and reply count. 10,000 known posts cost $5, and 300,000 a month cost $150. Finding posts by keyword, or reading a whole profile history, has no logged-out route: it needs the official X API at $0.005 per post read, $5.00 per 1,000. To start, contact sales, or run your own scraper on ProxyField residential proxies with the Easy Trial: 1 GB for $2.95.
Through the sales team on our 24/7 live chat
| Fetch | Entry | Volume, from |
|---|---|---|
| HTTP | $0.50 | $0.20 |
| JS rendering | $2.00 | $1.00 |
Billed on successful results only.
Short answers:
Best tool to scrape tweets without the X API
Pick the tool by the job. Logged out, X serves single posts by their links, the JSON behind its embedded posts, and a basic profile. Keyword search, full timelines, replies and follower lists need a login or the X API.
Read moreShow less
| Your job | Tool without the X API | Cost | What you get |
|---|---|---|---|
| Posts you have links to (your accounts, a client's list, links collected elsewhere) | X's embed JSON through the Web Scraping API, plain HTTP | $0.50 per 1,000, from $0.20 | text, date, language, author, like and reply counts, media links; old posts too |
| The same, from your own script | the script below on ProxyField residential proxies | $2.95 for 1 GB, from $1.95/GB | the same JSON, about 1 to 4 KB per post |
| Post and profile pages as a visitor sees them | Web Scraping API with JS rendering | $2.00 per 1,000, from $1.00 | the rendered page: counts, bio, media |
| A profile's posts without a login | a headless browser such as Playwright on residential proxies | $2.95 for 1 GB, from $1.95/GB | name, bio, counts and some recent posts, not the full history |
| Keyword or hashtag search, full timelines | no logged-out route: the official X API | $0.005 per post read ($5.00 per 1,000) | the full archive back to March 2006 |
| A ready tool with no code | a hosted scraper from a marketplace | set by each seller, per 1,000 posts | check the five points below |
Worked example: 10,000 known post URLs cost $5 through the embed JSON, $20 as rendered pages and $50 through the X API. The ProxyField routes return only posts you already have links to; they do not find posts.
X's embed endpoints work without a login because they serve the posts that websites embed. The JSON endpoint is undocumented and can change. The documented one, oEmbed (publish.x.com/oembed), returns the text, author and date as HTML but no counts. The profile-timeline embed (syndication.twitter.com/srv/timeline-profile/...) answered 429 Rate limit exceeded on both calls in our test on 2026-10-25, so it is not a route for timelines.
Before you pay for a hosted X scraper, check:
-
How does it reach search and timelines? Without the X API, a tool signs in with X accounts or calls X's internal endpoints. Signed-in scraping can get the accounts suspended, and internal endpoints break when X changes them.
-
Is it billed per post returned, or per run, including runs that return nothing?
-
What post cap per profile does it state? The X API's own timeline stops at 3,200.
-
Which fields come back: text, date, language, like and repost counts, media links?
-
When was it last updated?
X's terms forbid scraping without its permission, whichever tool you use (see X's terms and the law).
Fetch known X posts with Python and ProxyField proxies
The script reads post links from post_urls.txt and fetches each post's embed JSON through ProxyField. It retries 3 times with 2, 4 and 8 s pauses and skips deleted or missing posts, for which the endpoint returns {}. Each post becomes one line of JSONL:
import json, re, time
import requests
PROXY = "http://<login>-country-us:<password>@gate.proxyfield.io:7000"
EMBED = "https://cdn.syndication.twimg.com/tweet-result" # the JSON behind X's embedded posts
def fetch(post_id, tries=3):
for i in range(tries):
try:
r = requests.get(EMBED, params={"id": post_id, "lang": "en", "token": "a"},
proxies={"http": PROXY, "https": PROXY}, timeout=30)
except requests.RequestException:
r = None
if r is not None and r.status_code == 200:
data = r.json() # {} for a deleted or missing post
return data if data.get("__typename") == "Tweet" else None
if r is not None and r.status_code == 404:
return None
time.sleep(2 * 2 ** i) # 2, 4, 8 s before the next try
return None
urls = [line.strip() for line in open("post_urls.txt") if line.strip()]
with open("posts.jsonl", "w") as out:
for url in urls:
post_id = re.search(r"/status/(\d+)", url).group(1)
p = fetch(post_id)
if p is None:
continue
out.write(json.dumps({
"id": p["id_str"], "author": p["user"]["screen_name"],
"created_at": p["created_at"], "lang": p["lang"], "text": p["text"],
"likes": p.get("favorite_count"), "replies": p.get("conversation_count"),
}, ensure_ascii=False) + "\n")
time.sleep(1)Read moreShow less
We ran it on 2026-10-25, from our own IP without the proxy line, on https://x.com/jack/status/20, https://x.com/elonmusk/status/1585341984679469056 and a missing post. It wrote two lines and skipped the missing post:
{"id": "20", "author": "jack", "created_at": "2006-03-21T20:50:14.000Z", "lang": "en", "text": "just setting up my twttr", "likes": 309632, "replies": 18117}
{"id": "1585341984679469056", "author": "elonmusk", "created_at": "2022-10-26T18:45:58.000Z", "lang": "en", "text": "Entering Twitter HQ – let that sink in! https://t.co/D68z4K2wq7", "likes": 1218788, "replies": 57964}- Token The endpoint returned
{}for every post when thetokenparameter was missing; with any short value it returned the post. - Size The two responses were 920 and 3,637 bytes of JSON; media adds to it. Connection setup through a proxy adds more, so measure your own GB per 1,000 posts on the Easy Trial.
- Not in the JSON Repost and view counts are not there. For them, render the post page.
For profile pages, drive a headless browser such as Playwright through the same proxy. More on proxies in Python requests.
ProxyField residential proxies cost $2.95/GB for 1 GB and from $1.95/GB at volume, with no subscription; the traffic never expires, and all purchases are final. You get 412M+ residential IPs in 195+ countries, a new IP on every request or a sticky session from 30 s to 36 h, over HTTP, HTTPS or SOCKS5.
What a few hundred thousand tweets a month cost
On ProxyField, 200,000 known posts a month cost $100 through the embed JSON over plain HTTP, 300,000 cost $150 and 500,000 cost $250. As rendered pages they cost $400, $600 and $1,000. If you find the posts by keyword or hashtag, as most sentiment projects do, that is search, and search needs the official X API. At $0.005 per post read, 200,000 posts cost $1,000, 300,000 cost $1,500 and 500,000 cost $2,500 (X's pricing page).
Read moreShow less
| Posts a month | Embed JSON, plain HTTP (volume price) | Rendered post page (volume price) | X API search |
|---|---|---|---|
| Cost per 1,000 posts | $0.50 ($0.20) | $2.00 ($1.00) | $5.00 |
| 100,000 | $50 ($20) | $200 ($100) | $500 |
| 200,000 | $100 ($40) | $400 ($200) | $1,000 |
| 300,000 | $150 ($60) | $600 ($300) | $1,500 |
| 500,000 | $250 ($100) | $1,000 ($500) | $2,500 |
| Finds posts by keyword | no | no | yes |
What each route gives a sentiment model:
- Embed JSON the text, the language tag (
lang), the date and the author, which is all a sentiment model needs, plus like and reply counts to weight posts. - A rendered post page adds the repost count and the media, as X shows them logged out.
- The X API text,
created_at,langandpublic_metricsfor posts it finds by search. Pay-per-use is capped at 3 million post reads a month.
Scoring the sentiment of the text is a separate step on your side, with its own cost; ProxyField sells the fetching. You pay only for successful results. Billed: a page X answered with a 2xx status. Not billed: blocked requests, timeouts and errors on our side; a blocked request is retried on new IPs, and the retries are not billed. Every response says whether it was billed. A target's 404 or 410, and a sign-in page the site serves with a 200 status, count as billed results.
How to scrape all tweets from one X profile with Python, including old posts
Use the X API's full-archive search with from:<handle>, in month windows. It reaches back to March 2006. The user timeline stops at the 3,200 most recent posts, and a logged-out profile page shows even fewer. If the profile is your own, the archive you download from your X account settings holds all your posts at no cost.
X's API has two routes (timelines, search and rate limits docs):
| Route | Endpoint | How far back | Per request | Rate limit per app |
|---|---|---|---|---|
| User posts timeline | GET /2/users/:id/tweets | up to the 3,200 most recent posts | up to 100 | 10,000 requests per 15 min |
| Full-archive search | GET /2/tweets/search/all with from:<handle> | back to March 2006, pay-per-use and Enterprise | up to 500 | 1 request per second, 300 per 15 min |
Read moreShow less
Size the job before you run it:
| Posts on the account | Requests at 500 posts each | Cost at $0.005 per post | Time at 1 request per second |
|---|---|---|---|
| 10,000 | 20 | $50 | under a minute |
| 50,000 | 100 | $250 | about 2 minutes |
| 200,000 | 400 | $1,000 | over 15 min: the 300-per-15-min limit applies |
Month windows add requests for months with few posts, so count one request per month as a floor. The script walks one profile month by month from January 2020. It follows next_token, backs off on 429 and 5xx, drops duplicates and writes one post per line to a JSONL file:
import json, os, time
from datetime import datetime, timezone
import requests
TOKEN = os.environ["X_BEARER_TOKEN"] # from the X developer console
<login> = "example" # the handle, without @
URL = "https://api.x.com/2/tweets/search/all" # full-archive search
HEADERS = {"Authorization": f"Bearer {TOKEN}"}
def months(first, last): # (year, month) windows, last excluded
y, m = first
while (y, m) < last:
ny, nm = (y + 1, 1) if m == 12 else (y, m + 1)
yield (datetime(y, m, 1, tzinfo=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
datetime(ny, nm, 1, tzinfo=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"))
y, m = ny, nm
seen = set()
with open(f"{USER}.jsonl", "a") as out:
for start, end in months((2020, 1), (2026, 10)):
params = {"query": f"from:{USER}", "start_time": start, "end_time": end,
"max_results": 500, "tweet.fields": "created_at,lang,public_metrics"}
wait = 5
while True:
r = requests.get(URL, headers=HEADERS, params=params, timeout=30)
if r.status_code == 429 or r.status_code >= 500:
time.sleep(wait) # rate limit or server error: back off
wait = min(wait * 2, 900)
continue
r.raise_for_status()
wait = 5
data = r.json()
for post in data.get("data", []):
if post["id"] not in seen: # no duplicates across windows
seen.add(post["id"])
out.write(json.dumps(post) + "\n")
token = data.get("meta", {}).get("next_token")
if not token:
break
params["next_token"] = token
time.sleep(1)
print(start[:7], len(seen))What it writes: one line per post, with the fields X documents for tweet.fields=created_at,lang,public_metrics (example values):
{"id": "1712345678901234567", "text": "Shipping the new release today.", "created_at": "2023-10-12T14:03:11.000Z", "lang": "en", "public_metrics": {"retweet_count": 12, "reply_count": 4, "like_count": 87, "quote_count": 1}, "edit_history_tweet_ids": ["1712345678901234567"]}The script also prints the month and the running total, such as 2020-01 312.
Before you run it:
-
Access
Full-archive search is for X API pay-per-use and Enterprise customers; the bearer token comes from the X developer console.
-
Page size
Keep
max_resultsat 500: the endpoint returns 10 posts per request if you leave it out (the allowed range is 10 to 500). -
Cost
Each post returned is a post read at $0.005, and pay-per-use is capped at 3 million post reads a month. The script appends to the file, so a full rerun pays for the same posts again: delete the file first, or start from the last month written.
-
What does not come back
Deleted and protected posts are gone. Like, repost and reply counts in
public_metricsare the totals when you fetch, not when the post went out. -
Filters
Add
-is:retweetto the query to skip reposts and-is:replyto skip replies.
Public X data you can collect
X shows logged-out visitors only part of the site: a public post opened by its own link, the embed data for that post, and a profile's name, bio and counts, often with a sign-in prompt over the rest. Search, full timelines, replies, follower lists and trends need a login, and data behind a login is out of scope.
| Page | URL pattern | What a logged-out visitor usually sees |
|---|---|---|
| Single post | x.com/<user>/status/<id> | Text, author, date, media, reply, repost and like counts |
| Profile | x.com/<user> | Name, handle, bio, location, link, join date, follower and following counts; some posts |
| Embed JSON | cdn.syndication.twimg.com/tweet-result?id=<id> | Text, author, date, language, like and reply counts, media |
| oEmbed | publish.x.com/oembed?url=<post URL> | Text, author and date as embed HTML |
How much X shows without a login changes often, so test your page types before you plan a volume. The API returns the page itself as HTML, Markdown or plain text, or a JSON envelope with the status code, the final URL, the headers, the cookies and the content; it has no parsed X fields, so your parser reads them.
Pricing: from $0.20 per 1,000 X results
X results cost the Web Scraping API's prices: $0.50 per 1,000 successful results over plain HTTP, from $0.20 at volume, and $2.00 per 1,000 with JS rendering, from $1.00. Send post and profile pages on x.com with JS rendering on, because X builds them in the browser; the embed endpoints return JSON, so plain HTTP is enough for them. You pay only for successful results. The volume prices start at 10,000 successful results in one purchase. The API takes a POST to https://api.proxyfield.io/v1/scrape with your API key in an Authorization: Bearer <api_key> header and a JSON body: url, render (false for plain HTTP, true for JS rendering), country and output (html, markdown, text or json). With rendering, wait_for takes a CSS selector to wait for, so the post X builds in the browser is on the page before it comes back. Parameter reference: API documentation.
| Workload (30-day month) | Results | JS rendering at $2.00 | Plain HTTP (embed JSON) at $0.50 |
|---|---|---|---|
| 500 profiles, counts checked daily | 15,000 | $30 | – |
| 10,000 known post URLs, once | 10,000 | $20 | $5 |
| 200,000 post URLs a month | 200,000 | $400 | $100 |
| 300,000 post URLs a month | 300,000 | $600 | $150 |
| 500,000 post URLs a month | 500,000 | $1,000 | $250 |
# The request the Web Scraping API takes (access through Contact sales)
curl https://api.proxyfield.io/v1/scrape \
-H "Authorization: Bearer <api_key>" \
-H "Content-Type: application/json" \
-d '{"url": "https://x.com/jack/status/20", "render": true, "country": "us", "output": "json"}'Example response (shortened; content is the rendered page's HTML):
{
"metadata": {"id": "req_2fb8e4a1b69066f7", "api": "web_scrape", "status": "success", "billed": true},
"status_code": 200,
"final_url": "https://x.com/jack/status/20",
"content_type": "text/html; charset=utf-8",
"content": "<!DOCTYPE html><html dir=\"ltr\" lang=\"en\">…</html>"
}Contact sales through the sales team on our 24/7 live chat.
At the volume prices the 500,000 rendered posts cost $500 and the 500,000 embed results $100. Payment and balance: the same crypto balance as proxies, which never expires; there is no subscription or monthly fee. All prices: pricing.
Through the sales team on our 24/7 live chat
The X API or scraping?
X's official API is the documented route to search, timelines and follower data. It is priced per use with no subscription: $0.005 per post read and $0.010 per user lookup, capped at 3 million post reads a month on pay-per-use (X API pricing). Scraping reads only what X shows logged-out visitors: known posts and profile counts, not search or full timelines.
| Item | X API | Web Scraping API |
|---|---|---|
| Search and timelines | Yes, full archive back to March 2006 | No: they need a login |
| Known post URLs and profile counts | Yes | Yes, as shown logged out |
| Approval | X developer account | None from X |
| Cost per 1,000 posts | $5.00 | $0.50 (embed JSON, from $0.20) or $2.00 rendered (from $1.00) |
What teams use X data for
- Sentiment analysis score the text of posts found through the X API, or of a known list of posts fetched through the embed JSON.
- Brand and campaign monitoring reach and engagement on known posts by your brand or competitors.
- Influencer research follower counts and bios for a list of accounts, tracked over time.
- Research and journalism archiving public posts cited in a story or study.
Not allowed on ProxyField: spam, fraud (including fake accounts or engagement), accessing data behind logins you are not entitled to and breaking the law. Running your own X accounts through fixed IPs is a different job: X (Twitter) proxies.
X's terms and the law
X's terms of service forbid crawling or scraping without its prior written consent, so scraping X breaches its terms, separate from whether it breaks a law. Posts are personal data in the EU and UK, so GDPR needs a lawful basis and data minimisation, and post texts and media can be protected by copyright. The full answer is in Is It Legal to Scrape LinkedIn, Instagram, X and Other Social Media? and Is Web Scraping Legal? US, EU and UK Rules (2026).
ProxyField's acceptable use policy allows collecting public web data and social media monitoring, and forbids fraud, including fake accounts or engagement, and accessing data behind logins you are not entitled to. This is not legal advice.
ProxyField's rule for every site: public pages only, and follow the site's terms.
More scraper pages
- Social media scraper public pages from several networks through the same Web Scraping API.
- Instagram what their public pages show logged out, fetched through the same Web Scraping API.
- TikTok what their public pages show logged out, fetched through the same Web Scraping API.
- LinkedIn what their public pages show logged out, fetched through the same Web Scraping API.
- Web Scraping API any URL, plain HTTP or with JS rendering, from $0.20 per 1,000 successful results.
- Google SERP API the Google results page as JSON, $1.20 per 1,000 successful results, from $0.80.
- All scraper APIs
Twitter FAQ
24/7 support by live chat, including help with setup and integration.
What is the best tool to scrape Twitter?
ProxyField's Web Scraping API fetches public X posts for $0.50 per 1,000 through X's embed JSON (from $0.20), with text, date, language, likes and replies, and rendered post and profile pages for $2.00 per 1,000 (from $1.00), billed on successful results. The official X API is the tool for keyword search, full timelines and old posts by date, at $0.005 per post read. A script of your own on residential proxies, from $1.95/GB, fits teams that want to run the scraper themselves.
How much would it cost to scrape a few hundred thousand tweets a month for sentiment analysis?
If you already have the post URLs, ProxyField's Web Scraping API fetches 200,000 for $100 and 500,000 for $250 through X's embed JSON ($0.50 per 1,000, $0.20 at volume), or $400 and $1,000 as rendered pages. If you find the posts by keyword, you need the X API at $5.00 per 1,000: 200,000 posts cost $1,000 and 500,000 cost $2,500 a month. The embed JSON gives the text, language and date a sentiment model needs; scoring is a separate step with its own cost.
How do I scrape all the tweets from one X profile with Python, including posts from a few years back?
Use the X API's full-archive search, GET /2/tweets/search/all with the query from:<handle>, in month windows with start_time and end_time, following next_token and backing off on 429, as in the script on this page. The user timeline endpoint stops at the 3,200 most recent posts, and a logged-out profile page shows even fewer. Set max_results to 500 (the default is 10). A 50,000-post account is about 100 requests and $250 at $0.005 per post; the endpoint allows 1 request per second and 300 per 15 minutes per app.
What's the best tool to scrape tweets from X without paying for access to the official X API?
ProxyField's Web Scraping API fetches public X posts you have links to for $0.50 per 1,000 (from $0.20), through the JSON behind X's embedded posts, with no login and no X API key: text, date, language, author, likes and replies. 10,000 known posts cost $5, against $50 through the X API. For profiles, a headless browser on residential proxies ($2.95 for 1 GB) or rendered pages at $2.00 per 1,000 read what a logged-out visitor sees. Keyword search and full timelines have no route without the API or a login, and X's terms forbid scraping without permission.
How do I scrape Twitter?
Collect the URLs of the public posts and profiles you need, fetch posts through X's embed JSON and profiles with a headless browser or a scraping API with JS rendering, and read the text and counts. Rotate IPs, keep the rate low and stop at X's sign-in prompt rather than logging in.
What data can you scrape from Twitter?
From logged-out pages: a public post's text, author, date, language, media and reply, repost and like counts, and a profile's name, bio, location, link, join date and follower counts. Search results, full timelines, replies and follower lists need a login and are out of scope.
Does Twitter have an API for scraping?
X has an official API for reading posts, users and search, priced per use: $0.005 per post read and $0.010 per user lookup, capped at 3 million post reads a month. Its full-archive search goes back to March 2006.
How do I scrape Twitter without the API?
Use the JSON behind X's embedded posts for known post URLs, as in the script on this page, and a headless browser through rotating residential IPs for profile pages. Without the API you get only what X shows logged-out visitors.
Do I need to log in to extract data from X.com?
For search, timelines, replies and follower lists, yes, and that is out of scope: X's terms forbid scraping, and ProxyField's acceptable use policy forbids accessing data behind logins you are not entitled to. Individual public posts and basic profile data are visible logged out.
How do I scrape Twitter with Python?
For known post URLs, call https://cdn.syndication.twimg.com/tweet-result with the post ID through a proxy using requests, as in the script on this page; it returns JSON with the text and counts. For a whole profile history, use the X API's full-archive search; for profile pages, drive Playwright through the proxy.
Show 10 more questionsShow fewer questions
How do I scrape tweets from a profile?
Logged out, a profile shows only some of its posts. The X API's user timeline returns up to the 3,200 most recent, and full-archive search with from:<handle> returns older ones at $0.005 per post. For posts whose URLs you already have, fetch each through the embed JSON.
How do I scrape Twitter followers?
Follower lists need a login, so they are out of scope for scraping; the X API returns them, priced per user returned. The follower count itself is on the public profile page.
How do I scrape Twitter media?
A public post's page links to its images and videos, which you can save for analysis. Media are usually protected by copyright, so store links and metadata rather than republishing files.
How do I scrape replies to a tweet?
X shows the replies under a post mainly to signed-in users, so a logged-out scraper sees few or none. The X API's search by conversation is the route for replies.
How do I scrape old tweets?
If you have their URLs, old public posts open by link: the embed JSON returned a post from March 2006 in our test. Finding old posts by keyword or date needs the X API's full-archive search, which goes back to March 2006.
Can I scrape X search results for a keyword or hashtag?
Not logged out: X's search is behind a login, so scraping it means signing in with an account. X's terms forbid scraping without its permission, and ProxyField's acceptable use policy forbids accessing data behind logins you are not entitled to. Use the X API's search, at $0.005 per post read.
How can I scrape Twitter for free?
For a few posts, a Python script calling X's embed JSON from your own IP costs nothing. At volume you pay for IPs, an API or X's own API; ProxyField's smallest purchase is 1 GB of residential traffic for $2.95.
What is the fastest Twitter scraper?
For known posts, the embed JSON is the lightest method: it returned 920 bytes for a short post and 3,637 bytes for one with media in our test, and needs no browser. Profile pages need a headless browser and take longer. ProxyField publishes no speed figure for the Web Scraping API.
Is there a Twitter scraper on GitHub?
Yes, several, but many stopped working when X put most of the site behind a login, and many of the rest log in with accounts, which breaks X's terms. Check the last update and whether a project works logged out.
Do X's terms allow web scraping?
No: X's terms of service forbid crawling or scraping without X's prior written consent. Whether a particular scrape also breaks a law is a separate question; this is not legal advice.
Related pages
Collect Twitter (X) pages with the Web Scraping API
Billed on successful results only. Contact sales for the Web Scraping API and the volume you need, or run your own scraper on residential proxies. 24/7 live chat helps with setup and integration.