Site scraper

Twitter (X) Scraper: Public X Pages via API

HTTP $0.50JS rendering $2.00 per 1,000 results or your own scraper from $1.95/GB

ProxyField's Web Scraping API is a Twitter scraper for public X posts, with no X API key and no login: send the post links you have and pay $0.50 per 1,000 successful results over plain HTTP (from $0.20 at volume), or $2.00 per 1,000 as rendered pages (from $1.00). Through the JSON behind X's embedded posts, each result gives the text, date, language, author, like count and reply count. 10,000 known posts cost $5, and 300,000 a month cost $150. Finding posts by keyword, or reading a whole profile history, has no logged-out route: it needs the official X API at $0.005 per post read, $5.00 per 1,000. To start, contact sales, or run your own scraper on ProxyField residential proxies with the Easy Trial: 1 GB for $2.95.

Web Scraping API per 1,000 successful results
Web Scraping API prices per 1,000 successful results
FetchEntryVolume, from
HTTP$0.50$0.20
JS rendering$2.00$1.00

Billed on successful results only.

Your own scraper on residential proxies
from $1.95/GB Easy Trial: 1 GB for $2.95
gate.proxyfield.io:7000

Short answers:

Data

Best tool to scrape tweets without the X API

Pick the tool by the job. Logged out, X serves single posts by their links, the JSON behind its embedded posts, and a basic profile. Keyword search, full timelines, replies and follower lists need a login or the X API.

Read moreShow less
Your jobTool without the X APICostWhat you get
Posts you have links to (your accounts, a client's list, links collected elsewhere)X's embed JSON through the Web Scraping API, plain HTTP$0.50 per 1,000, from $0.20text, date, language, author, like and reply counts, media links; old posts too
The same, from your own scriptthe script below on ProxyField residential proxies$2.95 for 1 GB, from $1.95/GBthe same JSON, about 1 to 4 KB per post
Post and profile pages as a visitor sees themWeb Scraping API with JS rendering$2.00 per 1,000, from $1.00the rendered page: counts, bio, media
A profile's posts without a logina headless browser such as Playwright on residential proxies$2.95 for 1 GB, from $1.95/GBname, bio, counts and some recent posts, not the full history
Keyword or hashtag search, full timelinesno logged-out route: the official X API$0.005 per post read ($5.00 per 1,000)the full archive back to March 2006
A ready tool with no codea hosted scraper from a marketplaceset by each seller, per 1,000 postscheck the five points below

Worked example: 10,000 known post URLs cost $5 through the embed JSON, $20 as rendered pages and $50 through the X API. The ProxyField routes return only posts you already have links to; they do not find posts.

X's embed endpoints work without a login because they serve the posts that websites embed. The JSON endpoint is undocumented and can change. The documented one, oEmbed (publish.x.com/oembed), returns the text, author and date as HTML but no counts. The profile-timeline embed (syndication.twitter.com/srv/timeline-profile/...) answered 429 Rate limit exceeded on both calls in our test on 2026-10-25, so it is not a route for timelines.

Before you pay for a hosted X scraper, check:

  1. How does it reach search and timelines? Without the X API, a tool signs in with X accounts or calls X's internal endpoints. Signed-in scraping can get the accounts suspended, and internal endpoints break when X changes them.

  2. Is it billed per post returned, or per run, including runs that return nothing?

  3. What post cap per profile does it state? The X API's own timeline stops at 3,200.

  4. Which fields come back: text, date, language, like and repost counts, media links?

  5. When was it last updated?

X's terms forbid scraping without its permission, whichever tool you use (see X's terms and the law).

Do it yourself

Fetch known X posts with Python and ProxyField proxies

The script reads post links from post_urls.txt and fetches each post's embed JSON through ProxyField. It retries 3 times with 2, 4 and 8 s pauses and skips deleted or missing posts, for which the endpoint returns {}. Each post becomes one line of JSONL:

Python
import json, re, time
import requests

PROXY = "http://<login>-country-us:<password>@gate.proxyfield.io:7000"
EMBED = "https://cdn.syndication.twimg.com/tweet-result"   # the JSON behind X's embedded posts

def fetch(post_id, tries=3):
    for i in range(tries):
        try:
            r = requests.get(EMBED, params={"id": post_id, "lang": "en", "token": "a"},
                             proxies={"http": PROXY, "https": PROXY}, timeout=30)
        except requests.RequestException:
            r = None
        if r is not None and r.status_code == 200:
            data = r.json()                                   # {} for a deleted or missing post
            return data if data.get("__typename") == "Tweet" else None
        if r is not None and r.status_code == 404:
            return None
        time.sleep(2 * 2 ** i)                                # 2, 4, 8 s before the next try
    return None

urls = [line.strip() for line in open("post_urls.txt") if line.strip()]
with open("posts.jsonl", "w") as out:
    for url in urls:
        post_id = re.search(r"/status/(\d+)", url).group(1)
        p = fetch(post_id)
        if p is None:
            continue
        out.write(json.dumps({
            "id": p["id_str"], "author": p["user"]["screen_name"],
            "created_at": p["created_at"], "lang": p["lang"], "text": p["text"],
            "likes": p.get("favorite_count"), "replies": p.get("conversation_count"),
        }, ensure_ascii=False) + "\n")
        time.sleep(1)
Read moreShow less

We ran it on 2026-10-25, from our own IP without the proxy line, on https://x.com/jack/status/20, https://x.com/elonmusk/status/1585341984679469056 and a missing post. It wrote two lines and skipped the missing post:

{"id": "20", "author": "jack", "created_at": "2006-03-21T20:50:14.000Z", "lang": "en", "text": "just setting up my twttr", "likes": 309632, "replies": 18117}
{"id": "1585341984679469056", "author": "elonmusk", "created_at": "2022-10-26T18:45:58.000Z", "lang": "en", "text": "Entering Twitter HQ – let that sink in! https://t.co/D68z4K2wq7", "likes": 1218788, "replies": 57964}
  • Token The endpoint returned {} for every post when the token parameter was missing; with any short value it returned the post.
  • Size The two responses were 920 and 3,637 bytes of JSON; media adds to it. Connection setup through a proxy adds more, so measure your own GB per 1,000 posts on the Easy Trial.
  • Not in the JSON Repost and view counts are not there. For them, render the post page.

For profile pages, drive a headless browser such as Playwright through the same proxy. More on proxies in Python requests.

ProxyField residential proxies cost $2.95/GB for 1 GB and from $1.95/GB at volume, with no subscription; the traffic never expires, and all purchases are final. You get 412M+ residential IPs in 195+ countries, a new IP on every request or a sticky session from 30 s to 36 h, over HTTP, HTTPS or SOCKS5.

What a few hundred thousand tweets a month cost

On ProxyField, 200,000 known posts a month cost $100 through the embed JSON over plain HTTP, 300,000 cost $150 and 500,000 cost $250. As rendered pages they cost $400, $600 and $1,000. If you find the posts by keyword or hashtag, as most sentiment projects do, that is search, and search needs the official X API. At $0.005 per post read, 200,000 posts cost $1,000, 300,000 cost $1,500 and 500,000 cost $2,500 (X's pricing page).

Read moreShow less
Posts a monthEmbed JSON, plain HTTP (volume price)Rendered post page (volume price)X API search
Cost per 1,000 posts$0.50 ($0.20)$2.00 ($1.00)$5.00
100,000$50 ($20)$200 ($100)$500
200,000$100 ($40)$400 ($200)$1,000
300,000$150 ($60)$600 ($300)$1,500
500,000$250 ($100)$1,000 ($500)$2,500
Finds posts by keywordnonoyes

What each route gives a sentiment model:

  • Embed JSON the text, the language tag (lang), the date and the author, which is all a sentiment model needs, plus like and reply counts to weight posts.
  • A rendered post page adds the repost count and the media, as X shows them logged out.
  • The X API text, created_at, lang and public_metrics for posts it finds by search. Pay-per-use is capped at 3 million post reads a month.

Scoring the sentiment of the text is a separate step on your side, with its own cost; ProxyField sells the fetching. You pay only for successful results. Billed: a page X answered with a 2xx status. Not billed: blocked requests, timeouts and errors on our side; a blocked request is retried on new IPs, and the retries are not billed. Every response says whether it was billed. A target's 404 or 410, and a sign-in page the site serves with a 200 status, count as billed results.

Do it yourself

How to scrape all tweets from one X profile with Python, including old posts

Use the X API's full-archive search with from:<handle>, in month windows. It reaches back to March 2006. The user timeline stops at the 3,200 most recent posts, and a logged-out profile page shows even fewer. If the profile is your own, the archive you download from your X account settings holds all your posts at no cost.

X's API has two routes (timelines, search and rate limits docs):

RouteEndpointHow far backPer requestRate limit per app
User posts timelineGET /2/users/:id/tweetsup to the 3,200 most recent postsup to 10010,000 requests per 15 min
Full-archive searchGET /2/tweets/search/all with from:<handle>back to March 2006, pay-per-use and Enterpriseup to 5001 request per second, 300 per 15 min
Read moreShow less

Size the job before you run it:

Posts on the accountRequests at 500 posts eachCost at $0.005 per postTime at 1 request per second
10,00020$50under a minute
50,000100$250about 2 minutes
200,000400$1,000over 15 min: the 300-per-15-min limit applies

Month windows add requests for months with few posts, so count one request per month as a floor. The script walks one profile month by month from January 2020. It follows next_token, backs off on 429 and 5xx, drops duplicates and writes one post per line to a JSONL file:

Python
import json, os, time
from datetime import datetime, timezone
import requests

TOKEN = os.environ["X_BEARER_TOKEN"]            # from the X developer console
<login> = "example"                                 # the handle, without @
URL = "https://api.x.com/2/tweets/search/all"    # full-archive search
HEADERS = {"Authorization": f"Bearer {TOKEN}"}

def months(first, last):                         # (year, month) windows, last excluded
    y, m = first
    while (y, m) < last:
        ny, nm = (y + 1, 1) if m == 12 else (y, m + 1)
        yield (datetime(y, m, 1, tzinfo=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
               datetime(ny, nm, 1, tzinfo=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"))
        y, m = ny, nm

seen = set()
with open(f"{USER}.jsonl", "a") as out:
    for start, end in months((2020, 1), (2026, 10)):
        params = {"query": f"from:{USER}", "start_time": start, "end_time": end,
                  "max_results": 500, "tweet.fields": "created_at,lang,public_metrics"}
        wait = 5
        while True:
            r = requests.get(URL, headers=HEADERS, params=params, timeout=30)
            if r.status_code == 429 or r.status_code >= 500:
                time.sleep(wait)                 # rate limit or server error: back off
                wait = min(wait * 2, 900)
                continue
            r.raise_for_status()
            wait = 5
            data = r.json()
            for post in data.get("data", []):
                if post["id"] not in seen:       # no duplicates across windows
                    seen.add(post["id"])
                    out.write(json.dumps(post) + "\n")
            token = data.get("meta", {}).get("next_token")
            if not token:
                break
            params["next_token"] = token
            time.sleep(1)
        print(start[:7], len(seen))

What it writes: one line per post, with the fields X documents for tweet.fields=created_at,lang,public_metrics (example values):

JSON
{"id": "1712345678901234567", "text": "Shipping the new release today.", "created_at": "2023-10-12T14:03:11.000Z", "lang": "en", "public_metrics": {"retweet_count": 12, "reply_count": 4, "like_count": 87, "quote_count": 1}, "edit_history_tweet_ids": ["1712345678901234567"]}

The script also prints the month and the running total, such as 2020-01 312.

Before you run it:

  1. Access

    Full-archive search is for X API pay-per-use and Enterprise customers; the bearer token comes from the X developer console.

  2. Page size

    Keep max_results at 500: the endpoint returns 10 posts per request if you leave it out (the allowed range is 10 to 500).

  3. Cost

    Each post returned is a post read at $0.005, and pay-per-use is capped at 3 million post reads a month. The script appends to the file, so a full rerun pays for the same posts again: delete the file first, or start from the last month written.

  4. What does not come back

    Deleted and protected posts are gone. Like, repost and reply counts in public_metrics are the totals when you fetch, not when the post went out.

  5. Filters

    Add -is:retweet to the query to skip reposts and -is:reply to skip replies.

Public X data you can collect

X shows logged-out visitors only part of the site: a public post opened by its own link, the embed data for that post, and a profile's name, bio and counts, often with a sign-in prompt over the rest. Search, full timelines, replies, follower lists and trends need a login, and data behind a login is out of scope.

PageURL patternWhat a logged-out visitor usually sees
Single postx.com/<user>/status/<id>Text, author, date, media, reply, repost and like counts
Profilex.com/<user>Name, handle, bio, location, link, join date, follower and following counts; some posts
Embed JSONcdn.syndication.twimg.com/tweet-result?id=<id>Text, author, date, language, like and reply counts, media
oEmbedpublish.x.com/oembed?url=<post URL>Text, author and date as embed HTML

How much X shows without a login changes often, so test your page types before you plan a volume. The API returns the page itself as HTML, Markdown or plain text, or a JSON envelope with the status code, the final URL, the headers, the cookies and the content; it has no parsed X fields, so your parser reads them.

Pricing

Pricing: from $0.20 per 1,000 X results

X results cost the Web Scraping API's prices: $0.50 per 1,000 successful results over plain HTTP, from $0.20 at volume, and $2.00 per 1,000 with JS rendering, from $1.00. Send post and profile pages on x.com with JS rendering on, because X builds them in the browser; the embed endpoints return JSON, so plain HTTP is enough for them. You pay only for successful results. The volume prices start at 10,000 successful results in one purchase. The API takes a POST to https://api.proxyfield.io/v1/scrape with your API key in an Authorization: Bearer <api_key> header and a JSON body: url, render (false for plain HTTP, true for JS rendering), country and output (html, markdown, text or json). With rendering, wait_for takes a CSS selector to wait for, so the post X builds in the browser is on the page before it comes back. Parameter reference: API documentation.

Workload (30-day month)
Workload (30-day month), Results, JS rendering at $2.00, Plain HTTP (embed JSON) at $0.50
Workload (30-day month)ResultsJS rendering at $2.00Plain HTTP (embed JSON) at $0.50
500 profiles, counts checked daily15,000$30–
10,000 known post URLs, once10,000$20$5
200,000 post URLs a month200,000$400$100
300,000 post URLs a month300,000$600$150
500,000 post URLs a month500,000$1,000$250
Shell
# The request the Web Scraping API takes (access through Contact sales)
curl https://api.proxyfield.io/v1/scrape \
  -H "Authorization: Bearer <api_key>" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://x.com/jack/status/20", "render": true, "country": "us", "output": "json"}'

Example response (shortened; content is the rendered page's HTML):

JSON
{
  "metadata": {"id": "req_2fb8e4a1b69066f7", "api": "web_scrape", "status": "success", "billed": true},
  "status_code": 200,
  "final_url": "https://x.com/jack/status/20",
  "content_type": "text/html; charset=utf-8",
  "content": "<!DOCTYPE html><html dir=\"ltr\" lang=\"en\">…</html>"
}

Contact sales through the sales team on our 24/7 live chat.

At the volume prices the 500,000 rendered posts cost $500 and the 500,000 embed results $100. Payment and balance: the same crypto balance as proxies, which never expires; there is no subscription or monthly fee. All prices: pricing.

Through the sales team on our 24/7 live chat

Compare

The X API or scraping?

X's official API is the documented route to search, timelines and follower data. It is priced per use with no subscription: $0.005 per post read and $0.010 per user lookup, capped at 3 million post reads a month on pay-per-use (X API pricing). Scraping reads only what X shows logged-out visitors: known posts and profile counts, not search or full timelines.

ItemX APIWeb Scraping API
Search and timelinesYes, full archive back to March 2006No: they need a login
Known post URLs and profile countsYesYes, as shown logged out
ApprovalX developer accountNone from X
Cost per 1,000 posts$5.00$0.50 (embed JSON, from $0.20) or $2.00 rendered (from $1.00)
Use cases

What teams use X data for

  • Sentiment analysis score the text of posts found through the X API, or of a known list of posts fetched through the embed JSON.
  • Brand and campaign monitoring reach and engagement on known posts by your brand or competitors.
  • Influencer research follower counts and bios for a list of accounts, tracked over time.
  • Research and journalism archiving public posts cited in a story or study.

Not allowed on ProxyField: spam, fraud (including fake accounts or engagement), accessing data behind logins you are not entitled to and breaking the law. Running your own X accounts through fixed IPs is a different job: X (Twitter) proxies.

FAQ

Twitter FAQ

24/7 support by live chat, including help with setup and integration.

What is the best tool to scrape Twitter?

ProxyField's Web Scraping API fetches public X posts for $0.50 per 1,000 through X's embed JSON (from $0.20), with text, date, language, likes and replies, and rendered post and profile pages for $2.00 per 1,000 (from $1.00), billed on successful results. The official X API is the tool for keyword search, full timelines and old posts by date, at $0.005 per post read. A script of your own on residential proxies, from $1.95/GB, fits teams that want to run the scraper themselves.

How much would it cost to scrape a few hundred thousand tweets a month for sentiment analysis?

If you already have the post URLs, ProxyField's Web Scraping API fetches 200,000 for $100 and 500,000 for $250 through X's embed JSON ($0.50 per 1,000, $0.20 at volume), or $400 and $1,000 as rendered pages. If you find the posts by keyword, you need the X API at $5.00 per 1,000: 200,000 posts cost $1,000 and 500,000 cost $2,500 a month. The embed JSON gives the text, language and date a sentiment model needs; scoring is a separate step with its own cost.

How do I scrape all the tweets from one X profile with Python, including posts from a few years back?

Use the X API's full-archive search, GET /2/tweets/search/all with the query from:<handle>, in month windows with start_time and end_time, following next_token and backing off on 429, as in the script on this page. The user timeline endpoint stops at the 3,200 most recent posts, and a logged-out profile page shows even fewer. Set max_results to 500 (the default is 10). A 50,000-post account is about 100 requests and $250 at $0.005 per post; the endpoint allows 1 request per second and 300 per 15 minutes per app.

What's the best tool to scrape tweets from X without paying for access to the official X API?

ProxyField's Web Scraping API fetches public X posts you have links to for $0.50 per 1,000 (from $0.20), through the JSON behind X's embedded posts, with no login and no X API key: text, date, language, author, likes and replies. 10,000 known posts cost $5, against $50 through the X API. For profiles, a headless browser on residential proxies ($2.95 for 1 GB) or rendered pages at $2.00 per 1,000 read what a logged-out visitor sees. Keyword search and full timelines have no route without the API or a login, and X's terms forbid scraping without permission.

How do I scrape Twitter?

Collect the URLs of the public posts and profiles you need, fetch posts through X's embed JSON and profiles with a headless browser or a scraping API with JS rendering, and read the text and counts. Rotate IPs, keep the rate low and stop at X's sign-in prompt rather than logging in.

What data can you scrape from Twitter?

From logged-out pages: a public post's text, author, date, language, media and reply, repost and like counts, and a profile's name, bio, location, link, join date and follower counts. Search results, full timelines, replies and follower lists need a login and are out of scope.

Does Twitter have an API for scraping?

X has an official API for reading posts, users and search, priced per use: $0.005 per post read and $0.010 per user lookup, capped at 3 million post reads a month. Its full-archive search goes back to March 2006.

How do I scrape Twitter without the API?

Use the JSON behind X's embedded posts for known post URLs, as in the script on this page, and a headless browser through rotating residential IPs for profile pages. Without the API you get only what X shows logged-out visitors.

Do I need to log in to extract data from X.com?

For search, timelines, replies and follower lists, yes, and that is out of scope: X's terms forbid scraping, and ProxyField's acceptable use policy forbids accessing data behind logins you are not entitled to. Individual public posts and basic profile data are visible logged out.

How do I scrape Twitter with Python?

For known post URLs, call https://cdn.syndication.twimg.com/tweet-result with the post ID through a proxy using requests, as in the script on this page; it returns JSON with the text and counts. For a whole profile history, use the X API's full-archive search; for profile pages, drive Playwright through the proxy.

Show 10 more questionsShow fewer questions

How do I scrape tweets from a profile?

Logged out, a profile shows only some of its posts. The X API's user timeline returns up to the 3,200 most recent, and full-archive search with from:<handle> returns older ones at $0.005 per post. For posts whose URLs you already have, fetch each through the embed JSON.

How do I scrape Twitter followers?

Follower lists need a login, so they are out of scope for scraping; the X API returns them, priced per user returned. The follower count itself is on the public profile page.

How do I scrape Twitter media?

A public post's page links to its images and videos, which you can save for analysis. Media are usually protected by copyright, so store links and metadata rather than republishing files.

How do I scrape replies to a tweet?

X shows the replies under a post mainly to signed-in users, so a logged-out scraper sees few or none. The X API's search by conversation is the route for replies.

How do I scrape old tweets?

If you have their URLs, old public posts open by link: the embed JSON returned a post from March 2006 in our test. Finding old posts by keyword or date needs the X API's full-archive search, which goes back to March 2006.

Can I scrape X search results for a keyword or hashtag?

Not logged out: X's search is behind a login, so scraping it means signing in with an account. X's terms forbid scraping without its permission, and ProxyField's acceptable use policy forbids accessing data behind logins you are not entitled to. Use the X API's search, at $0.005 per post read.

How can I scrape Twitter for free?

For a few posts, a Python script calling X's embed JSON from your own IP costs nothing. At volume you pay for IPs, an API or X's own API; ProxyField's smallest purchase is 1 GB of residential traffic for $2.95.

What is the fastest Twitter scraper?

For known posts, the embed JSON is the lightest method: it returned 920 bytes for a short post and 3,637 bytes for one with media in our test, and needs no browser. Profile pages need a headless browser and take longer. ProxyField publishes no speed figure for the Web Scraping API.

Is there a Twitter scraper on GitHub?

Yes, several, but many stopped working when X put most of the site behind a login, and many of the rest log in with accounts, which breaks X's terms. Check the last update and whether a project works logged out.

Do X's terms allow web scraping?

No: X's terms of service forbid crawling or scraping without X's prior written consent. Whether a particular scrape also breaks a law is a separate question; this is not legal advice.

Collect Twitter (X) pages with the Web Scraping API

Billed on successful results only. Contact sales for the Web Scraping API and the volume you need, or run your own scraper on residential proxies. 24/7 live chat helps with setup and integration.