News Scraper: Public News Pages via API
ProxyField's Web Scraping API is the news scraper API for your own list of publishers: it fetches any public article page, RSS feed or news sitemap for $0.50 per 1,000 successful results over plain HTTP, that is $0.0005 a page, from $0.20 per 1,000 at volume, so 1M pages cost $500, or $200 at the volume price. It sells the fetch step, not a pre-parsed news archive, which wins when your list includes niche or local outlets that a vendor's source index leaves out. Contact sales to start, or run your own news scraper on residential proxies with the Easy Trial: 1 GB for $2.95.
Through the sales team on our 24/7 live chat
| Fetch | Entry | Volume, from |
|---|---|---|
| HTTP | $0.50 | $0.20 |
| JS rendering | $2.00 | $1.00 |
Billed on successful results only.
Best news scraper API for thousands of news sites: pick by what you need
| You need | Choose | Why |
|---|---|---|
| Articles from your own list of thousands of publishers, including niche and local outlets | ProxyField Web Scraping API | any public URL, full page, $0.50 per 1,000 successful results, from $0.20; one generic parser for most sites |
| A searchable archive of articles already parsed, with tags or sentiment, from a fixed source list | a licensed news API | the vendor collects and parses; you get only the sources it covers, under its licence |
| Keyword alerts on press coverage across many outlets | a news or media-monitoring API first, the Web Scraping API for the outlets it misses | see the press-monitoring checklist below |
| Full text and dates for article URLs you already have | the pipeline below | one parser, about 60 lines of Python, on any fetch route |
Read moreShow less
At thousands of sites the cost is a function of pages, not sites: one page is one result, $0.0005 at the entry price and $0.0002 at volume. What you get back is the full public page for your parser; what you do not get is a vendor's archive, sentiment or entity tags. For a country's edition of a site, your own scraper on ProxyField residential proxies exits in 195+ countries, with targeting by country, region (state) and city, across 412M+ residential IPs; the API's country parameter sets the country the request comes from (the exit IP), as a two-letter code such as de; the countries it accepts: the proxy pool's 195+ countries. To find article URLs across thousands of sites before you fetch them, two open sources help: the GDELT DOC 2.0 API searches a rolling 3 months of the news GDELT monitors, in 65 machine-translated languages, and returns up to 250 articles per query as JSON or RSS (GDELT); Common Crawl's CC-NEWS publishes news articles from sites worldwide as daily WARC files (CC-NEWS).
News API, media-monitoring API or news scraper API: are they the same?
No: a news API sells articles its vendor has already collected and parsed, and a news scraping API fetches the public pages you send. The news API is searchable by keyword, source and date; the scraping API leaves the parsing to you. For press and brand monitoring across many outlets, start with a news or media-monitoring API, and scrape the outlets it does not cover.
| Item | Licensed news API | Publisher RSS feed | Web Scraping API |
|---|---|---|---|
| Sources | The vendor's list | One publisher per feed | Any public URL you send |
| Content | Parsed articles, per licence | Title, link, date, often only a summary | The full public page, for your parser |
| Who parses | The vendor | Your feed reader | Your code: one generic parser for most sites |
| Cost | The vendor's plan | None | $0.50 per 1,000 successful results, from $0.20 |
| Best for | Brand and press monitoring across many outlets, keyword search | A few named publishers | Your own list of publishers, niche and local outlets, fields the others leave out |
Press-monitoring checklist: which side meets each need
| Need | News or media-monitoring API | Web Scraping API |
|---|---|---|
| Keyword search over many outlets at once | yes | no: you fetch the outlets you list |
| Alerts on new mentions | usually built in | your code, run on each new article |
| Deduplication across syndicated copies | usually built in | canonical URL as the key (pipeline step 2) |
| Outlets outside the vendor's list | no | yes, any public URL |
| The full public page, for your own fields | depends on the licence | yes |
| Cost of 200 outlets polled every hour | the vendor's plan | 144,000 requests a month, $72 at $0.50 per 1,000 |
The $72 covers the feed polls; the new articles you fetch from them are extra pages at the same rate.
Collect full text and dates from many news sites: the pipeline
The recipe in one sentence: find article URLs in feeds or news sitemaps, deduplicate them by canonical URL, and run one generic parser that takes the date from datePublished, then article:published_time, then the feed's pubDate. You write that parser once, not a scraper per site, because most news pages carry the same NewsArticle structured data; the Python script on this page is that parser, about 60 lines.
Read moreShow less
-
Discover
Poll each site's RSS or Atom feed (linked in the page head as
application/rss+xml) or its news sitemap (named inrobots.txt); add GDELT or CC-NEWS for sites you have no feed for. -
Deduplicate
Key every article by its canonical URL (
<link rel="canonical">), so a story in three feeds is fetched once. -
Fetch
Send the URL to the Web Scraping API, or fetch it yourself through rotating residential IPs.
-
Read the date
Take
datePublishedfrom the page'sNewsArticleJSON-LD; if it is missing, thearticle:published_timemeta tag; then the feed'spubDate. StoredateModifiedseparately. -
Read the text
Take the paragraphs inside
<article>; where a page has none, run an open-source extractor such as trafilatura (Python) or Mozilla Readability (JavaScript) on the HTML. -
Store the text, URL, publisher, author and which source supplied the date, so you can audit dates later.
The Python script below returns one record per article:
{"url": "https://news.example.com/2026/10/22/some-article",
"headline": "Example headline",
"published": "2026-10-22T07:30:00Z",
"date_source": "json-ld",
"author": "Jane Doe",
"text": "The first paragraph of the article ..."}| Field | First source | Fallback |
|---|---|---|
url | <link rel="canonical"> | the URL you fetched |
headline | JSON-LD headline | og:title meta tag |
published | JSON-LD datePublished | article:published_time meta tag, then the feed's pubDate |
date_source | which of the three gave the date: json-ld, meta or feed | – |
author | JSON-LD author (or its name) | – |
text | the <p> paragraphs inside <article> | trafilatura on the whole HTML |
What it costs: 200 publishers at 30 new articles a day is 180,000 pages a month, $90 at $0.50 per 1,000 successful results ($36 at $0.20). Where it stops: paywalled text, sign-in walls, pages without an <article> element (trafilatura or Mozilla Readability fills in) and pages that hide the date entirely. A hosted article-extraction API can replace step 5, at its own per-call price on top of each fetch; with the parser on this page, the fetch at $0.50 per 1,000 is the whole per-page cost. ProxyField supplies the fetch step for any extractor. The API's own output: it returns the page itself as HTML, Markdown or plain text, or a JSON envelope with the status code, the final URL, the headers, the cookies and the content; it has no parsed news fields, so your parser reads them.
Pricing: $0.50 per 1,000 pages, from $0.20
Feeds, sitemaps and most article pages are plain HTML, so plain HTTP at $0.50 per 1,000 successful results covers most news jobs; JS rendering costs $2.00 per 1,000, from $1.00 at volume, for the sites that build the article with scripts. You pay only for successful results. Billed: a page the site answered with a 2xx status. Not billed: blocked requests, timeouts and errors on our side; a blocked request is retried on new IPs, and the retries are not billed. Every response says whether it was billed. A target's 404 or 410, and a sign-in page the site serves with a 200 status, count as billed results. For availability and access, contact sales through the sales team on our 24/7 live chat. The volume price starts at 10,000 successful results in one purchase.
One result is one page fetched: a feed, a sitemap or an article. Polling a feed every 2 hours instead of every hour halves its line. Payment and balance for the API: the same crypto balance as proxies, which never expires; there is no subscription or monthly fee. All prices: pricing.
Through the sales team on our 24/7 live chat
| Workload (30-day month, plain HTTP unless noted) | Pages a month | At $0.50 per 1,000 | At $0.20 per 1,000 |
|---|---|---|---|
| 200 publishers, 30 new articles each a day | 180,000 | $90 | $36 |
| 1,000 publishers, 30 new articles each a day | 900,000 | $450 | $180 |
| The same 1,000 feeds checked every hour | 720,000 | $360 | $144 |
| 5,000 publishers, 10 new articles each a day | 1,500,000 | $750 | $300 |
| 20 script-built sites, 1,000 articles a day (JS rendering, $2.00 → $1.00) | 30,000 | $60 | $30 |
What a news scraper collects
The Web Scraping API takes any URL and returns the page; your code pulls the fields out of it. ProxyField sells no separate news API with pre-parsed article fields.
| News page | What the public page shows |
|---|---|
| Article page | Headline, byline, publish and update dates, body text, summary, section, tags, main image, canonical URL |
| Section front or topic page | Article headlines and links, teasers, timestamps |
| RSS or Atom feed | Title, link, publish date and summary of the latest articles |
| News sitemap | Article URLs with publish dates and titles |
Paywalled text and anything behind a login are out of scope: ProxyField's acceptable use policy forbids accessing data behind logins you are not entitled to.
What you send
You send an article, feed or sitemap URL to the Web Scraping API and get the page back. The API takes a POST to https://api.proxyfield.io/v1/scrape with your API key in an Authorization: Bearer <api_key> header and a JSON body: url, render (false for plain HTTP, true for JS rendering), country and output (html, markdown, text or json). For a site that builds the article with scripts, wait_for takes a CSS selector to wait for, with rendering. country sets the country the request comes from (the exit IP), as a two-letter code such as de; the countries it accepts: the proxy pool's 195+ countries. Parameter reference: API documentation.
An article URL you would send looks like https://news.example.com/2026/10/22/some-article; a feed (/rss.xml) or a news sitemap is sent the same way.
# The request the Web Scraping API takes (access through Contact sales)
curl https://api.proxyfield.io/v1/scrape \
-H "Authorization: Bearer <api_key>" \
-H "Content-Type: application/json" \
-d '{"url": "https://news.example.com/2026/10/22/some-article", "render": false, "country": "us", "output": "json"}'Example response (shortened; content is the article page's HTML):
{
"metadata": {"id": "req_cbfe3e55db65f8d3", "api": "web_scrape", "status": "success", "billed": true},
"status_code": 200,
"final_url": "https://news.example.com/2026/10/22/some-article",
"content_type": "text/html; charset=utf-8",
"content": "<!DOCTYPE html><html lang=\"en\"><head>…<script type=\"application/ld+json\">…</html>"
}How to scrape news articles with Python
This script reads an RSS feed, skips URLs it has seen, fetches each article through rotating residential IPs and builds the record above: headline and date from the JSON-LD, then the meta tag, then the feed, with trafilatura as the fallback for the text. ProxyField gives you a new IP on every request or a sticky session from 30 s to 36 h, with 412M+ residential IPs in 195+ countries and targeting by country, region (state) and city.
Read moreShow less
import json, time
import xml.etree.ElementTree as ET
import requests
import trafilatura
from bs4 import BeautifulSoup
proxy = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
s = requests.Session()
s.proxies = {"http": proxy, "https": proxy}
s.headers["User-Agent"] = "news-research/1.0 (contact: [email protected])"
seen = set()
def ld_article(soup):
for tag in soup.find_all("script", type="application/ld+json"):
try:
data = json.loads(tag.string or "")
except ValueError:
continue
items = data if isinstance(data, list) else data.get("@graph", [data])
for item in items:
kind = item.get("@type") if isinstance(item, dict) else None
kinds = kind if isinstance(kind, list) else [kind]
if {"NewsArticle", "Article", "ReportageNewsArticle"} & set(kinds):
return item
return {}
def article(url, feed_date):
html = s.get(url, timeout=30).text
soup = BeautifulSoup(html, "html.parser")
ld = ld_article(soup)
meta = lambda p: (soup.find("meta", property=p) or {}).get("content")
canonical = (soup.find("link", rel="canonical") or {}).get("href") or url
date, source = None, None
for value, where in ((ld.get("datePublished"), "json-ld"),
(meta("article:published_time"), "meta"),
(feed_date, "feed")):
if value:
date, source = value, where
break
author = ld.get("author")
if isinstance(author, dict):
author = author.get("name")
body = soup.find("article")
text = " ".join(p.get_text(" ", strip=True) for p in body.find_all("p")) if body else ""
return {"url": canonical,
"headline": ld.get("headline") or meta("og:title"),
"published": date, "date_source": source, "author": author,
"text": text or trafilatura.extract(html)}
feed = ET.fromstring(s.get("https://news.example.com/rss.xml", timeout=30).content)
for item in feed.iter("item"):
link = item.findtext("link")
if link in seen:
continue
row = article(link, item.findtext("pubDate"))
seen.update({link, row["url"]})
print(json.dumps(row)[:300])
time.sleep(3)Install the libraries with pip install requests beautifulsoup4 trafilatura. A feed's pubDate is in RFC 822 form (Wed, 22 Oct 2026 07:30:00 GMT), not ISO 8601, so normalise dates before you store them. Access opens once the payment is confirmed; the gateway, credentials or allowlist are in your account. The username format is <login>-country-us-city-new_york; other tokens: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default) after the location, as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600.
For a site that builds the article with JavaScript, drive a browser with Playwright on the same gateway, or send the URL to the Web Scraping API with JS rendering. Residential proxies cost $2.95/GB for 1 GB and from $1.95/GB at volume; the traffic never expires, and all purchases are final.
When a news site refuses a request
Large publishers rate-limit automated traffic with 403 or 429, and some show a regional edition or a paywall instead of the article. With the Web Scraping API you pay only for successful results. With your own scraper, slow down, spread requests over rotating IPs, send an honest User-Agent and read the feed rather than recrawling section fronts. A paywall is not a block to work around: if you need that text, license it. More on 403 errors: Cloudflare 403 Forbidden.
What teams use news data for
- Media monitoring and PR mentions of a brand, a person or a competitor across outlets.
- Market and financial signals company news, deals and regulatory updates as they are published.
- NLP and AI datasets news corpora for classification, summarization and topic tracking, kept for internal analysis.
- Research and fact-checking archives of what was published, when, and how a story changed.
Legal questions
Each publisher's terms of use set the rules for its site, and many restrict automated collection or reuse of articles; check them and the site's robots.txt (RFC 9309) first. Headlines, links, dates and facts are commonly collected for analysis, while article text is protected by copyright, so keep full text for internal use and quote short extracts with attribution unless you have a licence. In the EU, press publishers also hold a related right over online use of their publications (Article 15 of the Copyright in the Digital Single Market Directive). How to read a site's rules: how to check if a website allows scraping. For the wider question, read Is Web Scraping Legal? US, EU and UK Rules (2026). This is not legal advice.
More scrapers
- Google News scraper for the Google News results themselves.
- Google SERP API: Google results, including the news stories that rank, $1.20 per 1,000 successful results, from $0.80.
- Reddit scraper and social media scraping on the same Web Scraping API.
- ChatGPT scraper API: the ChatGPT answer, $1.50 per 1,000 successful results, from $1.00.
- All scrapers: scrapers.
News Scraper FAQ
24/7 support by live chat, including help with setup and integration.
What's the best news scraper API for collecting articles from thousands of news sites?
ProxyField's Web Scraping API fetches any public URL, so it reads article pages, feeds and sitemaps from thousands of news sites with no source list, at $0.50 per 1,000 successful results over plain HTTP ($0.0005 a page), from $0.20 at volume: 5,000 publishers at 10 articles a day is 1.5M pages, $750 a month or $300 at the volume price. It suits a list of publishers you choose, with one generic parser that reads each article's NewsArticle JSON-LD. For ready-parsed articles from a vendor's fixed source list, a licensed news API is the other route.
How can I collect full article text and publish dates from many news sites without writing a scraper for each one?
Run one pipeline with one generic parser: find articles through RSS feeds or news sitemaps, deduplicate by canonical URL, fetch each page, take the date from the NewsArticle JSON-LD (then the article:published_time tag, then the feed date) and the text from <article> or an extractor such as trafilatura. Store which source gave the date; the parser on this page is about 60 lines of Python. Fetching 180,000 pages a month through ProxyField's Web Scraping API costs $90 at $0.50 per 1,000 successful results.
Is a news scraping API the same thing as a news API, and which one suits monitoring press coverage?
No. A news API sells articles its vendor has already collected and parsed, searchable by keyword, source and date; a news scraping API fetches the public pages you choose and leaves the parsing to you. For monitoring press coverage, start with a news or media-monitoring API and scrape the outlets it does not cover, such as niche or local sites: polling 200 of them every hour through ProxyField's Web Scraping API is 144,000 requests a month, $72 at $0.50 per 1,000.
How much does a news scraper cost?
ProxyField's Web Scraping API fetches public news pages for $0.50 per 1,000 successful results over plain HTTP, from $0.20 at volume, and $2.00 per 1,000 with JS rendering, from $1.00. 1M pages cost $500, or $200 at the volume price; reading 30 new articles a day from each of 200 publishers is 180,000 pages a month, $90 at the entry price. Your own scraper on residential proxies starts at 1 GB for $2.95.
How many news sites can a news scraper API cover?
ProxyField's Web Scraping API has no source list: it fetches any public URL you send, so it covers as many publishers as you list, and the cost depends on pages, not sites. 1,000 publishers at 30 new articles a day is 900,000 pages a month, $450 at $0.50 per 1,000 successful results or $180 at the volume price.
How much does it cost to monitor press coverage with a scraping API?
It depends on how many outlets you poll and how often. Polling 200 outlets' feeds every hour through ProxyField's Web Scraping API is 144,000 requests a month, $72 at $0.50 per 1,000 successful results, plus the new articles you fetch. A news or media-monitoring API is usually the first tool for keyword alerts; the scraping API covers the outlets it misses.
What is a news scraper?
A tool or script that turns public news pages into data: headline, byline, publish date, body text, section and links. It usually pairs with a crawler that finds new article URLs from feeds, sitemaps and section fronts.
How does web scraping news sites work?
A crawler collects new article URLs, a fetcher downloads each page, and a parser extracts the fields and stores them with the canonical URL as the key. Most news pages are plain server-rendered HTML, so the fetch step rarely needs a browser.
How do I scrape news articles?
Take the article URLs from a feed or sitemap, fetch each page a few seconds apart, and read the headline, date and author from its JSON-LD, then the text from the article body. Send the URLs to the Web Scraping API, or run the Python on this page through rotating residential proxies.
How do I scrape news articles with Python?
Read the feed with xml.etree, fetch each article with requests through a proxy and parse it with Beautiful Soup, as in the script on this page. It returns url, headline, published, date_source, author and text, with trafilatura as the fallback for the body text.
Show 15 more questionsShow fewer questions
What is the difference between a news crawler and a news scraper?
A news crawler discovers pages: it follows feeds, sitemaps and section links to find article URLs. A news scraper extracts the fields from each page it is given; most news pipelines run both.
Is there a news scraper API?
ProxyField has no separate news API with parsed article fields; its Web Scraping API fetches any news URL and returns the page for your parser, at $0.50 per 1,000 successful results over plain HTTP. Many article pages carry NewsArticle JSON-LD, so one generic parser reads most publishers.
Is there a free news scraper API?
Publisher RSS feeds can be read at no cost, and the GDELT DOC 2.0 API lists articles from the news GDELT monitors. ProxyField's Web Scraping API is billed per 1,000 successful results; the smallest proxy purchase is 1 GB of residential traffic for $2.95, with no subscription.
What data can a news scraper extract?
From public pages: headline, byline, publish and update dates, body text, summary, section, tags, main image and canonical URL. Feeds and news sitemaps add the article list with publish times.
Where does a news scraper get the publish date?
From the datePublished field of the article's NewsArticle JSON-LD first, then the article:published_time meta tag, then the date in the RSS feed or news sitemap. Keep dateModified apart and record which source supplied the date.
How does a news crawler find new articles?
It polls RSS or Atom feeds, which most publishers offer per section, and news sitemaps, which list recent articles with their publish times. The sitemap is usually named in the site's robots.txt; section fronts fill the gaps.
Can I scrape paywalled news articles?
Not text you are not entitled to: ProxyField's acceptable use policy forbids accessing data behind logins you are not entitled to. The headline, summary and metadata a paywalled page shows publicly can be read; for the full text, use your own subscription within its terms or license the content.
Are proxies alone enough to scrape news sites?
For most news sites, yes: the pages are plain HTML, and rotating residential IPs with sensible pacing spread the load. Sites that build articles with JavaScript also need a browser or JS rendering.
How do I handle news websites that block my crawler?
Slow down, read the feed or sitemap instead of recrawling section fronts, send an honest User-Agent and spread requests over rotating IPs. If a publisher keeps refusing, ask it for a feed or a licence.
Do news scrapers need JavaScript rendering?
Usually not, because most news articles are server-rendered HTML that plain HTTP reads. For the sites that load the article with scripts, the Web Scraping API's JS rendering costs $2.00 per 1,000 successful results, from $1.00 at volume, or you drive Playwright on the same proxies.
Can I scrape news from a specific country or edition?
Yes with your own scraper: ProxyField residential proxies cover 195+ countries with targeting by country, region (state) and city, so you see the edition a reader there gets. Many publishers also serve each edition at its own URL, which you can send to the Web Scraping API like any other page.
How do I extract clean article text from a news page?
Take the paragraphs inside the page's <article> element, or run an open-source extractor such as trafilatura, which strips menus, ads and related links.
What tools are commonly used for news scraping?
Python with requests and Beautiful Soup, feed parsers for RSS, Scrapy for large crawls, trafilatura or newspaper-style libraries for the text, and Playwright for script-built pages.
How often should a news scraper check for new articles?
Match the site's pace: every 10 to 60 minutes for a busy outlet's feed, a few times a day for a small one. Checking 1,000 feeds every hour is 720,000 requests a month, $360 at $0.50 per 1,000, so poll quiet feeds less often.
Is copying full news articles allowed?
Republishing full articles generally needs the publisher's licence, because the text is copyrighted. Headlines, links, dates and facts are commonly collected for analysis, and short quotes with attribution are a different case; this is not legal advice.
Related pages
Start collecting public pages with the Web Scraping API
Billed on successful results only. Contact sales for the Web Scraping API and the volume you need, or run your own scraper on residential proxies. 24/7 live chat helps with setup and integration.