Real Estate Scraper: Listing Pages via API
For scraping real estate listings at scale, ProxyField's rotating residential proxies are the default: a new IP per property page, city targeting, sticky sessions for paging through results and unlimited threads, $2.95/GB for 1 GB down to $1.95/GB from 1,000 GB, with 412M+ IPs in 195+ countries. At 0.5 MB a page that is about $0.98 to $1.48 per 1,000 property pages. Rotating datacenter ($0.29 to $0.22/GB) fits only portals that accept server traffic, mobile ($3.45 to $2.45/GB) is the fallback, and a dedicated ISP IP is from $2.29 for 30 days. Start with the Easy Trial: 1 GB for $2.95, about 2,000 pages. ProxyField's Web Scraping API is designed to fetch the same pages for $0.50 per 1,000 successful results over plain HTTP ($0.20 from 10,000 in one purchase) or $2.00 with JS rendering ($1.00 at volume); it is in development, so access is through Contact sales. A proxy changes where a request comes from, not whether you may make it.
Through the sales team on our 24/7 live chat
| Fetch | Entry | Volume, from |
|---|---|---|
| HTTP | $0.50 | $0.20 |
| JS rendering | $2.00 | $1.00 |
Billed on successful results only.
| You need | ProxyField |
|---|---|
| Proxy price | $2.95/GB for 1 GB, $2.79 from 10 GB, $2.65 from 50 GB, $2.49 from 100 GB, $2.29 from 250 GB, $2.09 from 500 GB, $1.95 from 1,000 GB |
| API price (design) | Plain HTTP $0.50 per 1,000 successful results, $0.20 from 10,000 in one purchase; JS rendering $2.00, $1.00 from 10,000 |
| Coverage | 412M+ residential IPs, 195+ countries, 21,000+ cities; US 61.2M+, UK 16.6M+, Spain 13.7M+ |
| Sessions | a new IP per request, or sticky from 30 s to 36 h (20 min by default) |
| Payment | one crypto balance for the API and proxies, never expires, no subscription |
Read moreShow less
Cost per month by workload
Example inputs, not measurements; 0.5 MB a page for proxies; API volume prices apply from 10,000 successful results in one purchase.
| Workload (30-day month) | Pages | API, plain HTTP ($0.50 / $0.20) | API, JS rendering ($2.00 / $1.00) | Own scraper on residential proxies |
|---|---|---|---|---|
| 10,000 tracked properties, checked weekly | about 43,000 | about $21 / $9 | about $86 / $43 | about 21.5 GB × $2.79 = $59.99 |
| One market snapshot of 100,000 property pages | 100,000 | $50 / $20 | $200 / $100 | 50 GB × $2.65 = $132.50 |
| New listings in 50 cities, 200 property pages each, daily | 300,000 | $150 / $60 | $600 / $300 | 150 GB × $2.49 = $373.50 |
| 1,000,000 listing pages | 1,000,000 | $500 / $200 | $2,000 / $1,000 | 500 GB × $2.09 = $1,045 |
Blocking images in your own scraper cuts page weight; measure a sample before you size a purchase.
Pilot before you commit
Run the same 1,000 pages of your own portals through each option and compare:
| Metric | How to measure it | ProxyField figure |
|---|---|---|
| Successful page loads | pages with the listing data present, out of 1,000 | a 99.6% success rate, according to our service data; measure your own portals |
| Response time | median seconds per page in your scraper's log | under 1 s typical response for residential, according to our service data |
| Cost per 1,000 pages | GB used in the dashboard × your rate, or API results billed | about $0.98 (at $1.95/GB) to $1.48 (at $2.95/GB) at 0.5 MB a page on proxies; $0.20 to $2.00 on the API design |
| Test budget | what the pilot costs | the Easy Trial: 1 GB for $2.95, about 2,000 pages at 0.5 MB, on the same pool as a 1 TB order |
What real estate scraping is
Real estate scraping is collecting public listing data from property portals automatically: asking prices, rents, sizes, locations, listing dates and status changes. Investors, agencies, lenders and proptech teams use it to see a market as a table instead of one listing at a time, and to track how it moves week by week.
Property data you can collect
| Field | Where it is | Note |
|---|---|---|
| Address or area, coordinates | Search results, property page | Some portals show only the street or district |
| Asking price or rent, price changes | Search results, property page | The price history shows cuts and relistings |
| Beds, baths, floor area, lot size, property type | Property page | Units differ by country: square feet or square metres |
| Listing date, status, days on market | Property page | For sale, under offer, sold, let |
| Sold prices | Sold or price-history pages, where the portal shows them | The inputs for comparables |
| Agent or agency name | Property page | Personal data when it names a person |
| Photos and description | Property page | Usually copyrighted: store facts, not content |
Not in scope: data behind a login, such as saved searches or broker-only MLS fields, and owners' contact details. The API is designed to return the page itself as HTML, Markdown or plain text, or a JSON envelope with the status code, the final URL, the headers, the cookies and the content; it has no parsed listing fields, so your parser reads them.
Portal terms: check them before you collect
Most large portals restrict automated collection in their terms, separately from whether it breaks a law. Read the terms of each portal you plan to collect from; ProxyField's rule for every site is public pages only, and follow the site's terms.
Read moreShow less
| Portal | What its terms say | Source |
|---|---|---|
| Rightmove (UK) | Clause 5.2 bans bots, crawlers, scrapers and other automated means to access or collect data; 5.5 allows extraction only by direct human interaction unless Rightmove's Legal team gives prior written consent; 5.6 says you "may not use the Platform for any other research purposes related to the property market"; 8.3 needs consent to copy material; 13.5 restricts loading, embedding or framing its images; under 14.2.3 Rightmove disclaims "any consequences" of the systems it puts in place "to prevent automated programs". The footer says "Rightmove prohibits the scraping of its content." | Rightmove terms of use, last updated February 2026 |
| Zoopla (UK) | Under the heading "No text or data mining, or web scraping" (an unnumbered subheading of "How you may use content on our Site"): "You shall not conduct, facilitate, authorise or permit any text or data mining or web scraping", by any robot, bot, spider or scraper; commercial use of content needs a licence; the acceptable use policy (3.1.6) forbids viewing the site with automated software without consent. | Zoopla terms, last updated November 2023 |
| idealista (Spain, Italy, Portugal) | The legal notice restricts automated copying of its content without written permission; idealista also sells data through its idealista/data service. | idealista legal notice |
| Zillow (US) | Its terms of use forbid automated collection; its data APIs go to approved partners. | Zillow scraper |
If you do not have permission
- Ask the portal for written consent or licensed access; Rightmove documents its partner APIs at api-docs.rightmove.co.uk.
- Use a licensed property-data provider whose licence covers your fields, volume, retention and redistribution.
- Use open data for the parts it covers: in England and Wales, HM Land Registry's Price Paid Data (sold prices) and the energy performance certificate (EPC) register (floor areas); HM Land Registry also offers datasets through its land and property data API.
Public listings can still hold personal data, such as an agent's name: the UK ICO and other regulators' joint statement on data scraping says that publicly available personal data is still covered by data protection law.
What's the best way to collect Rightmove or Zoopla property data in the UK with proxies?
Do not point proxies at Rightmove or Zoopla without written consent, because both portals' terms forbid scraping; use consent, a licensed property-data provider, or open data such as HM Land Registry's Price Paid Data. With consent, fetch the pages through UK residential IPs (16.6M+ in 640+ cities) at the rate the consent allows.
Read moreShow less
The clauses: Rightmove's terms (last updated February 2026) ban "bots, crawlers, scrapers or other automated programs" (5.2), allow extraction only "by means of direct human interaction" unless its Legal team consents in writing (5.5.1), forbid "technical means" that enable such access (5.5.2), reserve the right to withdraw a consent (5.7), and disclaim "any consequences" of the systems Rightmove puts in place "to prevent automated programs" (14.2.3). Zoopla's terms (last updated November 2023), under the subheading "No text or data mining, or web scraping" in "How you may use content on our Site", say "You shall not conduct, facilitate, authorise or permit any text or data mining or web scraping", and commercial use of its content needs a licence; its acceptable use policy (3.1.6) forbids viewing the site "with any automated software without our consent".
The order of preference:
-
Written consent or licensed access from the portal
Rightmove documents its partner APIs at api-docs.rightmove.co.uk.
-
A licensed property-data provider whose licence covers your fields, volume, retention and redistribution.
-
Open data for what it covers (table below): no consent needed.
-
For a personal house hunt the portals' own saved searches and email alerts, not a scraper.
| Open data (England and Wales) | What it gives | What it replaces | Access |
|---|---|---|---|
| HM Land Registry Price Paid Data | every residential sale for value: price, date, address, property type | sold-price pages on the portals | CSV downloads, updated monthly, Open Government Licence |
| Energy performance certificate data | certificates registered since 2008, including floor areas | the size field of a listing | bulk CSV (GOV.UK One Login) and an API |
| HM Land Registry land and property data API | Land Registry datasets by API | manual title and ownership lookups | API access |
Asking prices, new listings and days on market are not in open data: for those, consent or a licence is the route.
Only with written consent, the setup:
| Setting | Value |
|---|---|
| Client | one clearly identified client, as named in the consent |
| Rate | the rate limit the consent sets, and no more |
| IPs | 16.6M+ residential IPs in the United Kingdom, 640+ cities (UK proxies) |
| Line | gate.proxyfield.io:7000:<login>-country-gb-city-<city>-session-uk01-ttl-600:<password>, with <city> from the UK city list in your dashboard |
| Session | -session-<id>-ttl-600 keeps one IP for 10 minutes while you page through one search |
| Data | the facts you need (price, size, area, dates), read from the data embedded in each page; not photos or descriptions |
Which proxies work for scraping idealista listings in Spain without getting blocked?
No proxy guarantees you will not be blocked, and idealista restricts automated copying without written permission; with it, use Spanish residential IPs (13.7M+ in 690+ cities) and three rules:
-
A new IP per property page
<login>-country-es-city-madrid. -
A sticky session only while you page through one search
-session-mad01-ttl-1200holds one IP for 20 minutes. -
Datacenter IPs only on sites that accept server traffic, not on idealista.
Spanish residential IPs cost the same as every country: $2.95/GB for 1 GB, $2.79 from 10 GB, $2.65 from 50 GB, $2.49 from 100 GB, $2.29 from 250 GB, $2.09 from 500 GB and $1.95 from 1,000 GB; the 180K+ Spanish mobile IPs cost $3.45/GB down to $2.45.
Cost, for example inputs, not measurements: 100,000 property pages are $20 to $50 through the Web Scraping API design, or $132.50 through proxies (about 50 GB at 0.5 MB a page, at $2.65/GB).
Signs you are being blocked, and what to change: pages come back without listing data, as a challenge page, or slower and slower. Then lower your rate first, start a new session id (-session-mad02), widen the line from the city to the country (<login>-country-es), and only then move the job to the Spanish mobile IPs (Movistar, Orange, Vodafone and Digi networks).
| idealista market | Residential IPs | Line, new IP per page | Line, paging one search | Mobile fallback |
|---|---|---|---|---|
| Spain | 13.7M+, 690+ cities | <login>-country-es-city-madrid | <login>-country-es-city-madrid-session-mad01-ttl-1200 | 180K+ mobile IPs on Movistar, Orange, Vodafone and Digi networks |
| Italy | 10.2M+, 210+ cities | <login>-country-it | <login>-country-it-session-it01-ttl-1200 | 260K+ mobile IPs on TIM, Vodafone, WindTre and Iliad networks |
| Portugal | 300K+ | <login>-country-pt | <login>-country-pt-session-pt01-ttl-1200 | new residential IPs |
Portals by market
The Web Scraping API takes a URL, so it is designed to work on any public portal. For your own scraper, fetch each portal through residential IPs in its country, so it shows the local version.
| Market | Portals | ProxyField residential IPs |
|---|---|---|
| United States | Zillow, Redfin, Realtor.com, Trulia, Apartments.com; LoopNet and Crexi for commercial property | 61.2M+ in the United States, all 50 states, 2,800+ cities |
| United Kingdom | Rightmove, Zoopla | 16.6M+ in the United Kingdom, 640+ cities |
| Spain, Italy, Portugal | idealista | 13.7M+ in Spain, 10.2M+ in Italy, 300K+ in Portugal |
| Australia | realestate.com.au, Domain | 4.1M+ in Australia |
Zillow has its own page: Zillow scraper. The API's country parameter sets the country the request comes from (the exit IP), as a two-letter code such as de; there is no city setting. It accepts the proxy pool's 195+ countries.
Real estate scraping at scale: proxies or the API
For listing scraping at scale, use rotating residential proxies with city targeting, sticky sessions to page through results and unlimited threads, from $1.95/GB at 1,000 GB and above; or the Web Scraping API (in development) when you would rather send URLs than run IPs. From 1 TB the price is individual: contact sales.
Which proxy type, by portal:
| Proxy type | Use it for | ProxyField price |
|---|---|---|
| Rotating residential | protected portals: the default | $2.95/GB for 1 GB down to $1.95/GB from 1,000 GB |
| Rotating datacenter | portals that accept server traffic | $0.29/GB down to $0.22/GB |
| Mobile 5G/4G | a fallback where residential IPs are challenged | $3.45/GB down to $2.45/GB |
| Dedicated ISP | one long session on one fixed IP | from $2.29 per IP for 30 days |
| Need at scale | ProxyField setting |
|---|---|
| Spread the load | Rotating residential: a new IP on every request, from 412M+ residential IPs |
| Page through one search | A sticky session: 30 s to 36 h (ttl 30–129,600), 20 min by default; an IP idle for 120 s is replaced |
| Local listing sets | Country, region (all 50 US states) and city targeting, at the same price per GB; targeting stops at the city: no ZIP and no ASN targeting, so filter by ZIP or postcode in the portal's own search URL |
| Many workers | Unlimited concurrent threads on the proxies; the API is designed for 20 concurrent requests per account (higher through sales) and batches of up to 1,000 requests, results kept 48 hours |
Monthly cost by workload and the pilot test are at the top of the page.
How the Web Scraping API is designed to fetch a listing page
You send the URL of a public search or property page and choose plain HTTP or JS rendering; the API returns the page. The design: a POST to https://api.proxyfield.io/v1/scrape with your API key in an Authorization: Bearer <api_key> header and a JSON body: url, render (false for plain HTTP, true for JS rendering), country and output (html, markdown, text or json). For portals that load results with scripts, wait_for takes a CSS selector to wait for, with rendering. Parameter reference: API documentation.
| Choice | Price per 1,000 successful results | Use it for |
|---|---|---|
| Plain HTTP | $0.50, or $0.20 from 10,000 in one purchase | Portals that embed the listing data in the HTML |
| JS rendering | $2.00, or $1.00 from 10,000 in one purchase | Map searches and portals that load results with scripts |
# The request as designed (in development; access through Contact sales)
curl https://api.proxyfield.io/v1/scrape \
-H "Authorization: Bearer <api_key>" \
-H "Content-Type: application/json" \
-d '{"url": "https://www.realtor.com/realestateandhomes-search/Denver_CO", "render": false, "country": "us", "output": "json"}'Example response in the API's design (shortened; content is the page's HTML, with the listing JSON inside):
{
"metadata": {"id": "req_99c17f138edd3ee9", "api": "web_scrape", "status": "success", "billed": true},
"status_code": 200,
"final_url": "https://www.realtor.com/realestateandhomes-search/Denver_CO",
"content_type": "text/html; charset=utf-8",
"content": "<!DOCTYPE html><html lang=\"en\">…<script id=\"__NEXT_DATA__\" type=\"application/json\">…</html>"
}Pricing: $0.50 per 1,000 listing pages, from $0.20
Every search or property page is one result at the Web Scraping API's price: $0.50 per 1,000 successful results over plain HTTP, or $0.20 from 10,000 results in one purchase, and $2.00 per 1,000 with JS rendering, or $1.00 from 10,000. You pay only for successful results.
Billed: a page the portal answered with a 2xx status; a target's 404 or 410, and a sign-in page the site serves with a 200 status, also count as billed results. Not billed: error responses, blocked and timed-out requests, and our internal retries on new IPs. Every response says whether it was billed. Payment: the same crypto balance as proxies, which never expires; there is no subscription or monthly fee. All prices: pricing.
Through the sales team on our 24/7 live chat.
| Rate | Entry | Volume, from |
|---|---|---|
| HTTP | $0.50 | $0.20 |
| JS rendering | $2.00 | $1.00 |
Billed on successful results only.
Official data feeds or scraping?
Portals and listing services share data officially only with partners: MLS listing feeds go to member brokers and their vendors, and portal APIs, where they exist, need an application and approval. Some portals publish market-level data for download, such as median prices by city. Scraping reads the public pages any visitor sees, within each portal's terms (above). MLS data has its own page: MLS scraper.
What teams use real estate data for
- Investment screening price per square foot or square metre, rental yield and days on market by area.
- Comparables and valuation recent sold or asking prices near a property.
- Market reports inventory, new listings and price cuts by city each week.
- Agency intelligence which agencies list what, where and at what price.
- Rental analysis rents by size and area, and how fast units let.
Scrape listings with Python and ProxyField proxies
Many portals embed their listing data as JSON in the page for their own front end, as application/ld+json blocks or a framework's data script. Fetch the page through residential IPs in the portal's country and read that JSON instead of the visible layout.
import json
import requests
from bs4 import BeautifulSoup
proxy = "http://<login>-country-us-region-colorado-city-<city>:<password>@gate.proxyfield.io:7000"
url = "https://www.realtor.com/realestateandhomes-search/Denver_CO" # a public search page
resp = requests.get(url, proxies={"http": proxy, "https": proxy},
headers={"User-Agent": "Mozilla/5.0", "Accept-Language": "en-US"},
timeout=30)
if resp.status_code != 200:
print("Status", resp.status_code, "- slow down and retry later")
else:
soup = BeautifulSoup(resp.text, "html.parser")
blocks = soup.select('script[type="application/ld+json"], script#__NEXT_DATA__')
for tag in blocks:
data = json.loads(tag.string or "{}")
print(tag.get("id") or "ld+json", str(data)[:200]) # find the listing objectsProxyField residential proxies cost $2.95/GB for 1 GB and from $1.95/GB at volume, with no subscription; the traffic never expires, and all purchases are final. You get a new IP on every request or a sticky session from 30 s to 36 h, over HTTP, HTTPS or SOCKS5. For map searches that need a browser, the same settings work in Playwright.
Portal terms and the law
Beyond each portal's terms (above), prices, sizes and dates are facts, while photos and descriptions belong to agents or portals and are usually protected by copyright. In the EU and UK a portal's listings can be a database protected by the database right against extracting a substantial part, and agents' and owners' names are personal data under the GDPR and the UK GDPR. More: Is It Legal to Scrape eBay, Walmart, Zillow and Other Marketplaces? and Is Web Scraping Legal? US, EU and UK Rules (2026).
ProxyField's acceptable use policy allows collecting public web data and market research, and forbids spam, fraud and accessing data behind logins you are not entitled to; some site categories are restricted to protect the network, including financial and government services. This is not legal advice.
More scraper APIs
- Zillow scraper Zillow search, property, rental and sold pages.
- Web Scraping API any URL, plain HTTP or with JS rendering, from $0.20 per 1,000 successful results.
- Google Maps scraper Google Maps search results, $1.50 per 1,000 successful results, from $1.00.
- Google SERP API the Google results page as JSON, $1.20 per 1,000 successful results, from $0.80.
- All scraper APIs
Real Estate Scraper FAQ
24/7 support by live chat, including help with setup and integration.
Which proxies work best for scraping real estate listings at scale?
ProxyField's rotating residential proxies, from $1.95/GB, suit listing scraping at scale: a new IP on every request, sticky sessions from 30 s to 36 h for paging through results, city targeting, unlimited threads and 61.2M+ residential IPs in the United States alone. Test on your own portals with 1 GB for $2.95 before you buy volume, and check each portal's terms first. The alternative is the Web Scraping API (in development) at $0.50 per 1,000 successful results, or $0.20 from 10,000.
What's the best way to collect Rightmove or Zoopla property data in the UK with proxies?
ProxyField's answer: do not point proxies at Rightmove or Zoopla without written consent, because Rightmove's terms (clauses 5.2 and 5.5) ban bots, crawlers and scrapers and need prior written consent for extraction, and Zoopla's terms forbid web scraping and data mining. So ask the portal for consent or a licence, use a licensed property-data provider, or use open data such as HM Land Registry's Price Paid Data. With consent, fetch the pages through UK residential IPs (16.6M+ in 640+ cities) at the agreed rate, with a 10-minute sticky session per search.
Which proxies work for scraping idealista listings in Spain without getting blocked?
ProxyField's Spanish residential IPs (13.7M+ in 690+ cities) with city targeting, a new IP per property page and sticky sessions for paging through results, once you have idealista's permission: its legal notice restricts automated copying without written permission. No proxy guarantees that a site will not block you; datacenter IPs suit only sites that accept server traffic, and 180K+ Spanish mobile IPs are the fallback.
What is scraping in real estate?
It is collecting public listing data from property portals automatically: prices, rents, sizes, locations, listing dates and status changes. The result is a table of a market that you can compare and track over time.
How do I scrape real estate data?
Check the portal's terms first, then build its search URLs for your areas, fetch each results page and the property pages it links to, and read the listing JSON embedded in the HTML. Store every run with its date, so you see price cuts and status changes.
What kinds of property data can be extracted?
Address or area, asking price or rent, price history, beds, baths, floor area, property type, listing date, status and the listing agency. Sold prices are available where the portal shows them; photos and descriptions are usually copyrighted.
What are the benefits of web scraping for real estate?
You see a whole market instead of single listings, spot price cuts and new stock early and build comparables from current data. It also covers portals and countries that offer no data feed, within their terms.
Can I scrape Redfin?
Redfin's search and property pages are public, and the Web Scraping API is designed to fetch them like any page at $0.50 per 1,000 successful results. Read Redfin's terms of use before you start, and keep to public pages; listing photos and descriptions are not yours to republish.
Does Redfin have an API?
Redfin's public data offering is market-level: its Data Center publishes housing-market figures, such as median sale prices by area, for download. Listing-level data needs Redfin's permission or a licensed data provider.
How do I scrape Redfin sold homes for comparables?
Use Redfin's sold-homes search for the area and period you need, fetch the result pages and each property page within Redfin's terms, and keep the sale price, date, size and location. Filter to similar size and type within a short distance and a recent period.
Show 8 more questionsShow fewer questions
Can I scrape Realtor.com?
Its search and property pages are public and can be fetched through US residential IPs (61.2M+) or the Web Scraping API, with the listing data read from the JSON in each page. Realtor.com's terms apply: read them before you start.
Can I scrape Trulia?
Trulia is part of Zillow Group, and its terms forbid automated collection, the same as Zillow's. Ask for permission or use a licensed data provider; the Zillow scraper page covers the Zillow Group options.
Can I scrape LoopNet or Crexi for commercial property listings?
Their public listing pages can be fetched like any portal's, but some details on both sites sit behind a login, which is out of scope. Collect only what a logged-out visitor sees and check each site's terms first.
Can I scrape rental listings from Apartments.com?
Its public search and property pages show rents, unit sizes, availability and building details to any visitor; check Apartments.com's terms before you collect them. Technically they can be fetched through US residential IPs, or through the Web Scraping API at $0.50 per 1,000 successful results.
Can I scrape realestate.com.au?
Read its terms first; the portal is built for local visitors, so any permitted collection goes through Australian residential IPs. ProxyField has 4.1M+ residential IPs in Australia.
How do I process and store scraped real estate data for analysis?
Normalise the address, price and size fields, convert units, give each property a stable key such as the portal's listing ID, and store one row per property per run in a database. Then you can query price changes, days on market and medians by area.
Can I automate recurring real estate scraping jobs?
Yes: run the scraper from a scheduler such as cron, daily for new listings and weekly for tracked properties, and alert when a run returns far fewer results than usual. The Web Scraping API is designed for batches of up to 1,000 requests and bills only successful results.
Do you offer real estate data scraping services?
ProxyField sells the tools and you run the project: residential proxies from $1.95/GB for your own scraper today, or the Web Scraping API (in development, Contact sales) at $0.50 per 1,000 successful results. The 24/7 live chat helps with setup and integration; with 1 TB+ of proxy traffic, the price is individual through sales.
Related pages
Start collecting public pages with the Web Scraping API
Billed on successful results only. Contact sales for the Web Scraping API and the volume you need, or run your own scraper on residential proxies. 24/7 live chat helps with setup and integration.