Lead Scraper API: Public B2B Company Pages
ProxyField's lead scraper works today as the Python script on this page, run through ProxyField proxies: it visits each company's home, contact and about pages and writes one record per company with name, city, country, business emails, phones, social links and source URLs, as JSON lines and a CSV file. At an example 100 KB a page, 1,000 companies are about 0.3 GB: $0.90 of residential traffic at $2.95/GB, or $0.09 on rotating datacenter at $0.29/GB, and the Easy Trial (1 GB for $2.95) covers about 3,000 companies. The Web Scraping API (in development, Contact sales) is designed to fetch the same pages for you at $0.50 per 1,000 successful results, $0.20 from 10,000 in one purchase: about $1.53 per 1,000 companies, or $0.61. Neither sells named contacts or verified emails; those come from a B2B lead database.
Contact sales for the API
| Fetch | Entry | Volume, from |
|---|---|---|
| HTTP | $0.50 | $0.20 |
| JS rendering | $2.00 | $1.00 |
Billed on successful results only.
Lead scraping is building a prospect list from public web pages: business directories, company websites, trade associations, public registers such as Companies House and SEC EDGAR, maps, review sites, job posts and event pages. It is one way to build the list inside lead generation, which also covers referrals, ads, events and inbound forms.
Which lead scraper fits your job?
Pick by job: the script on proxies if you want company data today from sources you choose, the API if you want the same pages without running proxies, and a B2B lead database if you want named contacts.
| Your job | Pick | Cost per 1,000 companies | What you get back | What it will not do |
|---|---|---|---|---|
| A few thousand companies from directories and websites you choose, starting today | The Python script below on ProxyField proxies | About $0.90 on residential ($2.95/GB), $0.09 on rotating datacenter ($0.29/GB), at an example 100 KB a page | One record per company: name, city, country, business emails, phones, social links, source URLs; JSON lines and CSV | Named people, private emails, verified emails |
| The same pages without running proxies or handling blocks | ProxyField Web Scraping API (in development, Contact sales) | About $1.53, or $0.61 from 10,000 results in one purchase; JS rendering $2.00 per 1,000 pages | The page as HTML, Markdown, text or a JSON envelope; the same parser makes it a record | Parsed contact fields: your parser reads them |
| Named contacts with emails, ready to load into a CRM | A B2B lead database | Per contact or per credit, usually on a subscription | Records from the vendor's own data | Sources you choose; the data is as fresh as the vendor's last update |
Read moreShow less
When you compare lead scrapers, check 3 things: which sources they reach, what counts as a billable result, and where their contact data comes from. Every route still needs deduplication, email verification and a lawful basis for outreach.
What you get back: company and contact fields
Worked example: 1,000 target companies.
| Step | Pages | At $0.50 per 1,000 |
|---|---|---|
| Collect: directory results pages listing 20 companies each | 50 | $0.025 |
| Enrich: home, contact and about page of each company's website | 3,000 | $1.50 |
| Total for 1,000 companies | 3,050 | about $1.53 |
Scale it to 10,000 companies and the 30,500 pages are one purchase above 10,000 results, so they cost $0.20 per 1,000: $6.10, or $0.61 per 1,000 companies. The Python code below turns each company into one record like this (example values):
{"domain": "example.com", "name": "Example Pumps Ltd", "city": "Leeds", "country": "GB", "emails": ["[email protected]"], "phones": ["+44 113 496 0000"], "social": ["https://www.linkedin.com/company/example-pumps"], "sources": ["https://example.com/", "https://example.com/contact"]}| Field | Where it is on the page | Note |
|---|---|---|
| Company name, website, category | Directory listing, the site's title, schema.org Organization or LocalBusiness JSON-LD | The core of every record |
| Address, city, country | Directory listing, the contact page, the JSON-LD address | Lets you filter by territory |
| Business phone | Directory listing, tel: links, the JSON-LD telephone | As the company publishes it |
| Business email | mailto: links on the contact or imprint page | Often a role address such as sales@ |
| Size range, industry | About pages, association lists, registers | Ranges, not exact headcounts |
| Rating and review count | Maps and review sites | A proxy for size and activity |
| Social profile links | Links in the site's header or footer | The company's own pages on LinkedIn, X, Facebook and others; the script keeps the link, not the profile |
What you do not get: private emails or phone numbers that people have not published, data behind logins, and "verified" contact lists. The API returns the page itself as HTML, Markdown or plain text, or a JSON envelope with the status code, the final URL, the headers, the cookies and the content; it has no parsed company or contact fields, so your parser reads them.
What is lead scraping, and which sources do B2B teams scrape?
Lead scraping is building a prospect list from public web pages instead of buying one: company names, websites, locations, categories and the business contact details companies publish themselves. It is one part of lead generation, next to referrals, ads, events and inbound forms. The pipeline runs in 6 steps: (1) directory and map pages, (2) company websites, (3) parse, (4) deduplicate, (5) verify emails, (6) CRM. B2B sales teams draw on these sources, each suited to a different goal.
Read moreShow less
| Source | Use it for | What it gives you | Blocking (our assessment, not a measurement) | Note |
|---|---|---|---|---|
| Business directories, such as Yellow Pages, Yelp, Thomasnet, Europages, Kompass | Local and regional B2B lists by category | Name, category, address, phone, website, often 20 or more companies per results page | Low to medium | The lowest cost per record, because one page lists many companies |
| Company websites | Enriching every record | Services, locations, team pages, the email and phone on the contact page | Low | The source for enrichment: 2 or 3 pages per company |
| Trade associations and chambers of commerce | Niche industries | Member directories by industry and region | Low | Often small sites with plain HTML |
| Public company registers | Legal names, numbers and officers | Legal name, registration number, address, officers | Low | Use the official data access where it exists, such as Companies House (UK) and SEC EDGAR (US) |
| Maps and local listings | Local businesses by city | Name, category, address, phone, rating and review count | High | Google Maps has its own API: Google Maps scraper, $1.50 per 1,000 successful results |
| Review and software-vendor sites, such as G2, Capterra, Clutch | Niche B2B and software markets | Who sells or buys in a category | Medium | Good for niche B2B markets |
| Job posts, such as Indeed and company career pages | Timing: who is hiring now | Which companies hire for a role: a buying signal | Medium to high | See the job scraper |
| Event, sponsor and press pages | Timing: who is active in a market | Trade-fair exhibitor lists, sponsors, award lists, press releases | Low | Timing signals for outreach |
| LinkedIn company pages | Not a scraping source | Industry, size range, headquarters | High | LinkedIn prohibits scraping software in its User Agreement; use LinkedIn's own tools |
Startup and company databases that show most data only to signed-in or paying users are out of scope: use their own exports or APIs.
How the Web Scraping API fetches a directory page
In the API's design, you send the URL of a public directory results page or company page and choose plain HTTP or JS rendering, and the API returns the page. The API is in development; access is through Contact sales. It takes a POST to https://api.proxyfield.io/v1/scrape with your API key in an Authorization: Bearer <api_key> header and a JSON body: url, render (false for plain HTTP, true for JS rendering), country and output (html, markdown, text or json). For directories that build their results with scripts, wait_for takes a CSS selector to wait for before the rendered page is returned. Parameter reference: API documentation. More on the API itself: Web Scraping API.
# The request the Web Scraping API takes (access through Contact sales)
curl https://api.proxyfield.io/v1/scrape \
-H "Authorization: Bearer <api_key>" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/contact", "render": false, "country": "us", "output": "json"}'Example response (shortened; content is the page's HTML):
{
"metadata": {"id": "req_e0f4a27c8d15b963", "api": "web_scrape", "status": "success", "billed": true},
"status_code": 200,
"final_url": "https://example.com/contact",
"content_type": "text/html; charset=utf-8",
"content": "<!DOCTYPE html><html lang=\"en\">…</html>"
}| Choice | Price per 1,000 successful results | Use it for |
|---|---|---|
| Plain HTTP | $0.50, from $0.20 at volume | Directories and company websites, whose text is in the HTML |
| JS rendering | $2.00, from $1.00 at volume | Directories that build their results with scripts |
Pricing: what a lead costs
A lead costs about $1.53 per 1,000 companies at the entry price and $0.61 per 1,000 companies at the volume price, when a directory page lists 20 companies and you enrich each from 3 pages. Each page is one result: $0.50 per 1,000 successful results over plain HTTP, or $0.20 from 10,000 results in one purchase, and $2.00 per 1,000 with JS rendering, or $1.00 from 10,000.
What counts as a billed result:
- Billed: a page the site answered with a 2xx status (a sign-in page served with 200 is a 2xx answer), and a target's 404 or 410.
- Not billed: error responses, blocked and timed-out requests, and our internal retries on new IPs; only the final delivered page counts.
- Every response says whether it was billed, in the
billedfield.
B2B lead databases usually charge per contact or per credit on a subscription. Scraping costs less per record but leaves the cleaning, deduplication and compliance to you. Payment and balance: the same crypto balance as proxies, which never expires; there is no subscription or monthly fee. All prices: pricing.
Through the sales team on our 24/7 live chat
Sales contact: the sales team on our 24/7 live chat.
| Per 1,000 companies | Pages | Entry price ($0.50 per 1,000 pages) | Volume price ($0.20, from 10,000 results) |
|---|---|---|---|
| Collect from directory pages listing 20 each | 50 | $0.025 | $0.01 |
| Enrich from 3 website pages each | 3,000 | $1.50 | $0.60 |
| Collect and enrich | 3,050 | about $1.53 | $0.61 |
Which proxies for lead scraping?
For your own lead scraper, pick by target: company websites on rotating datacenter ($0.29/GB down to $0.22/GB), directories that limit requests per IP on rotating residential ($2.95/GB for 1 GB, from $1.95/GB), and a target that needs one IP for days on a dedicated ISP IP (from $2.29 per IP / 30 days). Use a new IP on every request and the country of the directories you collect from, such as <login>-country-us on gate.proxyfield.io:7000; for rotating datacenter, add -dc after the login: <login>-dc-country-us (country targeting only). Keep 1 request every few seconds per site and send a User-Agent that says who you are, as the Python script below does. With the Web Scraping API you need no proxies of your own.
Read moreShow less
| Target | Proxy | ProxyField price | 10 GB, about 32,000 companies (example 100 KB a page) |
|---|---|---|---|
| Company websites and low-defence directories | Rotating datacenter; test 100 pages first and move to residential if many fail | $0.29/GB down to $0.22/GB | $2.90 |
| Directories that limit requests per IP | Rotating residential, a new IP per request | $2.95/GB for 1 GB, from $1.95/GB | $27.90 |
| A target that needs the same IP for days | A dedicated static residential (ISP) IP | from $2.29 per IP / 30 days | per IP, not per GB |
What the proxy side costs: at an example 100 KB a page (an input, not a measurement), 10 GB is about 100,000 pages, so about 32,000 companies collected and enriched (3,050 pages per 1,000 companies): $27.90 on residential, or $2.90 on rotating datacenter at the 1–49 GB rate. That is why datacenter comes first where it works. 100 GB of residential is $249.00, the whole purchase at the tier rate.
Residential proxies come with 412M+ IPs in 195+ countries and 21,000+ cities, country, region and city targeting, sticky sessions from 30 s to 36 h, HTTP, HTTPS and SOCKS5, and unlimited concurrent threads. Rotating datacenter proxies cover the same countries as residential. See datacenter proxies and residential proxies.
A proxy does not make collection lawful: see Lead data and the law below.
Build a lead list in 5 steps
-
Define the target industry, location, company size, hiring signals and the job titles you want, as directory categories and cities.
-
Collect companies from directory, association and map results pages: name, website, address, phone.
-
Deduplicate by website domain, then by phone and address.
-
Enrich each company from its own website: the contact and about pages, services, locations.
-
Verify and store run every email through a verification tool such as NeverBounce or ZeroBounce before you send, because published addresses go stale as people change jobs; keep the source URL and the date, and drop records you have no lawful basis to keep.
Scrape company websites with Python and ProxyField proxies
For a few thousand companies, a Python script through rotating ProxyField IPs works today: it fetches the home, contact and about page of each domain (the 3 pages of the worked example) and keeps what the company publishes, from its schema.org JSON-LD, its mailto: and tel: links and its links to its own social profiles. It prints one JSON line per company and writes leads.csv for a spreadsheet or a CRM import. Keep 1 request every few seconds per site and send a User-Agent that says who you are.
Read moreShow less
import csv, json, re, time
from urllib.parse import urlparse
import requests
from bs4 import BeautifulSoup
proxy = "http://<login>-country-us:<password>@gate.proxyfield.io:7000" # residential
# proxy = "http://<login>-dc-country-us:<password>@gate.proxyfield.io:7000" # rotating datacenter, for company websites
domains = ["example.com"] # from your directory results
headers = {"User-Agent": "LeadResearch/1.0 (contact: [email protected])"}
SOCIAL = {"linkedin.com", "x.com", "twitter.com", "facebook.com", "instagram.com", "youtube.com"}
FIELDS = ["domain", "name", "city", "country", "emails", "phones", "social", "sources"]
def as_list(v):
return v if isinstance(v, list) else [v] if v else []
def org_nodes(soup):
"""Blocks of schema.org JSON-LD that describe the business: a name plus a phone, email or address."""
for tag in soup.find_all("script", type="application/ld+json"):
try:
data = json.loads(tag.string or "")
except ValueError:
continue
items = data if isinstance(data, list) else data.get("@graph", [data])
for item in items:
if isinstance(item, dict) and item.get("name") and (
item.get("telephone") or item.get("email") or item.get("address")):
yield item
rows = []
for domain in domains:
rec = {"domain": domain, "name": None, "city": None, "country": None,
"emails": set(), "phones": set(), "social": set(), "sources": []}
for path in ("/", "/contact", "/about"):
url = f"https://{domain}{path}"
try:
r = requests.get(url, timeout=20, headers=headers,
proxies={"http": proxy, "https": proxy})
except requests.RequestException:
continue
if r.status_code != 200: # 403 or 429: slow down, do not retry at once
continue
rec["sources"].append(url)
soup = BeautifulSoup(r.text, "html.parser")
for org in org_nodes(soup):
rec["name"] = rec["name"] or org["name"]
addr = org.get("address")
if isinstance(addr, dict):
rec["city"] = rec["city"] or addr.get("addressLocality")
c = addr.get("addressCountry")
rec["country"] = rec["country"] or (c.get("name") if isinstance(c, dict) else c)
rec["phones"].update(as_list(org.get("telephone")))
rec["emails"].update(e.removeprefix("mailto:").lower() for e in as_list(org.get("email")))
if not rec["name"] and soup.title:
rec["name"] = soup.title.get_text(strip=True)
for a in soup.select("a[href]"):
href = a["href"].strip()
if href.startswith(("mailto:", "tel:")):
value = re.sub(r"^(mailto|tel):", "", href).split("?")[0].strip()
if href.startswith("mailto:"):
rec["emails"].add(value.lower())
else:
rec["phones"].add(value)
elif urlparse(href).netloc.lower().removeprefix("www.") in SOCIAL:
rec["social"].add(href.split("?")[0])
time.sleep(3) # pace each site
row = {k: sorted(v) if isinstance(v, set) else v for k, v in rec.items()}
print(json.dumps(row))
rows.append(row)
with open("leads.csv", "w", newline="") as f: # one row per company, lists joined with "; "
writer = csv.DictWriter(f, fieldnames=FIELDS)
writer.writeheader()
for row in rows:
writer.writerow({k: "; ".join(v) if isinstance(v, list) else v for k, v in row.items()})Each company prints one line like the record above, and leads.csv gets one row per company with the columns domain, name, city, country, emails, phones, social and sources. A domain with no reachable page still gets a row with empty fields, so you can see what failed. Install the two libraries with pip install requests beautifulsoup4; the code needs Python 3.9 or later. Before outreach, run the emails column through an email verification tool (step 5 above).
ProxyField residential proxies cost $2.95/GB for 1 GB and from $1.95/GB at volume, with no subscription; the traffic never expires, and all purchases are final. More on running scrapers through proxies: web scraping proxies.
Lead data and the law
Company data such as names, addresses and switchboard numbers is not personal data unless it identifies a person, such as a sole trader, but a named person's work email is, so GDPR and the UK GDPR apply to it: you need a lawful basis, usually legitimate interest, and you must tell people where you got their data, at the latest within a month (Article 14). The UK regulator's guidance on business-to-business marketing notes that public business information can still be personal data. Data you got from a third party must have been collected lawfully and be fit for direct marketing, as the European Commission explains. For email outreach, the UK's PECR allows marketing email to corporate subscribers with an opt-out but treats sole traders like consumers, who must have agreed to marketing email, and the US CAN-SPAM Act requires an opt-out, your valid physical postal address and honest headers; "the law makes no exception for business-to-business email" (FTC compliance guide). The ICO, European Commission and FTC pages were checked on 2026-10-28. The directories' own terms often forbid automated collection. More in Is Web Scraping Legal? US, EU and UK Rules (2026).
ProxyField's acceptable use policy allows collecting public web data and market research, and forbids sending spam and accessing data behind logins you are not entitled to. ProxyField's rule for every site: public pages only, and follow the site's terms. This is not legal advice.
More scraper APIs
- Web Scraping API any URL, plain HTTP or with JS rendering, from $0.20 per 1,000 successful results.
- Google Maps scraper Google Maps search results, $1.50 per 1,000 successful results, from $1.00.
- Job scraper public job listing pages through the Web Scraping API, for hiring signals.
- Google SERP API the Google results page as JSON, $1.20 per 1,000 successful results, from $0.80.
- All scraper APIs
Lead Scraper API FAQ
24/7 support by live chat, including help with setup and integration.
What's the best B2B lead scraper for pulling company and contact data from websites and directories?
ProxyField's route for pulling company and contact data from directories and websites you choose works today: the Python script on this page, run on ProxyField proxies, writes one record per company (name, city, country, business emails, phones, social links, source URLs) to JSON lines and CSV, for about $0.90 of residential traffic per 1,000 companies at an example 100 KB a page, or $0.09 on rotating datacenter. The Web Scraping API (in development, Contact sales) is designed to fetch the same pages for about $1.53 per 1,000 companies, or $0.61 from 10,000 results in one purchase. Neither gives you named people or verified emails: for contacts ready for a CRM, pick a B2B lead database, and run every email through a verification tool before outreach.
How much does lead scraping cost per contact compared with buying a B2B lead database?
With ProxyField, 1,000 companies collected from directory pages that list 20 each and enriched from 3 pages of each website are 3,050 pages: about $0.90 of residential traffic from your own script (at an example 100 KB a page), or about $1.53 on the Web Scraping API at $0.50 per 1,000 results, $0.61 at the $0.20 volume price (from 10,000 results in one purchase). Lead databases usually charge per contact or credit on a subscription, but include named contacts and cleaning that scraping leaves to you.
What is lead scraping, and which sources do B2B sales teams usually scrape?
Lead scraping is building a prospect list from public web pages instead of buying one. Sales teams usually scrape business directories such as Yellow Pages or Thomasnet, company websites, trade association member lists, public company registers such as Companies House and SEC EDGAR, maps and local listings, review sites such as G2 and Clutch, job posts as hiring signals, and event and sponsor pages.
What proxies do I need for B2B lead generation scraping?
ProxyField rotating datacenter proxies ($0.29/GB down to $0.22/GB) for company websites after a test, rotating residential ($2.95/GB for 1 GB, from $1.95/GB) for directories that limit requests per IP, and a dedicated ISP IP (from $2.29 per IP / 30 days) when a target needs one IP for days. Use a new IP on every request; at an example 100 KB a page, 10 GB is about 100,000 pages, $27.90 on residential or $2.90 on datacenter. With the Web Scraping API you need no proxies of your own.
Are datacenter proxies enough for scraping business directories?
Often for company websites and small directories: test 100 pages first and move the targets that fail to residential. Directories that limit requests per IP or challenge server IPs need rotating residential IPs. ProxyField's rotating datacenter proxies cost $0.29 down to $0.22/GB, residential $2.95 down to $1.95/GB.
How do I scrape leads?
Pick the directory categories and cities that match your target, collect the companies from the results pages, deduplicate them by website, then visit each company's site for its contact page. Keep the source URL and date with every record.
How do I scrape contact data for B2B lead generation?
Fetch each company's home and contact pages and keep the email and phone links the company publishes, as in the code on this page. Role addresses such as sales@ are the usual result; personal emails that nobody published are not on public pages.
How does lead scraping software work?
It takes a search, such as a category and a city, fetches the matching directory or map pages, extracts each company's fields and visits the company websites to add contact details. The better tools deduplicate the results and export them to a spreadsheet or CRM.
How accurate are email scrapers at finding emails?
A scraper finds only addresses that are on the page, so it is accurate about what it finds but misses companies that publish a contact form instead. Validate addresses before sending, because published addresses go stale.
How do I avoid duplicate leads when scraping?
Normalise the website to its domain and use it as the key, then merge records that share a phone number or an address. Keep the first-seen date and every source URL, so a re-run updates records instead of adding new ones.
Show 11 more questionsShow fewer questions
What is the biggest reason lead scrapers fail?
Blocks and layout changes: directories limit requests per IP, and a redesign breaks the selectors without an error. Rotate IPs, keep the rate low and alert when a run returns far fewer records than the last one.
Is scraping lead data legal?
Collecting public company data is generally lawful, but a named person's contact details are personal data under GDPR and the UK GDPR, and outreach falls under rules such as PECR in the UK and CAN-SPAM in the US. Directories' terms often forbid automated collection; this is not legal advice.
Is lead scraping the same as lead generation?
No. Lead generation is every way of finding prospects, including referrals, ads, events and inbound forms; lead scraping is one of them, building the list from public web pages. On ProxyField, a scraped list of 1,000 companies costs about $1.53 in pages on the Web Scraping API.
How do I verify scraped emails before outreach?
Run the list through an email verification tool such as NeverBounce or ZeroBounce, which check whether each address can receive mail, and drop the ones that fail. Published addresses go stale as people change jobs, so verify again before every campaign and keep the date you last checked each record.
Can I use public company registers such as Companies House for leads?
Yes, and you rarely need to scrape them: Companies House in the UK and SEC EDGAR in the US offer official data access to company names, numbers, addresses and filings. Officers' names are personal data, so the GDPR and UK GDPR rules on this page apply to them.
How do I scrape Yellow Pages?
Fetch the results pages for a category and city, read each listing's name, phone, address and website, and follow the pagination to the end. On ProxyField's Web Scraping API each results page costs $0.50 per 1,000 successful results; the directory's terms apply.
Can I scrape Crunchbase?
Crunchbase shows most company data only to signed-in or paying users, and data behind a login is out of scope on ProxyField. Use Crunchbase's own exports or API for that data, and public pages such as company websites for the rest.
How do I get scraped leads into my CRM?
Export the cleaned records to CSV with the CRM's field names and import them, or push them through the CRM's API from your script. Keep the source URL in a custom field, so you can answer where each record came from.
What is lead enrichment?
Lead enrichment is adding missing fields to a record you already have, such as the website, industry, size range or business phone. Scraping does it by visiting the company's own website and public profiles.
Is there low-cost lead scraper software?
Open-source scrapers cost nothing to license but need a developer and proxies; browser extensions handle a few pages at a time. ProxyField's Web Scraping API costs $0.50 per 1,000 successful pages, so 1,000 companies from directory pages that list 20 each cost $0.025 in fetching, and about $1.53 with 3 website pages each.
Do I need JavaScript rendering for lead scraping?
Usually not: most directories and company websites put their text in the HTML, so plain HTTP at $0.50 per 1,000 successful results is enough. Render only the directories that build their results with scripts, at $2.00 per 1,000.
Related pages
Start collecting public pages with the Web Scraping API
Billed on successful results only. Contact sales for the Web Scraping API and the volume you need, or run your own scraper on residential proxies. 24/7 live chat helps with setup and integration.