Web Scraping in Rust

For a static page, reqwest plus scraper is enough; add futures when you fetch many pages at once.

The example reads the practice sandbox scrapethissite.com, where each of 250 countries sits in a div.country with its name in h3.country-name and its capital in span.country-capital.

use scraper::{Html, Selector};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = reqwest::Client::builder()
        .user_agent("Mozilla/5.0 (compatible; rust-scraper/1.0)")
        .timeout(std::time::Duration::from_secs(30))
        .build()?;

    let body = client
        .get("https://www.scrapethissite.com/pages/simple/")
        .send().await?
        .error_for_status()?
        .text().await?;

    for (name, capital) in parse_countries(&body) {
        println!("{name}\t{capital}");
    }
    Ok(())
}

fn parse_countries(html: &str) -> Vec<(String, String)> {
    let doc = Html::parse_document(html);
    let card = Selector::parse("div.country").unwrap();
    let name = Selector::parse("h3.country-name").unwrap();
    let capital = Selector::parse("span.country-capital").unwrap();
    doc.select(&card)
        .map(|c| {
            let text = |sel: &Selector| {
                c.select(sel).next()
                    .map(|e| e.text().collect::<String>().trim().to_string())
                    .unwrap_or_default()
            };
            (text(&name), text(&capital))
        })
        .collect()
}

Three things to know about scraper:

• Selector::parse returns a Result; build selectors once, outside loops.
• element.text() yields the text nodes; collect them into a String and trim.
• Read attributes with element.value().attr("href"), which returns Option<&str>.

Parsing sits in its own function because scraper::Html is not Send: keeping it out of async code that crosses an .await avoids compiler errors when you later spawn tasks.

Rust's advantage shows when you fetch many pages concurrently. buffer_unordered(n) keeps n requests in flight; set n with the target site in mind, not your CPU.

use futures::{stream, StreamExt};

async fn fetch_all(client: &reqwest::Client, urls: Vec<String>) -> Vec<(String, usize)> {
    stream::iter(urls)
        .map(|url| async move {
            let res = client.get(&url).send().await.and_then(|r| r.error_for_status());
            let len = match res {
                Ok(r) => r.text().await.map(|t| t.len()).unwrap_or(0),
                Err(e) => { eprintln!("{url}: {e}"); 0 }
            };
            (url, len)
        })
        .buffer_unordered(8)        // 8 concurrent requests
        .collect()
        .await
}

For pagination on the sandbox's hockey table, build the URLs https://www.scrapethissite.com/pages/forms/?page_num=1 to N and pass them in. Retries are up to you: wrap the request in a loop that retries timeouts and 429 or 5xx responses with a growing delay. For pacing and concurrency in general, see how to make web scraping faster.

reqwest takes a proxy on the client builder. Proxy::all sends both HTTP and HTTPS requests through it, and basic_auth adds the credentials. The ProxyField gateway is gate.proxyfield.io:7000; <login>-country-us-city-new_york asks for an exit IP in New York. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.

let proxy = reqwest::Proxy::all("http://gate.proxyfield.io:7000")?
    .basic_auth("<login>-country-us-city-new_york", "<password>");

let client = reqwest::Client::builder()
    .proxy(proxy)
    .timeout(std::time::Duration::from_secs(30))
    .build()?;

println!("{}", client.get("https://api.ipify.org").send().await?.text().await?);

The gateway can give every request a new IP, or keep one IP for a sticky session from 30 s to 36 h. reqwest also reads the HTTP_PROXY and HTTPS_PROXY environment variables unless you call .no_proxy() on the builder, so check your environment if requests go somewhere you did not expect. Which IP type to use: proxies for web scraping.

scraper parses what the server sends and runs no JavaScript. If the data is missing from the HTML, first look for the JSON request the page makes (DevTools > Network > Fetch/XHR) and call it with reqwest and serde_json. If you need a real browser:

thirtyfour (WebDriver): start chromedriver --port=9515, then:

use thirtyfour::prelude::*;

#[tokio::main]
async fn main() -> WebDriverResult<()> {
    let mut caps = DesiredCapabilities::chrome();
    caps.add_arg("--headless=new")?;
    caps.add_arg("--proxy-server=http://gate.proxyfield.io:<port>")?;   // IP allowlist auth
    let driver = WebDriver::new("http://localhost:9515", caps).await?;
    driver.goto("https://example.com/js-page").await?;
    let heading = driver.find(By::Css("h1")).await?;
    println!("{}", heading.text().await?);
    driver.quit().await?;
    Ok(())
}

Chrome's --proxy-server flag carries no username or password, so this setup uses an IP allowlist: add your server's IP to your account's allowlist, and replace <port> with the port you set up in the dashboard for it. thirtyfour is the crate behind most "Rust Selenium" examples; it speaks the same WebDriver protocol as Selenium. The browser side, for any language: how to scrape dynamic web pages.

chromiumoxide or headless_chrome control Chrome directly over the DevTools protocol, without a driver process, and can intercept network requests; they suit jobs that render many pages from one long-running browser.

Pick Rust when the scraper runs for a long time at high volume on limited hardware, or when it belongs inside a Rust service. Pick Python when the site changes often and you rewrite selectors weekly: the network is the slow part of most scrapers, so the language rarely decides speed. Language-by-language comparison: choosing a language for web scraping. Other guides in this series: Ruby web scraping and how to build a web crawler.

High-volume scraping in Rust? ProxyField residential proxies: 412M+ residential IPs in 195+ countries, unlimited concurrent threads, an average response of about 400 ms according to our service data. Pay per GB, no subscription; start with the Easy Trial (1 GB). Traffic never expires, and all purchases are final.