Java Web Scraping

Yes, especially for long-running and large jobs: Java is fast, strongly typed, has mature HTTP and HTML libraries, and runs thousands of concurrent requests on virtual threads (Java 21+). Teams that already run Java services can scrape in the same stack, with the same logging, scheduling and deployment. Python has more scraping tutorials and frameworks, and Node.js has the most browser automation; Java's advantage is robustness at scale. Kotlin, Scala and other JVM languages use the same libraries.

Which one to pick: start with jsoup. If the page source does not contain the data, look for the page's JSON request in DevTools and call it with HttpClient. Only if neither works, use Playwright for Java or Selenium; HtmlUnit fits in between for light scripting.

Add jsoup to Maven (or Gradle); the current version is on its site (jsoup.org): 1.23.2 in October 2026.

<dependency>
  <groupId>org.jsoup</groupId>
  <artifactId>jsoup</artifactId>
  <version>1.23.2</version>
</dependency>

This program reads every book's title, price and stock status from a practice catalog, follows the "next" link through all pages, waits a second between requests and writes a CSV file.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;

import java.io.PrintWriter;
import java.nio.file.Files;
import java.nio.file.Path;

public class BookScraper {
    public static void main(String[] args) throws Exception {
        String url = "https://books.toscrape.com/catalogue/page-1.html";
        try (PrintWriter out = new PrintWriter(Files.newBufferedWriter(Path.of("books.csv")))) {
            out.println("title,price,in_stock");
            while (url != null) {
                Document doc = Jsoup.connect(url)
                        .userAgent("Mozilla/5.0 (compatible; books-demo/1.0)")
                        .timeout(30_000)
                        .get();
                for (Element book : doc.select("article.product_pod")) {
                    String title = book.selectFirst("h3 a").attr("title").replace("\"", "\"\"");
                    String price = book.selectFirst(".price_color").text();
                    boolean inStock = book.selectFirst(".instock") != null;
                    out.printf("\"%s\",%s,%b%n", title, price, inStock);
                }
                Element next = doc.selectFirst("li.next a");
                url = next != null ? next.absUrl("href") : null;   // absolute URL of the next page
                Thread.sleep(1_000);                                // one request per second
            }
        }
    }
}

What it does:

1. Jsoup.connect(url).get() downloads and parses the page in one call; userAgent() and timeout() set the header and the limit.
2. doc.select("article.product_pod") returns every element matching the CSS selector; selectFirst() returns one or null.
3. attr("title") reads an attribute and text() the visible text; absUrl("href") turns a relative link into an absolute one.
4. The loop ends when there is no "next" link.

jsoup also cleans untrusted HTML against an allowlist (Jsoup.clean(html, Safelist.basic())), which is useful before you store scraped HTML.

jsoup sees only the HTML the server sends. If the data is filled in by scripts, you have three options in Java.

HtmlUnit runs the page's JavaScript in its own engine, without a real browser; it handles many simple pages and forms, but complex front-end frameworks may not behave as in Chrome (htmlunit.org). HtmlUnit 5 uses the org.htmlunit package and needs JDK 17 or later.

import org.htmlunit.BrowserVersion;
import org.htmlunit.WebClient;
import org.htmlunit.html.DomNode;
import org.htmlunit.html.HtmlPage;

try (WebClient client = new WebClient(BrowserVersion.CHROME)) {
    client.getOptions().setCssEnabled(false);
    client.getOptions().setThrowExceptionOnScriptError(false);
    HtmlPage page = client.getPage("https://quotes.toscrape.com/js/");
    client.waitForBackgroundJavaScript(5_000);
    for (DomNode quote : page.querySelectorAll(".quote .text")) {
        System.out.println(quote.asNormalizedText());
    }
}

Selenium drives a real browser; Selenium Manager downloads the matching driver. More in what Selenium is.

import java.time.Duration;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

ChromeOptions options = new ChromeOptions();
options.addArguments("--headless=new");
WebDriver driver = new ChromeDriver(options);
try {
    driver.get("https://quotes.toscrape.com/js/");
    new WebDriverWait(driver, Duration.ofSeconds(15))
            .until(ExpectedConditions.presenceOfElementLocated(By.cssSelector(".quote")));
    driver.findElements(By.cssSelector(".quote .text")).forEach(e -> System.out.println(e.getText()));
} finally {
    driver.quit();
}

Playwright for Java (com.microsoft.playwright:playwright) has the same API as in Python and Node.js, with automatic waiting; see what Playwright is.

import com.microsoft.playwright.*;

try (Playwright pw = Playwright.create()) {
    Browser browser = pw.chromium().launch();
    Page page = browser.newPage();
    page.navigate("https://quotes.toscrape.com/js/");
    page.locator(".quote").first().waitFor();
    System.out.println(page.locator(".quote .text").allInnerTexts());
}

A browser costs far more time, memory and traffic per page than jsoup, so use it only for the pages that need it; headless browser scraping explains how to tell.

When a site limits requests per IP, send them through a rotating residential proxy. jsoup takes the proxy host and port with proxy() and the credentials with auth() (jsoup 1.17.1 and later). One JDK detail trips most people up: Java disables Basic authentication for proxies when tunneling HTTPS, so set jdk.http.auth.tunneling.disabledSchemes to an empty string before the first request, or HTTPS requests fail with 407 Proxy Authentication Required even with the right password.

static {
    System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");   // allow Basic for HTTPS tunnels
}

Document doc = Jsoup.connect("https://api.ipify.org?format=json")
        .proxy("gate.proxyfield.io", 7000)
        .auth(ctx -> ctx.isProxy() ? ctx.credentials("<login>-country-us-city-new_york", "<password>") : null)
        .ignoreContentType(true)
        .get();
System.out.println(doc.text());   // the exit IP

With java.net.http.HttpClient, set ProxySelector.of(new InetSocketAddress("gate.proxyfield.io", 7000)) and an Authenticator that returns the credentials for RequestorType.PROXY; the same system property applies. With an IP allowlist, auth is not needed, and you connect to the port you set up in the dashboard instead of 7000. <login>-country-us-city-new_york asks for an IP in New York, United States. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.

• Concurrency. On Java 21+, Executors.newVirtualThreadPerTaskExecutor() runs one lightweight thread per URL; cap requests per site with a Semaphore so you do not overload it.
• Thread safety. Use one Connection per request (or a Jsoup.newSession() per thread); parsed Document objects should not be shared between threads while they are modified.
• Retries. Retry timeouts, 429 and 5xx responses with exponential backoff; see web scraping best practices.
• Politeness. Read robots.txt and the site's terms, keep the rate low, and collect only public data; see is web scraping legal.

"Screen scraping" is the older name for reading data from what an application displays. On the web it means browser-based scraping, which in Java is Selenium or Playwright reading the rendered page, or taking a screenshot (page.screenshot() in Playwright). Reading text from images needs an OCR library, and data from desktop applications needs UI automation tools; both are outside web scraping.

ProxyField residential proxies for Java scrapers cost $2.95/GB for 1 GB down to $1.95/GB from 1,000 GB, with no subscription; the traffic never expires, and all purchases are final (residential and mobile proxy pricing).