Web Scraping in R With rvest

R is a good choice when the scraped data goes straight into analysis: the rows land in a data frame, and dplyr, ggplot2 or a model take it from there in the same script. rvest, written by Hadley Wickham and part of the tidyverse, keeps scraping short and pipe-friendly. Python has more scraping libraries and frameworks for large crawls (Python web scraping), but for research, reporting and a few thousand pages, R does the job without switching languages. Language trade-offs in general: best language for web scraping.

Versions from CRAN.

The polite package, which many older tutorials use, was removed from CRAN on 2026-09-12 (CRAN); the steps below use robotstxt and a pause instead.

install.packages(c("rvest", "dplyr", "stringr", "readr", "janitor", "httr2", "robotstxt"))
library(rvest)
library(dplyr)
library(stringr)
library(readr)

rvest pulls in xml2; dplyr, stringr and readr do the cleaning in step 4, and janitor tidies table headers in step 5. Install chromote too if you need JavaScript pages (step 7).

The examples use books.toscrape.com, a practice site made for learning to scrape.

page <- read_html("https://books.toscrape.com/")
page |> html_element("title") |> html_text2()

read_html() downloads the page and parses it into a document you can query. Before you scrape a real site, look at the page source in your browser (Ctrl+U): if the data is not there, the page renders it with JavaScript (step 7).

html_elements() returns every match, html_element() the first match per input. Find the selectors with the browser's Inspect panel or with the SelectorGadget bookmarklet that rvest's documentation recommends.

books <- page |> html_elements("article.product_pod")
length(books)   # 20 books on the page

titles <- books |> html_element("h3 a") |> html_attr("title")
prices <- books |> html_element("p.price_color") |> html_text2()
rating <- books |> html_element("p.star-rating") |> html_attr("class")
links  <- books |> html_element("h3 a") |> html_attr("href") |> url_absolute("https://books.toscrape.com/")

Use html_text2() rather than html_text(): it returns the text as a browser would show it, turning <br> and block elements into line breaks and collapsing extra spaces, while html_text() returns the raw text, including the HTML's indentation and newlines. url_absolute() turns relative links into full URLs. Writing CSS and XPath selectors in depth: HTML scraping.

Put the raw fields in a tibble, then clean every column in one mutate():

raw <- tibble(title = titles, price = prices, rating = rating, url = links)

books_df <- raw |>
  mutate(
    title  = str_squish(title),                              # trim and collapse spaces
    price  = parse_number(price),                            # "£51.77" -> 51.77
    rating = match(str_remove(rating, "star-rating "),       # "star-rating Three" -> 3
                   c("One", "Two", "Three", "Four", "Five"))
  ) |>
  filter(!is.na(title), title != "") |>                      # drop empty records
  distinct(url, .keep_all = TRUE)                            # one row per book

glimpse(books_df)
#> Rows: 20
#> Columns: 4
#> $ title  <chr> "A Light in the Attic", "Tipping the Velvet", "Soumission", "Sharp Objects", …
#> $ price  <dbl> 51.77, 53.74, 50.10, 47.82, …
#> $ rating <int> 3, 1, 1, 4, …
#> $ url    <chr> "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", …

Select fields from each book card (books |> html_element(...)), not from the whole page (page |> html_elements(...)). html_element() returns exactly one result per card, so a book without a price gets NA and every column has 20 values. Selecting each field from the whole page returns vectors of different lengths when one card lacks a field, and tibble() stops with an error such as Tibble columns must have compatible sizes.

Tables are where R shines. html_table() turns a <table> into a tibble in one call. On one element (html_element()) it returns one tibble; on several elements or the whole document it returns a list of tibbles, one per table.

detail <- read_html("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html")
info <- detail |> html_element("table.table-striped") |> html_table(header = FALSE)
info
#> # A tibble: 7 × 2
#>   X1                X2
#>   <chr>             <chr>
#> 1 UPC               a897fe39b1053632
#> 2 Product Type      Books
#> 3 Price (excl. tax) £51.77
#> …

# one row per book, with clean column names
info |>
  tidyr::pivot_wider(names_from = X1, values_from = X2) |>
  janitor::clean_names()        # upc, product_type, price_excl_tax, …

all_tables <- detail |> html_elements("table") |> html_table()   # a list of tibbles

The arguments, as the rvest reference documents them (html_table(x, header = NA, trim = TRUE, dec = ".", na.strings = "NA", convert = TRUE)):

Merged cells (colspan, rowspan) are handled as a browser shows them: the value is repeated in every cell the merged cell covers. Rows with too few cells are filled with NA; the old fill argument is deprecated for that reason. With header = TRUE, names stay exactly as in the page, so run janitor::clean_names() before you refer to them in code.

library(robotstxt)
paths_allowed("https://books.toscrape.com/catalogue/page-2.html")   # TRUE if robots.txt allows it

scrape_page <- function(url) {
  p <- read_html(url)
  b <- p |> html_elements("article.product_pod")
  tibble(
    title = b |> html_element("h3 a") |> html_attr("title"),
    price = b |> html_element("p.price_color") |> html_text2()
  )
}

urls <- sprintf("https://books.toscrape.com/catalogue/page-%d.html", 1:50)
all_books <- purrr::map(urls, \(u) { Sys.sleep(1); scrape_page(u) }) |> purrr::list_rbind()
write.csv(all_books, "books.csv", row.names = FALSE)
nrow(all_books)   # 1,000

Sys.sleep(1) keeps the pace at one page per second. robots.txt lists the paths a site asks crawlers to avoid; it is a request, not access control, and the site's terms apply too (RFC 9309). How to check both: how to check if scraping is allowed.

rvest 1.0.4 (February 2024) added read_html_live(), which loads the page in a real Chrome through chromote, so you can scrape HTML that JavaScript generates (rvest changelog). It returns a live object you can also scroll, click and type into. The function is marked experimental.

install.packages("chromote")   # needs Chrome or Chromium installed
live <- read_html_live("https://quotes.toscrape.com/js/")
live |> html_elements("div.quote span.text") |> html_text2() |> head(3)
live$click("li.next a")        # interact, then select again

Before you render, open the browser's Network tab: many JavaScript pages load their data from a JSON request, which httr2 can call directly and httr2::resp_body_json() parses into a list. RSelenium is the older way to drive a browser from R; for most new code, read_html_live() is simpler. Dynamic pages in general: how to scrape dynamic web pages.

read_html() is fine for small jobs. For larger ones, fetch with httr2, which adds a User-Agent, retries on temporary errors, a rate limit and a proxy, then parse the body with rvest.

library(httr2)

fetch <- function(url) {
  request(url) |>
    req_user_agent("books-tutorial/1.0 ([email protected])") |>
    req_retry(max_tries = 4) |>
    req_throttle(capacity = 30, fill_time_s = 60) |>      # about 30 requests a minute
    req_proxy("http://gate.proxyfield.io", port = 7000,
              username = "<login>-country-us-city-new_york", password = "<password>") |>
    req_perform() |>
    resp_body_html()
}

page <- fetch("https://books.toscrape.com/")
page |> html_elements("h3 a") |> html_attr("title") |> head()

req_proxy(req, url, port, username, password) is httr2's documented proxy function (httr2 reference). Remove the req_proxy line to send requests from your own IP. In the ProxyField username, <login>-country-us-city-new_york asks for exit IPs in New York. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.

session() keeps cookies between requests and lets you follow links (session_follow_link()) and submit forms (html_form(), html_form_set(), session_submit()), which covers simple searches and logins to your own accounts. Do not scrape data behind logins you are not entitled to, and read is web scraping legal before you collect personal data.

Sites limit how many requests one IP may send. If your R scraper starts getting 403 or 429 responses at a polite pace, or you need pages as they look from another country, route the httr2 requests through rotating proxies as in step 8: each request then leaves from a new IP. ProxyField residential proxies give you a new IP per request or a sticky session from 30 s to 36 h, with country, region and city targeting over HTTP, HTTPS or SOCKS5. They cost $2.95/GB for 1 GB and fall to $1.95/GB from 1,000 GB, with no subscription; traffic never expires, and all purchases are final. Plain HTML pages are small, so 1 GB covers thousands of pages like these. Start with residential proxies and the Easy Trial: 1 GB for $2.95.