Ruby Web Scraping

For most static pages, HTTParty plus Nokogiri is all you need.

gem install httparty nokogiri

require "httparty"
require "nokogiri"

url = "https://www.scrapethissite.com/pages/simple/"
response = HTTParty.get(url, headers: { "User-Agent" => "Mozilla/5.0" }, timeout: 30)
raise "HTTP #{response.code}" unless response.code == 200

doc = Nokogiri::HTML(response.body)
countries = doc.css("div.country").map do |card|
  {
    name: card.at_css("h3.country-name").text.strip,
    capital: card.at_css("span.country-capital").text.strip,
    population: card.at_css("span.country-population").text.to_i
  }
end
puts countries.size, countries.first

The page is a practice sandbox with 250 countries, each in a div.country. css returns every match and at_css the first; xpath and at_xpath are the XPath versions. Read an attribute with node["href"], and make relative links absolute with URI.join(url, href).to_s.

When the page number is a URL parameter, count up until a page comes back empty. On the sandbox's hockey table it is ?page_num=N:

require "csv"

teams, page = [], 1
loop do
  res = HTTParty.get("https://www.scrapethissite.com/pages/forms/",
                     query: { page_num: page }, timeout: 30)
  rows = Nokogiri::HTML(res.body).css("tr.team")
  break if rows.empty?
  rows.each do |tr|
    teams << [tr.at_css("td.name").text.strip, tr.at_css("td.year").text.to_i, tr.at_css("td.wins").text.to_i]
  end
  page += 1
  sleep 1
end

CSV.open("teams.csv", "w") do |csv|
  csv << %w[team year wins]
  teams.each { |row| csv << row }
end

Where the site has a "Next" link instead, read doc.at_css("a[rel=next]")&.[]("href") and stop when it is nil. Mechanize does the same with page.link_with(text: "Next")&.click, and keeps cookies between pages.

Unattended Ruby scrapers fail in three predictable ways: timeouts, blocks and layout changes.

• Retries with backoff: retry Net::OpenTimeout, Net::ReadTimeout and 429 or 5xx responses up to 3 times, sleeping 2, 4 and 8 seconds. With Faraday, the faraday-retry middleware does this for you.
• Honor Retry-After on 429 responses, and keep a delay per host.
• Fail loudly on nil: at_css returns nil when a selector matches nothing, and .text on nil raises NoMethodError. Check for nil and log the URL, so a layout change stops the run instead of writing empty rows.

HTTParty takes the proxy as four options. The ProxyField gateway is gate.proxyfield.io:7000; <login>-country-us-city-new_york asks for an exit IP in New York. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.

response = HTTParty.get("https://api.ipify.org",
  http_proxyaddr: "gate.proxyfield.io",
  http_proxyport: 7000,
  http_proxyuser: "<login>-country-us-city-new_york",
  http_proxypass: "<password>",
  timeout: 30)
puts response.body   # the IP the site sees

The same with Faraday and with Net::HTTP:

require "faraday"
conn = Faraday.new(proxy: "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000")
puts conn.get("https://api.ipify.org").body

require "net/http"
uri = URI("https://api.ipify.org")
Net::HTTP.start(uri.host, uri.port, "gate.proxyfield.io", 7000,
                "<login>-country-us-city-new_york", "<password>", use_ssl: true) do |http|
  puts http.get(uri.request_uri).body
end

The gateway can give each request a new IP, or keep one IP for a sticky session from 30 s to 36 h, for multi-page flows that need the same visitor. Which IP type to use for which site: proxies for web scraping.

Nokogiri does not run JavaScript. If the data is missing from response.body but visible in the browser, first look for the JSON request the page makes (DevTools > Network > Fetch/XHR) and call it with HTTParty. If you need the rendered page, drive a browser.

Ferrum controls Chrome directly, without a separate driver:

require "ferrum"

browser = Ferrum::Browser.new(timeout: 30, proxy: {
  host: "gate.proxyfield.io", port: 7000,
  user: "<login>-country-us-city-new_york", password: "<password>"
})
browser.go_to("https://example.com/js-page")
html = browser.body              # the HTML after scripts ran
browser.quit
puts Nokogiri::HTML(html).at_css("h1")&.text

Playwright in Ruby goes through the playwright-ruby-client gem, which uses the Playwright driver from npm (npm install playwright and npx playwright install chromium first):

require "playwright"

Playwright.create(playwright_cli_executable_path: "npx playwright") do |playwright|
  playwright.chromium.launch(headless: true, proxy: {
    server: "http://gate.proxyfield.io:7000",
    username: "<login>-country-us-city-new_york", password: "<password>"
  }) do |browser|
    page = browser.new_page
    page.goto("https://example.com/js-page")
    page.wait_for_selector("h1")
    puts page.text_content("h1")
  end
end

Selenium supports Ruby through the official selenium-webdriver gem. Chrome's --proxy-server flag takes no username or password, so with Selenium the simplest setup is an IP allowlist: add your server's IP to your account's allowlist, then pass --proxy-server=http://gate.proxyfield.io:<port>, where <port> is the port you set up in the dashboard for that IP. More on authenticated proxies in Selenium: Selenium proxy setup.

Browsers download every script, image and font, so they are slower and use more proxy traffic per page than plain HTTP; render only the pages that need it. The general approach is in how to scrape dynamic web pages.

Never scrape inside a web request: a slow site would hold up your users. Put the scraper in a job and schedule it.

# app/jobs/scrape_countries_job.rb
class ScrapeCountriesJob < ApplicationJob
  queue_as :default
  retry_on Net::ReadTimeout, wait: :polynomially_longer, attempts: 3

  def perform
    res = HTTParty.get("https://www.scrapethissite.com/pages/simple/", timeout: 30,
            http_proxyaddr: "gate.proxyfield.io", http_proxyport: 7000,
            http_proxyuser: ENV["PROXY_USER"], http_proxypass: ENV["PROXY_PASS"])
    Nokogiri::HTML(res.body).css("div.country").each do |card|
      Country.upsert({ name: card.at_css("h3.country-name").text.strip,
                       population: card.at_css("span.country-population").text.to_i },
                     unique_by: :name)
    end
  end
end

upsert with a unique index keeps reruns from duplicating rows. Schedule the job with your queue's recurring tasks: Solid Queue, the default queue backend in Rails 8, reads them from config/recurring.yml. Keep the proxy credentials in Rails credentials or environment variables.

Yes, for small and medium jobs, and especially when the rest of your app is Ruby: Nokogiri is fast and mature, and HTTParty and Faraday make requests short to write. Python has the larger scraping ecosystem (Scrapy, more tutorials, more parsers), and Node.js has the most browser tooling; for very high page counts per server, compiled languages such as Rust or Go use less memory. Comparisons by language: choosing a language for web scraping. Other guides in this series: web scraping in Rust, how to build a web scraper.

Running Ruby scrapers at volume? ProxyField residential proxies: 412M+ residential IPs in 195+ countries, unlimited concurrent threads, pay per GB with no subscription. Start with the Easy Trial (1 GB); traffic never expires, and all purchases are final.