Web Scraping With Selenium and Python

python3 -m pip install -U selenium

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")         # Chrome without a window
options.add_argument("--window-size=1366,900")

with webdriver.Chrome(options=options) as driver:   # quits Chrome even on errors
    driver.get("https://quotes.toscrape.com/js/")    # a practice page built by JavaScript
    WebDriverWait(driver, 10).until(
        EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.quote"))
    )
    print(driver.title)
    for q in driver.find_elements(By.CSS_SELECTOR, "div.quote")[:3]:
        print(q.find_element(By.CSS_SELECTOR, "small.author").text, "|",
              q.find_element(By.CSS_SELECTOR, "span.text").text[:50])

Output, run on 2026-10-27 with Selenium 4.51.0, Python 3.12 and Chrome 155 on Linux:

Quotes to Scrape
Albert Einstein | “The world as we have created it is a process of o
J.K. Rowling | “It is our choices, Harry, that show what we truly
Albert Einstein | “There are only two ways to live your life. One is

You need only Chrome and the package: Selenium Manager downloads the matching chromedriver on the first run. The wait matters: without it, find_elements() can run before the page's JavaScript has added the quotes. The seven steps below take the same script further: locating, waiting, infinite scroll, screenshots and saving to CSV.

Selenium drives a real browser through WebDriver, the W3C standard that Chrome, Firefox, Edge and Safari implement. A browser runs the page's JavaScript, so Selenium sees what a visitor sees after the scripts have run. That makes it a good fit for:

• pages that load their content with JavaScript and have no JSON request you can call directly;
• data behind clicks, tabs, "load more" buttons or infinite scroll;
• screenshots of pages as rendered;
• teams that already run Selenium for testing and want one tool.

It is a poor fit for large crawls of static pages: a browser costs far more CPU, memory and traffic per page than an HTTP request. For those, see Python web scraping with Requests or Scrapy. Selenium in general: what Selenium is.

python3 -m venv .venv && source .venv/bin/activate
pip install selenium

That is all: since Selenium 4.6, Selenium Manager finds or downloads the matching driver (chromedriver, geckodriver) when it is missing, so you no longer download drivers by hand (Selenium Manager docs). The current Python package is 4.51.0, which needs Python 3.10 or newer (PyPI).

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")        # no window; drop it to watch the browser
options.add_argument("--window-size=1366,900")
driver = webdriver.Chrome(options=options)
driver.get("https://quotes.toscrape.com/js/")  # a practice page rendered by JavaScript
print(driver.title)

--headless=new runs the full Chrome without a window. Selenium's Chrome options page lists it with --start-maximized and --user-data-dir=... among the commonly used arguments (Selenium: Chrome options), and the project recommends it over the old headless mode (Selenium blog: Headless is going away). Since Chrome 132, plain --headless means the same, because the old mode ships only as the separate binary chrome-headless-shell (Chrome docs). Set a window size: headless Chrome otherwise starts small, and some sites show a mobile layout. Background on headless mode: what a headless browser is.

Troubleshooting headless Chrome:

By default Selenium 4 works with Chrome 75 and newer.

Selenium 4 has two methods, find_element() (the first match, or an error) and find_elements() (a list, empty if nothing matches), each with a By strategy.

from selenium.webdriver.common.by import By

quotes = driver.find_elements(By.CSS_SELECTOR, "div.quote")
for q in quotes:
    text = q.find_element(By.CSS_SELECTOR, "span.text").text
    author = q.find_element(By.CSS_SELECTOR, "small.author").text
    tags = [t.text for t in q.find_elements(By.CSS_SELECTOR, "a.tag")]
    print(author, "|", text[:60], tags)

next_link = driver.find_element(By.XPATH, "//li[@class='next']/a")
print(next_link.get_attribute("href"))

The old find_element_by_css_selector() style methods were removed in Selenium 4; code that uses them needs the By form above. Writing good selectors: HTML scraping with CSS selectors and XPath.

find_element() raises NoSuchElementException when nothing matches; find_elements() returns an empty list instead. Why that happens to elements you can see: the section after step 4.

driver.get() returns when the page's load event fires, but JavaScript often adds content later. Selenium offers two waits (Selenium: waiting strategies):

• Explicit wait (recommended): wait for one condition on one element, up to a timeout.
• Implicit wait: a global setting that makes every find_element() retry for up to N seconds. Do not mix it with explicit waits, because the timeouts add up in hard-to-predict ways.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 15)
wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.quote")))   # in the DOM
button = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "li.next a")))  # visible and enabled
button.click()
wait.until(EC.staleness_of(quotes[0]))   # the old content is gone: the next page has loaded

Common conditions: presence_of_element_located, visibility_of_element_located ("wait until visible"), element_to_be_clickable, text_to_be_present_in_element, url_contains. Avoid time.sleep() for waiting on content: it is either too short or wastes seconds on every page.

This is the error, as Chrome 155 and Selenium 4.51.0 printed it when our script looked for the quotes before the page's JavaScript had added them:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"div.quote"}
  (Session info: chrome=155.0.8059.79); For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors#nosuchelementexception

Selenium's own definition: "The element can not be found at the exact moment you attempted to locate it" (Selenium: understanding common errors). Seeing the element in your browser does not mean Selenium can see it at that moment, in that document. Check the five causes in this order:

The fixes for causes 1 to 4, each tested on a local page that put an element in an iframe, one in a shadow root and one 2 seconds after load (all three raised NoSuchElementException before the fix):

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# 1. Too early: wait up to 15 s for the element instead of failing at once
el = WebDriverWait(driver, 15).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, "div.late"))
)

# 2. Iframe: switch into the frame, read, switch back
driver.switch_to.frame(driver.find_element(By.CSS_SELECTOR, "iframe#prices"))
price = driver.find_element(By.CSS_SELECTOR, "span.price").text
driver.switch_to.default_content()

# 3. Shadow DOM: search inside the host's shadow root
host = driver.find_element(By.CSS_SELECTOR, "my-card")
name = host.shadow_root.find_element(By.CSS_SELECTOR, "p.name").text

# 4. New tab: switch to the newest window
driver.switch_to.window(driver.window_handles[-1])

driver.switch_to.frame() also takes a frame's name, id or index; EC.frame_to_be_available_and_switch_to_it(...) waits for the frame and switches in one step.

The errors people confuse with it

Infinite-scroll pages load more items when you reach the bottom. Scroll with JavaScript, wait for new items, and stop when the page height stops growing.

import time

driver.get("https://quotes.toscrape.com/scroll")
WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, "div.quote")))
last = 0
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(1.5)                                   # give the next batch time to load
    count = len(driver.find_elements(By.CSS_SELECTOR, "div.quote"))
    if count == last:
        break
    last = count
print(last, "quotes loaded")

Before you scroll at all, open the Network tab: infinite scroll almost always calls a JSON endpoint with a page or offset parameter. Calling that endpoint with Requests gets the same data without a browser.

driver.save_screenshot("page.png")                                  # the visible viewport
driver.find_element(By.CSS_SELECTOR, "div.quote").screenshot("first-quote.png")  # one element

For a full-page image in Chrome, set the window height to the page height before the screenshot (driver.set_window_size(1366, driver.execute_script("return document.body.scrollHeight"))); Firefox's driver has driver.get_full_page_screenshot_as_file("full.png").

Selenium's element lookups make one WebDriver call each, which adds up on large pages. A faster pattern: let the browser render, then parse driver.page_source with Beautiful Soup or lxml.

import csv
from bs4 import BeautifulSoup

soup = BeautifulSoup(driver.page_source, "html.parser")
rows = [{"text": q.select_one("span.text").get_text(strip=True),
         "author": q.select_one("small.author").get_text(strip=True)} for q in soup.select("div.quote")]
with open("quotes.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["text", "author"])
    w.writeheader()
    w.writerows(rows)
driver.quit()

Always call driver.quit(), also on errors (try / finally), or orphaned browsers will eat the server's memory.

Selenium is slow mostly because a browser is slow. What helps:

• Do not load images. options.add_experimental_option("prefs", {"profile.managed_default_content_settings.images": 2}) cuts load time and traffic.
• Use the eager page load strategy (options.page_load_strategy = "eager") so get() returns when the HTML is parsed, not when every image has loaded.
• Reuse one driver for many pages instead of starting a browser per URL.
• Parse page_source once instead of hundreds of find_element() calls.
• Run several drivers in parallel, a few per CPU core, or spread them with Selenium Grid.
• Use the browser only where needed: fetch listing pages with Requests and open Selenium only for the pages that need JavaScript.

SeleniumBase (4.56.4, MIT, October 2026) is a framework on top of Selenium for Python with built-in waits, simpler methods such as sb.open() and sb.click(), a test runner and extra browser modes. It suits teams who want Selenium's browser coverage with less boilerplate. Other ways to drive a browser from Python, including Playwright: Puppeteer, Selenium and Playwright alternatives and best headless browsers.

Chrome's --proxy-server flag takes an address but no username and password, so the short form works with IP allowlist authentication: add your server's IP to your account's allowlist, then pass the port you set up in the dashboard for it as <port>:

options.add_argument("--proxy-server=http://gate.proxyfield.io:<port>")   # IP allowlist auth

With the IP allowlist, the product, location and session are set per port in the dashboard. With a username and password, city targeting goes in the username, as in <login>-country-us-city-new_york for New York. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional. For username and password authentication, rotation without restarting the browser, Firefox and SOCKS5, follow the Selenium proxy setup guide. With ProxyField each request can leave from a new residential IP, or a sticky session keeps one IP from 30 s to 36 h for flows that must stay on one address, with country, region and city targeting. Residential proxies cost $2.95/GB for 1 GB and fall to $1.95/GB from 1,000 GB, with no subscription; traffic never expires, and all purchases are final. Blocking images, as above, also cuts the GB a browser scraper uses. Scrape public data at a reasonable pace (is web scraping legal), then start with residential proxies and the Easy Trial: 1 GB for $2.95.