How to Scrape a Table From a Website

The full signature in pandas 3.1.0; every argument after io is keyword-only:

pandas.read_html(io, *, match='.+', flavor=None, header=None, index_col=None, skiprows=None,
                 attrs=None, parse_dates=False, thousands=',', encoding=None, decimal='.',
                 converters=None, na_values=None, keep_default_na=True, displayed_only=True,
                 extract_links=None, dtype_backend=<no_default>, storage_options=None)

The reference's notes, in short:

• A list or a failure: the function "will always return a list of DataFrame or it will fail", apart from rare cases, such as a single row whose <td> cells hold only whitespace.
• What it reads: <table> elements, their <tr> and <th> rows and the <td> cells in them. It "attempts to properly handle colspan and rowspan attributes", and builds the header from <thead>, or from body rows that hold only <th> cells.
• A parser is required: lxml, or Beautiful Soup with html5lib. With flavor=None it tries lxml first and falls back to Beautiful Soup with html5lib.
• lxml and https: the reference notes that "lxml only accepts the http, ftp and file url protocols". In our run an https:// URL still worked with lxml alone, because pandas downloads the page itself; if a URL fails, fetch it with Requests and pass the text in StringIO.
• Expect cleanup: "Expect to do some cleanup after you call this function", for example column names that come back as NaN with header=0. header is applied after skiprows.

What the reference does not give you is a timeout, retries or a proxy: read_html(url) has no parameter for them. A job that needs those fetches the page with Requests and passes the text in (Method 1 below).

Before you choose, right-click the table and pick Inspect. If you see <table>, <tr> and <td> tags, every method works. If the rows are <div>s styled to look like a table, use Beautiful Soup with CSS selectors. And if View Source shows no rows at all, the table is loaded by JavaScript; skip to the section on that.

pip install pandas lxml requests

from io import StringIO

import pandas as pd
import requests

url = "https://www.scrapethissite.com/pages/forms/"
html = requests.get(url, params={"per_page": 100}, headers={"User-Agent": "Mozilla/5.0"}, timeout=30).text
tables = pd.read_html(StringIO(html))          # one DataFrame per <table>
df = tables[0]
print(len(tables), df.shape)                   # 1 (100, 9)
df.to_csv("teams.csv", index=False)

On 2026-10-27 this returned one table of 100 rows and 9 columns (team, year, wins, losses and so on), with numbers already typed as integers and floats. Notes from that run:

• Wrap the HTML in StringIO. pandas 3.1.0 (current on PyPI on 2026-10-27; its package metadata requires Python 3.11 or later) treats a plain string as a file path: passing the HTML text directly failed with OSError: [Errno 36] File name too long.
• Why Requests and not read_html(url). Requests gives you a timeout, retries through a Session, and a proxy; read_html(url) has none of the three.

Every page of a paginated table

When the table spans several pages, loop the page parameter and stop when a page comes back with no rows:

import time

s = requests.Session()
s.headers["User-Agent"] = "Mozilla/5.0"
frames, page = [], 1
while True:
    r = s.get("https://www.scrapethissite.com/pages/forms/",
              params={"page_num": page, "per_page": 100}, timeout=30)
    df = pd.read_html(StringIO(r.text))[0]
    if df.empty:                       # past the last page: headers but no rows
        break
    frames.append(df)
    page += 1
    time.sleep(2)

teams = pd.concat(frames, ignore_index=True)
teams.to_csv("teams.csv", index=False)
print(teams.shape)                     # (582, 9)

It collected all 582 rows in 6 pages on 2026-10-27. Other paging patterns are in web scraping pagination.

Use Beautiful Soup when you need what read_html throws away (the URLs behind cell links, images, data- attributes), or when the "table" is built from <div>s.

import csv

import requests
from bs4 import BeautifulSoup

url = "https://www.scrapethissite.com/pages/forms/"
soup = BeautifulSoup(requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=30).text, "lxml")
table = soup.select_one("table.table")

headers = [th.get_text(strip=True) for th in table.select("tr th")]
rows = []
for tr in table.select("tr")[1:]:                      # skip the header row
    cells = [td.get_text(strip=True) for td in tr.select("td")]
    if cells:
        rows.append(cells)

with open("teams_page1.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(headers)
    writer.writerows(rows)
print(len(headers), "columns,", len(rows), "rows")      # 9 columns, 25 rows

To keep a link, read it alongside the text: td.select_one("a")["href"] when the cell has one. Merged cells need care: a cell with colspan="2" covers two columns and one with rowspan="2" covers two rows, so a naive loop shifts every later value one column to the left. Repeat the cell's value for its span, or use read_html, which expands spans for you. More selector techniques are in the Beautiful Soup tutorial.

Google Sheets: =IMPORTHTML("https://example.com/page", "table", 1) imports the first table on the page; change the last number for the second, third and so on. For a CSV, choose File > Download > Comma-separated values (.csv). It works on public pages whose table is in the server's HTML, and you can't set headers or a proxy. Google's limits for these functions, their sibling IMPORTXML and refresh behavior are covered in web scraping with Google Sheets; check them there before you build on a formula.

Excel: Data > Get Data > From Other Sources > From Web, paste the URL, and the Navigator lists every table it found; load one and refresh it later with one click, or save the sheet as CSV. Step-by-step instructions, and what to do when Power Query finds nothing, are in web scraping with Excel. For Excel macros, see VBA web scraping.

If View Source shows an empty <tbody> or a loading spinner, the rows arrive later from a script. That data almost always travels as JSON, which is easier to tabulate than HTML.

Open DevTools, go to Network > Fetch/XHR, reload the page and look for a response that holds the rows. On the practice page scrapethissite.com/pages/ajax-javascript/, clicking a year calls ?ajax=true&year=2015, which returns a JSON list of films:

films = pd.DataFrame(requests.get(
    "https://www.scrapethissite.com/pages/ajax-javascript/",
    params={"ajax": "true", "year": 2015}, timeout=30).json())
print(films.shape)                     # (16, 5): title, year, awards, nominations, best_picture

If you can't reproduce the request, render the page in Playwright or Selenium, wait for the rows, and pass the rendered HTML to read_html. Both routes are explained in scrape dynamic web pages.

Scraped tables rarely arrive analysis-ready. The usual fixes in pandas:

• Numbers stored as text because of symbols: df["Price"].str.replace(r"[$,]", "", regex=True).astype(float).
• Percentages: strip % and divide by 100.
• Footnote markers such as [1] or * in cells: df.replace(r"\[\d+\]", "", regex=True).
• Repeated header rows in the middle of long tables: drop rows where a column equals its own header.
• Empty cells come in as NaN (the "OT Losses" column above is empty for early seasons); decide whether that means zero or missing.