Web Scraping Pagination

Open the list in your browser, go to page 2, and watch two things: the address bar and the Network tab of DevTools. Together they tell you which of the four kinds you have.

Most sites mix them: a page-number URL behind a Next button, or a cursor API behind infinite scroll. Prefer the lowest layer you can reach. A URL you build yourself or a JSON endpoint is faster and more stable than clicking through a browser.

When every page links to the next one, the loop is: fetch the page, extract the items, find the Next link, turn it into an absolute URL, repeat until there is no link. The practice table at scrapethissite.com (582 hockey team rows, 25 per page) works this way:

import time
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

url = "https://www.scrapethissite.com/pages/forms/?page_num=1"
seen_urls, seen_rows, rows = set(), set(), []
s = requests.Session()
s.headers["User-Agent"] = "Mozilla/5.0"

while url and url not in seen_urls:
    seen_urls.add(url)
    soup = BeautifulSoup(s.get(url, timeout=30).text, "lxml")
    for tr in soup.select("tr.team"):
        row = [td.get_text(strip=True) for td in tr.select("td")]
        key = (row[0], row[1])              # team name + year
        if key not in seen_rows:
            seen_rows.add(key)
            rows.append(row)
    nxt = soup.select_one('a[aria-label="Next"]')
    url = urljoin(url, nxt["href"]) if nxt else None
    time.sleep(2)                           # be polite

print(len(seen_urls), "pages,", len(rows), "teams")

Run on 2026-10-23, it printed 24 pages, 582 teams. Two guards in it matter on real sites:

• The visited-URL set stops loops. On this practice site, the Next link on the unnumbered first page (/pages/forms/) points to ?page_num=1, which is the same page under a new URL. Starting from /pages/forms/ would fetch page 1 twice; starting from ?page_num=1 and keeping the set avoids it.
• The row key stops duplicates when two URLs return the same items. Use the item's own ID or URL when it has one.

urljoin matters because Next links are usually relative (?page_num=3, /page/3/). Selector choice is covered in HTML scraping.

When the page number is in the URL, you don't need the Next link at all. Build each URL yourself, and stop when a page comes back with no items. Many sites also let you ask for more items per page, which cuts the number of requests:

import time
import requests
from bs4 import BeautifulSoup

base = "https://www.scrapethissite.com/pages/forms/"
s = requests.Session()
s.headers["User-Agent"] = "Mozilla/5.0"
rows, page = [], 1
while True:
    r = s.get(base, params={"page_num": page, "per_page": 100}, timeout=30)
    r.raise_for_status()
    teams = BeautifulSoup(r.text, "lxml").select("tr.team")
    if not teams:            # an empty page means you are past the end
        break
    rows += [[td.get_text(strip=True) for td in tr.select("td")] for tr in teams]
    page += 1
    time.sleep(2)
print(len(rows), "teams")

With 100 rows per page, the same 582 rows took 6 requests instead of 24 (run on 2026-10-23). Note the stop rule: a page past the end of this site (?page_num=25 at 25 per page) returns status 200 with an empty table, not a 404. Many sites behave the same way, so test for "no items", not only for an error code. Other sites redirect to page 1 or repeat the last page, which the duplicate key catches.

Offset pagination is the same loop with a different parameter: ?offset=0&limit=50, then offset=50, offset=100, until the response is empty. If the first page shows a total ("582 results") or a last page number, you can compute every URL up front and fetch them concurrently; make web scraping faster shows how, and asynchronous web scraping in Python has the asyncio version.

Infinite scroll and "Load more" buttons are pagination hidden behind JavaScript: the page asks a server endpoint for the next batch and appends it. Find that request and you can usually skip the browser.

1. Open DevTools, go to the Network tab, filter by Fetch/XHR, and scroll or click "Load more".
2. Look for a request that returns the new items as JSON. Note how it pages: page=2, offset=40, cursor=... or after=....
3. Copy it ("Copy as cURL"), replay it in your code, and loop the paging parameter with the stop rules above.

The approach is covered in depth in scraping hidden APIs. When the endpoint needs tokens you can't reproduce, scroll a real browser instead and stop when the page stops growing:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    page = p.chromium.launch().new_page()
    page.goto("https://example.com/feed")
    last = 0
    while True:
        page.mouse.wheel(0, 5000)
        page.wait_for_timeout(1500)                    # give the next batch time to load
        count = page.locator("article").count()
        if count == last:                              # nothing new arrived: the end
            break
        last = count
    items = page.locator("article").all_inner_texts()

Replace the URL and the article selector with the real ones. For a "Load more" button, click it instead of scrolling and stop when the button disappears. More browser patterns, including waiting for the network response instead of a fixed delay, are in scrape dynamic web pages.

APIs that hold large or changing lists often page with a cursor: each response includes an opaque token, and you send it back to get the next batch. You cannot compute the next URL; you have to read it from the last response.

import requests

items, cursor = [], None
while True:
    params = {"limit": 100}
    if cursor:
        params["cursor"] = cursor
    data = requests.get("https://api.example.com/v1/items", params=params, timeout=30).json()
    items += data["items"]
    cursor = data.get("next_cursor")     # the field name differs per API
    if not cursor:
        break

Read the API's documentation for the field names (next, nextPageToken, after, has_more) and for how long a cursor stays valid. Some cursors expire after minutes, so a crawl that pauses too long has to start the list again.

The patterns are the same in every tool; only the syntax changes.

Scrapy removes duplicate requests for you (its default duplicate filter skips a URL it has already requested in the same run), which takes care of the visited-URL set. The full Scrapy walkthrough is the Scrapy tutorial; the Node.js version of the Next-link loop is in the Cheerio web scraping tutorial.

Pagination bugs rarely crash a scraper. They quietly skip or repeat rows. The usual causes:

• The list changes while you page through it. A new item at the top pushes every other item down by one, so the last item of page 1 shows up again on page 2. If the site offers a stable sort (oldest first, or by ID), use it, and de-duplicate by ID.
• The sort order is unstable. Items with the same sort value can swap places between requests. Add a second sort key if the site allows it, or accept duplicates and remove them by ID.
• A result cap. Many search pages stop at a fixed number of pages even when they report more results. Split the query into smaller slices (by category, price range or date) so that each slice fits under the cap.
• A failed page. One timeout in the middle of 500 pages leaves a gap. Retry failed pages a few times, log the page numbers that still fail, and save progress (the last page or cursor) so a crash doesn't send you back to page 1.

Count what you got against what the site says it has. If page 1 reports 582 results and you saved 575, something was skipped.

Deep pagination is where scrapers meet rate limits: hundreds of requests to the same path from one IP in a few minutes. Slow down first: add a delay, keep concurrency low, and respect the site's robots.txt and terms (see is web scraping legal?). A 429 response means you are going too fast; 429 Too Many Requests explains how to back off.

When a per-IP limit still stops a lawful crawl, route the requests through a proxy pool. Page-number and Next-link pages are independent requests, so each can leave from a new IP:

proxy = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
s.proxies.update({"http": proxy, "https": proxy})

gate.proxyfield.io:7000 is the ProxyField gateway, and <login>-country-us-city-new_york asks for an exit IP in New York. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional. The gateway can give each request a new IP, or keep one IP for a sticky session from 30 s to 36 h. Use a sticky session when the site ties the paging state to your session: a cursor issued to one IP and checked on the next request, or a "Load more" that relies on a session cookie. Sticky vs rotating proxies explains when each fits.

Pagination costs little traffic: at an assumed 150 KB per list page, a 2,000-page crawl uses about 0.3 GB, so the 1 GB Easy Trial ($2.95) covers it three times over. Pages rendered in a browser load scripts and images too and weigh several times more.

Paging through long lists? ProxyField residential proxies: 412M+ residential IPs in 195+ countries, a new IP on every request or sticky sessions up to 36 h, from $1.95/GB with no subscription; start with the Easy Trial (1 GB). Traffic never expires, and all purchases are final.