How to Scrape Images From a Website

# gallery-dl: install, then download a supported gallery into ./gallery-dl/
python -m pip install -U gallery-dl
gallery-dl "https://<supported-site>/<gallery-url>"
# wget: every JPG, PNG, WebP and GIF the page links to, into the current folder (no JavaScript)
wget -r -l 1 -nd -A jpg,jpeg,png,webp,gif "https://example.com/gallery"
# images on another host (a CDN): add -H -D cdn.example.com

To see whether a page loads images on scroll, open DevTools, go to the Network panel, filter by Img and scroll: if the list keeps growing, you need a tool from a "Yes" row.

Image scraping is collecting image files, or their URLs and details, from web pages automatically: product photos for a catalog, listing photos for price research, pictures for a training dataset, or every image on your own site for an audit. It has two jobs, and tools differ in how well they do each:

An image downloader does the second job for URLs you give it; an image scraper does both, and usually also keeps the context: the page URL, the alt text, the product or listing the image belongs to.

The <img src> attribute is only the first place to look. Modern pages spread the real image URL over several attributes and files.

Responsive images matter most for quality: srcset lists the same picture at several widths, and the browser picks one for your screen. An image scraper that keeps src alone often saves the smallest version. Per the HTML standard on srcset, each candidate is a URL followed by a width (480w) or a pixel density (2x).

With an auto-scroll extension (Download All Images in Chrome):

1. Install the extension from the Chrome Web Store and open the page with the images.
2. Run Deep Scan. The extension scrolls the page to the end, so lazy-loaded images and infinite-scroll galleries load, and lists every image it found.
3. Filter out icons and thumbnails: set a minimum width and height, pick the types you want (JPG, PNG, GIF, WebP, SVG) or the orientation, and exclude URLs that contain words such as logo or sprite.
4. Select the images and download them as single files or as one ZIP, with a filename pattern if you need one.

Expect to get what your browser rendered, at the size the page loaded. An extension without a scroll step (or "Save page as") saves only the images loaded at that moment, so on lazy-loading pages scroll to the bottom by hand first, or the downloads stop at the first screen.

With DevTools: open the Network panel, filter by Img, reload and scroll. Every image the page loads is listed with its URL, size and type; right-click to copy the URLs.

Extensions use your browser's proxy setting; see the Chrome proxy guide to change the IP. None of these scale past a few pages, and none keep the link between an image and the product or listing it belongs to.

The script below fetches a page, collects image URLs from src, srcset, lazy-load attributes and inline CSS, and downloads each one once. It needs pip install requests beautifulsoup4. Replace the start URL with a page you are allowed to scrape.

import hashlib, pathlib, re
from urllib.parse import urljoin, urlparse
import requests
from bs4 import BeautifulSoup

PAGE = "https://example.com/gallery"
OUT = pathlib.Path("images"); OUT.mkdir(exist_ok=True)
s = requests.Session()
s.headers["User-Agent"] = "Mozilla/5.0 (compatible; image-audit/1.0)"

def widest(srcset):
    best, best_w = None, -1
    for part in srcset.split(","):
        bits = part.strip().split()
        if not bits:
            continue
        w = int(bits[1][:-1]) if len(bits) > 1 and bits[1].endswith("w") else 0
        if w > best_w:
            best, best_w = bits[0], w
    return best

def image_urls(html, base):
    soup = BeautifulSoup(html, "html.parser")
    urls = set()
    for img in soup.select("img, source"):
        for attr in ("data-src", "data-lazy-src", "data-original", "src"):
            if img.get(attr) and not img[attr].startswith("data:"):
                urls.add(urljoin(base, img[attr])); break
        if img.get("srcset") and widest(img["srcset"]):
            urls.add(urljoin(base, widest(img["srcset"])))
    for tag in soup.select("[style*=url]"):
        for u in re.findall(r"url\(['\"]?([^'\")]+)", tag["style"]):
            urls.add(urljoin(base, u))
    og = soup.select_one('meta[property="og:image"]')
    if og and og.get("content"):
        urls.add(urljoin(base, og["content"]))
    return urls

seen = set()
for url in image_urls(s.get(PAGE, timeout=30).text, PAGE):
    r = s.get(url, timeout=30, headers={"Referer": PAGE})
    if r.status_code != 200 or not r.headers.get("Content-Type", "").startswith("image/"):
        continue                                   # skip error pages saved as .jpg
    digest = hashlib.sha256(r.content).hexdigest()
    if digest in seen:
        continue                                   # same file under another URL
    seen.add(digest)
    ext = r.headers["Content-Type"].split("/")[1].split(";")[0].replace("jpeg", "jpg")
    name = pathlib.Path(urlparse(url).path).stem or digest[:12]
    (OUT / f"{name}-{digest[:8]}.{ext}").write_bytes(r.content)

Three details do most of the work:

• The Content-Type check. A blocked or missing image often comes back as an HTML error page with a 200 or 403 status. Saving it as photo.jpg gives you a broken file; checking for image/ skips it.
• The hash. The same picture often sits under several URLs (sizes, CDN parameters). Hashing the bytes keeps one copy.
• The Referer header. Some sites refuse image requests that do not come from their own pages (hotlink protection). Sending the page URL as Referer is what a browser does.

If the image URLs are not in the page source, a script adds them after the page loads, often as you scroll. Two ways to get them:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={"server": "http://gate.proxyfield.io:7000",
                                       "username": "<login>-country-us-city-new_york", "password": "<password>"})
    page = browser.new_page()
    page.goto("https://example.com/gallery", wait_until="networkidle")
    last = 0
    while True:
        page.mouse.wheel(0, 4000)
        page.wait_for_timeout(1500)
        count = page.locator("img").count()
        if count == last:
            break
        last = count
    urls = page.eval_on_selector_all("img", "els => els.map(e => e.currentSrc || e.src)")
    browser.close()
print(len(urls), "images")

currentSrc returns the candidate the browser actually chose from srcset. Download the URLs with the Requests loop above (with its Content-Type and hash checks): a plain HTTP download is faster than saving files through the browser. More patterns for script-built pages: how to scrape dynamic web pages.

Thousands of images from many pages need a pipeline, not a loop:

• Separate finding from downloading. Write the image URLs, with their page URL and alt text, to a file or table first; download from that list. A crash then costs you nothing already done.
• Download in parallel, politely. Start low (about 5 parallel downloads per site) and raise the number while responses stay clean; image CDNs often accept more than the HTML server does. Back off on 429 responses. The patterns are in how to make web scraping faster.
• Resume. Skip URLs whose file already exists, and keep a list of failures to retry later.
• Store by hash. Name files by their hash (or keep the hash in your table) so duplicates never take space twice.
• Keep the context. An image without its product, listing or page is hard to use later.

Images are the heavy part of a page, so estimate traffic from the files, not the page count. A worked example: 10,000 images at an average of 200 KB each come to about 2 GB; with ProxyField residential proxies at the 1-9 GB rate of $2.95/GB, that is about $6 before blockchain network fees, and the HTML pages add a little more. Ways to cut it:

• Download each image once (the hash check above) and never re-download files you already have.
• Pick the size you need from srcset instead of always the widest.
• When you render pages only to find URLs, block image downloads in the browser and fetch the files afterwards from your list.

Full pricing by volume: residential and mobile proxy pricing. The traffic never expires, so a large one-off download can use the same balance as next month's run.

Image servers block for the same reasons page servers do: too many requests from one IP, requests that do not look like a browser, and missing headers.

1. Send browser-like headers, including User-Agent, Accept: image/avif,image/webp,*/* and the Referer of the page the image is on.
2. Pace per host. Keep a delay per domain, and honor Retry-After on 429 responses.
3. Rotate IPs on large jobs. With a rotating gateway, each download can leave from a new IP; use a sticky session when the site ties images to the session that loaded the page.
4. Respect the site's limits. If a site clearly refuses automated access, stop; the questions to ask first are in is web scraping legal.

ProxyField gives you the second half of that list: a new IP per request or sticky sessions from 30 seconds to 36 hours, country, region and city targeting, and unlimited concurrent threads, on residential proxies with 412M+ residential IPs in 195+ countries. The gateway is gate.proxyfield.io:7000; <login>-country-us-city-new_york asks for an IP in New York. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional.

Most photos on the web are protected by copyright whether or not they carry a notice: in the US, protection starts when the work is fixed, and a notice has not been required since 1989 (U.S. Copyright Office, Circular 1). Downloading images to analyze them, check your own listings or audit a site is a different question from republishing them. Read the site's terms, keep the source URL with every file, and get permission before you publish someone else's images. The wider picture is in is web scraping legal. This is not legal advice.

Downloading images at volume? ProxyField residential proxies: pay per GB from $1.95/GB, no subscription, start with the Easy Trial (1 GB). Traffic never expires, and all purchases are final.