Image Scraper in Python

pip install requests beautifulsoup4 lxml pillow

Requests 2.34.2, Beautiful Soup 4.15.0, lxml 6.1.3 and Pillow 13.0.0 were the current releases on PyPI on 2026-10-23. Pillow is optional; it checks that a saved file is a real image. Add playwright for pages that need a browser.

This guide is the Python version. For no-code tools, CSS background images and the copyright side of reusing images, see how to scrape images from a website.

from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

PAGE = "https://www.scrapethissite.com/pages/frames/?frame=i"
s = requests.Session()
s.headers["User-Agent"] = "image-scraper-tutorial/1.0 (contact: [email protected])"

soup = BeautifulSoup(s.get(PAGE, timeout=30).text, "lxml")
urls = []
for img in soup.select("img"):
    if img.get("width") == "1" and img.get("height") == "1":
        continue                                            # tracking pixel
    src = img.get("data-src") or img.get("src")             # lazy-loaded images keep the URL in data-src
    if img.get("srcset"):
        src = img["srcset"].split(",")[-1].split()[0]       # the last candidate is usually the largest
    if src and not src.startswith("data:"):
        urls.append(urljoin(PAGE, src))                     # make relative URLs absolute
urls = list(dict.fromkeys(urls))                            # drop repeats, keep order

What each line handles:

• urljoin turns /static/images/logo.png and //cdn.example.com/a.jpg into full URLs.
• data-src (sometimes data-lazy-src or data-original) holds the real URL on lazy-loaded images, while src is a placeholder. Look at one image in DevTools to see which attribute your site uses.
• srcset lists the same image in several widths (a-400.jpg 400w, a-800.jpg 800w). Taking the last entry usually gets the largest; parse the w values if you need to be sure.
• data: URIs are images embedded in the HTML as Base64, usually tiny icons; decode them with base64 if you need them.
• 1x1 pixels are tracking images. On 2026-10-23 this practice page had 17 <img> tags: the site logo, 14 turtle photos and two 1x1 tracking pixels.

To keep only the photos, select them by their class instead of every img: here soup.select("img.turtle-image"). Selector basics are in the Beautiful Soup tutorial.

import mimetypes
import os
from urllib.parse import urlparse

EXT = {"image/webp": ".webp", "image/avif": ".avif"}    # not in every machine's MIME tables


def download(url, folder="images"):
    os.makedirs(folder, exist_ok=True)
    with s.get(url, stream=True, timeout=30) as r:
        ctype = r.headers.get("Content-Type", "").split(";")[0].strip()
        if r.status_code != 200 or not ctype.startswith("image/"):
            print("skipped", r.status_code, ctype, url)     # an error page, not an image
            return None
        ext = EXT.get(ctype) or mimetypes.guess_extension(ctype) or ".img"
        name = os.path.splitext(os.path.basename(urlparse(url).path))[0] or "image"
        path = os.path.join(folder, name + ext)
        with open(path, "wb") as f:
            for chunk in r.iter_content(chunk_size=65536):
                f.write(chunk)
    return path

• stream=True and iter_content write the file in chunks instead of holding a large image in memory.
• The Content-Type check is what keeps HTML error pages out of your image folder. In our run on 2026-10-23, the logo downloaded as a 128x128 PNG, and the first turtle photo, hosted on upload.wikimedia.org, came back as 403 text/plain with a note pointing to Wikimedia's robot policy. Without the check, that note would have been saved as a .jpg that no viewer can open. Images often live on a CDN or another domain with its own rules, so read those too.
• mimetypes.guess_extension gives the extension from the type the server sent (.jpg, .png), which is more reliable than the URL: many image URLs end in ?w=800 or have no extension. Its answer depends on the machine's MIME tables: on Python 3.12 with only its built-in table (no /etc/mime.types), image/webp comes back as None, so the small EXT map handles WebP and AVIF first.
• Names collide when two folders contain image.jpg. Add a short hash of the URL to the file name (hashlib.sha1(url.encode()).hexdigest()[:10]) if your source reuses names.

To confirm a file is a real image, open it with Pillow: Image.open(path).verify() raises an error on a broken or truncated file.

Downloads spend most of their time waiting for the network, so a small thread pool speeds them up without rewriting anything:

import time
from concurrent.futures import ThreadPoolExecutor


def polite_download(url):
    time.sleep(0.5)                 # spread the requests out
    return download(url)

with ThreadPoolExecutor(max_workers=4) as pool:
    saved = [p for p in pool.map(polite_download, urls) if p]
print(len(saved), "images saved")

Keep the worker count low on one site (4 to 8 is a reasonable start) and raise it only while responses stay fast and without errors. Requests' Session is widely used across threads for simple GETs like these, but its thread safety is not documented; if you want certainty, create one session per thread. For thousands of files, an async client (httpx or aiohttp) with a semaphore uses fewer resources; the pattern is in asynchronous web scraping in Python.

If Step 1 finds placeholders or nothing at all, the page inserts its images with JavaScript, often only when you scroll to them. Let a browser do that, then read the URLs the browser actually chose:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    page = p.chromium.launch().new_page()
    page.goto("https://example.com/gallery")
    for _ in range(10):                          # scroll to trigger lazy loading
        page.mouse.wheel(0, 3000)
        page.wait_for_timeout(1000)
    urls = page.eval_on_selector_all(
        "img", "imgs => imgs.map(i => i.currentSrc || i.src).filter(Boolean)")

currentSrc is the file the browser picked from srcset for the current window size. Download the URLs with the Requests function above; a plain HTTP download is faster than saving images through the browser. Selenium works the same way: driver.execute_script("window.scrollTo(0, document.body.scrollHeight)") in a loop, then read currentSrc from each element. More on browser scraping in Python dynamic web scraping.

Two shortcuts before you reach for a browser: the image URLs may already be in the HTML inside a <script> block as JSON, or in a JSON request you can see in the Network tab. Either is faster than scrolling.

When you crawl many pages and download thousands of images, Scrapy's built-in ImagesPipeline handles the queue, the storage and skipping images it downloaded recently. It needs Pillow.

# settings.py
ITEM_PIPELINES = {"scrapy.pipelines.images.ImagesPipeline": 1}
IMAGES_STORE = "images"

# in the spider's parse callback
yield {"image_urls": [response.urljoin(u) for u in response.css("img::attr(src)").getall()]}

The pipeline downloads every URL in image_urls, stores each file under a name made from a hash of its URL, and writes the results into an images field of the item. It can also make thumbnails and skip images below a minimum size. Scrapy setup is in the Scrapy tutorial.

Images are the heaviest thing most scrapers download, so they decide your traffic. A rough guide: 10,000 images of about 200 KB each are about 2 GB, which is about $5.90 of traffic at ProxyField's 1-9 GB rate of $2.95/GB. Download only the size you need (pick the smaller srcset entry for thumbnails) and skip files you already have.

Many image hosts limit requests per IP. Slow down first; if a per-IP limit still stops a lawful download job, route the downloads through a proxy:

proxy = "http://<login>-country-us-city-new_york:<password>@gate.proxyfield.io:7000"
s.proxies.update({"http": proxy, "https": proxy})

gate.proxyfield.io:7000 is the ProxyField gateway; <login>-country-us-city-new_york asks for an exit IP in New York. Region, session and session time go in the same username: -region-<region>, -session-<id> and -ttl-<seconds> (30 to 129,600 seconds, 20 minutes by default), as in <login>-country-us-region-new_york-city-new_york-session-he9f2qga-ttl-600; every part after the login is optional. The gateway can give each request a new IP, or keep one IP for a sticky session from 30 s to 36 h. Python proxy options, including SOCKS5 and sessions, are on the Python proxy setup page.

Downloading an image does not give you the right to reuse it. Photos are usually protected by copyright, so check the license or get permission before you publish or sell what you collect; is web scraping legal? covers the rules. This is not legal advice.

Downloading images at scale? ProxyField residential proxies: 412M+ residential IPs in 195+ countries, a new IP on every request or sticky sessions up to 36 h, from $1.95/GB with no subscription; start with the Easy Trial (1 GB). Traffic never expires, and all purchases are final.