Web Scraping With Go: Colly and goquery

Yes, especially for large crawls. Go's goroutines make thousands of concurrent requests light on memory and simple to write, the standard library's net/http is production-grade, and a scraper compiles to a single static binary that is easy to run on a server or in a container. Python has more scraping tutorials and libraries and is quicker for one-off scripts; Go pays off when a scraper runs every day at volume. Choosing a language in general is covered in choosing a language for web scraping.

goquery parses a page you have fetched yourself, which keeps full control over the HTTP request:

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/PuerkitoBio/goquery"
)

func main() {
	client := &http.Client{Timeout: 30 * time.Second}
	req, _ := http.NewRequest("GET", "https://books.toscrape.com/", nil)
	req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; books-demo/1.0)")

	res, err := client.Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer res.Body.Close()
	if res.StatusCode != http.StatusOK {
		log.Fatalf("status %d", res.StatusCode)
	}

	doc, err := goquery.NewDocumentFromReader(res.Body)
	if err != nil {
		log.Fatal(err)
	}
	doc.Find("article.product_pod").Each(func(i int, s *goquery.Selection) {
		title, _ := s.Find("h3 a").Attr("title")
		price := s.Find(".price_color").Text()
		fmt.Printf("%s: %s\n", title, price)
	})
}

Find() takes a CSS selector, Each() loops over the matches, Attr() returns an attribute and whether it exists, and Text() the combined text. goquery does not support XPath; if you need XPath in Go, the antchfx/htmlquery package provides it.

Colly wraps fetching and parsing in a collector: you register callbacks for the elements you care about, and call Visit(). This program collects every book's title, price and stock status from all 50 pages of a practice catalog, two pages at a time with a random delay, and writes a CSV file.

package main

import (
	"encoding/csv"
	"log"
	"os"
	"strings"
	"sync"
	"time"

	"github.com/gocolly/colly/v2"
)

func main() {
	f, err := os.Create("books.csv")
	if err != nil {
		log.Fatal(err)
	}
	defer f.Close()
	w := csv.NewWriter(f)
	defer w.Flush()
	w.Write([]string{"title", "price", "in_stock"})
	var mu sync.Mutex // async callbacks run in several goroutines

	c := colly.NewCollector(
		colly.AllowedDomains("books.toscrape.com"),
		colly.Async(true),
	)
	c.SetRequestTimeout(30 * time.Second)
	c.Limit(&colly.LimitRule{DomainGlob: "*", Parallelism: 2, RandomDelay: time.Second})

	// one row per book on a list page
	c.OnHTML("article.product_pod", func(e *colly.HTMLElement) {
		inStock := strings.Contains(e.ChildText(".availability"), "In stock")
		mu.Lock()
		w.Write([]string{e.ChildAttr("h3 a", "title"), e.ChildText(".price_color"), yesNo(inStock)})
		mu.Unlock()
	})
	// follow the pagination
	c.OnHTML("li.next a[href]", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))     // relative links are resolved for you
	})
	c.OnError(func(r *colly.Response, err error) {
		log.Printf("error %d on %s: %v", r.StatusCode, r.Request.URL, err)
	})

	c.Visit("https://books.toscrape.com/catalogue/page-1.html")
	c.Wait() // wait for the async requests to finish
}

func yesNo(b bool) string {
	if b {
		return "yes"
	}
	return "no"
}

How it works:

1. colly.NewCollector() creates the collector; AllowedDomains keeps it on one site and Async(true) lets requests run in parallel.
2. Limit() sets the politeness rule: at most 2 requests at a time to the site, with up to 1 s of random delay.
3. OnHTML(selector, fn) runs fn for every element that matches, with helpers such as ChildText and ChildAttr.
4. e.Request.Visit() queues a link; Colly skips URLs it has already visited.
5. OnError receives failed requests; Wait() blocks until every queued request is done.

In async mode the callbacks run in several goroutines, so the sync.Mutex lets only one of them write to the CSV file at a time; sending rows to a channel and writing them from one goroutine works too.

Turning the scraper into a crawler

To crawl a whole site, follow every link instead of only the "next" link, and cap the depth:

c := colly.NewCollector(colly.AllowedDomains("books.toscrape.com"), colly.MaxDepth(3))
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
	e.Request.Visit(e.Attr("href"))
})
c.OnResponse(func(r *colly.Response) {
	log.Println("visited", r.Request.URL)
})
c.Visit("https://books.toscrape.com/")

Add colly.CacheDir("./cache") to keep downloaded pages on disk while you develop, so re-running the parser does not hit the site again. The general method is in how to crawl a website.

They are not alternatives so much as layers: Colly uses goquery internally, and e.DOM in a callback is a goquery selection. Use goquery alone when you fetch pages yourself (from an API client, a file, a headless browser) and only need parsing. Use Colly when you want the crawl handled for you: queues, deduplication, domain limits, parallelism, retries you write in OnError, cookies and proxy switching.

Colly and goquery see only the HTML the server sends. When the data is rendered by scripts, first look for its JSON request in the browser's DevTools (Network tab) and call it with net/http. If there is none, chromedp runs a headless Chrome:

ctx, cancel := chromedp.NewContext(context.Background())
defer cancel()
var quotes []string
err := chromedp.Run(ctx,
	chromedp.Navigate("https://quotes.toscrape.com/js/"),
	chromedp.WaitVisible(".quote", chromedp.ByQuery),
	chromedp.Evaluate(`[...document.querySelectorAll('.quote .text')].map(e => e.textContent)`, &quotes),
)

The full chromedp and rod examples, and when a browser is worth its cost, are in headless browser scraping.

gospider is a command-line crawler written in Go that lists the URLs it finds on a site: links, JavaScript files, sitemap and robots.txt entries. It is handy for mapping a site before you write a scraper (gospider on GitHub).

go install github.com/jaeles-project/gospider@latest
gospider -s "https://books.toscrape.com/" -d 2 -c 5 -k 1 -o output

-s is the start URL, -d the depth, -c the concurrent requests, -k the delay in seconds and -o the output folder; -p sets a proxy. Its last commit is from March 2024, so check that it still fits your needs, and use Colly when you need the page contents rather than the URL list.