Web Scraping in Golang: 2025 Step-by-Step Guide

By Marius Bernard

July 16, 2025

9 min read

Golang is an open-source programming language crafted by Google that’s won over developers of all levels — beginners included. With its clean syntax, minimal keywords, and clear design, Go lets you build fast, reliable tools without unnecessary complexity.

One area where Go truly shines?

Web scraping. Thanks to its top-notch performance and native concurrency, Go makes it possible to extract huge amounts of data — and handle the anti-bot defenses modern sites throw your way.

In this guide, we’ll break down exactly how you can build robust, scalable scrapers with Go — from the basics of simple HTML parsing to handling dynamic, JavaScript-heavy pages.

What You’ll Learn

Here’s what you’ll take away by the end of this guide:

How to build scrapers using both net/http and the Colly framework
How to scrape at scale with Go’s powerful goroutines
Practical ways to bypass anti-bot measures and implement smart rate limiting
How to handle dynamic content that relies on JavaScript
Tips to squeeze the best performance out of your scrapers for massive operations

Why Golang Excels at Web Scraping

Being a compiled language gives Go a clear performance edge over interpreted alternatives. Here’s what makes it stand out:

Performance: Go’s compiled code runs 5–10 times faster than Python when you’re doing heavy CPU parsing. If you’re processing millions of pages, that speed boost isn’t just nice — it’s crucial.
Native Concurrency: Goroutines and Channels are baked right into the language, so you can handle multiple scraping tasks at once without the headaches you get with Python’s GIL.
Memory Efficiency: Goroutines are lightweight (around ~2KB each). That means you can spin up thousands of concurrent scrapers without grinding your server to a halt.
Simple Deployment: Go compiles down to a single binary file. No messing with virtual environments or Node.js dependencies — just run it.

Step 1: Set Up Your Golang Scraping Environment

Before you dive in, make sure you’ve got Go installed (version 1.19+ is best). Kick things off with a fresh project folder and initialize your module:

mkdir web-scraper-go
cd web-scraper-go
go mod init github.com/yourusername/web-scraper-go

Next, create your main.go to get the ball rolling:

package main

import (
    "fmt"
    "log"
)

func main() {
    fmt.Println("Web Scraper initialized!")
}

Step 2: Master Request-Based Scraping (No Framework Needed)

A lot of tutorials push you straight into using Colly, but understanding how to scrape with Go’s standard library gives you total control. Here’s a straightforward example using only net/http and goquery:

package main

import (
    "fmt"
    "log"
    "net/http"
    "time"
    "github.com/PuerkitoBio/goquery"
)

// Custom HTTP client with timeout and headers
func createHTTPClient() *http.Client {
    return &http.Client{
        Timeout: 30 * time.Second,
        Transport: &http.Transport{
            MaxIdleConns:        100,
            MaxIdleConnsPerHost: 10,
        },
    }
}

// Scrape function with proper error handling
func scrapeWebsite(url string) error {
    client := createHTTPClient()
    
    // Create request with custom headers
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        return fmt.Errorf("creating request: %w", err)
    }
    
    // Set headers to avoid detection
    req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
    req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
    req.Header.Set("Accept-Language", "en-US,en;q=0.9")
    
    // Execute request
    resp, err := client.Do(req)
    if err != nil {
        return fmt.Errorf("executing request: %w", err)
    }
    defer resp.Body.Close()
    
    if resp.StatusCode != 200 {
        return fmt.Errorf("status code error: %d %s", resp.StatusCode, resp.Status)
    }
    
    // Parse HTML
    doc, err := goquery.NewDocumentFromReader(resp.Body)
    if err != nil {
        return fmt.Errorf("parsing HTML: %w", err)
    }
    
    // Extract data using CSS selectors
    doc.Find(".product").Each(func(i int, s *goquery.Selection) {
        title := s.Find(".title").Text()
        price := s.Find(".price").Text()
        link, _ := s.Find("a").Attr("href")
        
        fmt.Printf("Product %d:\n", i+1)
        fmt.Printf("  Title: %s\n", title)
        fmt.Printf("  Price: %s\n", price)
        fmt.Printf("  Link: %s\n\n", link)
    })
    
    return nil
}

func main() {
    if err := scrapeWebsite("https://example.com/products"); err != nil {
        log.Fatal(err)
    }
}

Pro Tip: Add Retry Logic

In the real world, things fail — a lot. Here’s how you can build resilience into your scrapers with simple retries and exponential backoff:

func scrapeWithRetry(url string, maxRetries int) error {
    var err error
    for i := 0; i < maxRetries; i++ {
        err = scrapeWebsite(url)
        if err == nil {
            return nil
        }
        
        // Exponential backoff
        waitTime := time.Duration(math.Pow(2, float64(i))) * time.Second
        log.Printf("Attempt %d failed, waiting %v before retry: %v", i+1, waitTime, err)
        time.Sleep(waitTime)
    }
    return fmt.Errorf("all %d attempts failed: %w", maxRetries, err)
}

Step 3: Scale Up with Concurrent Scraping

One of Go’s standout strengths is how effortlessly it handles concurrency. This makes it perfect for scraping huge amounts of structured data. Here’s a practical example of a concurrent scraper using goroutines and channels:

package main

import (
    "fmt"
    "log"
    "sync"
    "time"
)

type ScrapedData struct {
    URL   string
    Title string
    Price string
    Error error
}

func concurrentScraper(urls []string, workers int) []ScrapedData {
    // Create channels
    urlChan := make(chan string, len(urls))
    resultChan := make(chan ScrapedData, len(urls))
    
    // Use WaitGroup to track goroutines
    var wg sync.WaitGroup
    
    // Start worker goroutines
    for i := 0; i < workers; i++ {
        wg.Add(1)
        go func(workerID int) {
            defer wg.Done()
            
            for url := range urlChan {
                log.Printf("Worker %d scraping: %s", workerID, url)
                
                // Implement rate limiting per worker
                time.Sleep(time.Millisecond * 500)
                
                // Scrape the URL (simplified for example)
                data := ScrapedData{URL: url}
                
                // Your actual scraping logic here
                err := scrapeAndExtract(url, &data)
                if err != nil {
                    data.Error = err
                }
                
                resultChan <- data
            }
        }(i)
    }
    
    // Send URLs to workers
    for _, url := range urls {
        urlChan <- url
    }
    close(urlChan)
    
    // Wait for all workers to finish
    go func() {
        wg.Wait()
        close(resultChan)
    }()
    
    // Collect results
    var results []ScrapedData
    for result := range resultChan {
        results = append(results, result)
    }
    
    return results
}

// Advanced rate limiting with token bucket
type RateLimiter struct {
    tokens    chan struct{}
    ticker    *time.Ticker
    maxTokens int
}

func NewRateLimiter(rps int) *RateLimiter {
    rl := &RateLimiter{
        tokens:    make(chan struct{}, rps),
        ticker:    time.NewTicker(time.Second / time.Duration(rps)),
        maxTokens: rps,
    }
    
    // Fill initial tokens
    for i := 0; i < rps; i++ {
        rl.tokens <- struct{}{}
    }
    
    // Refill tokens
    go func() {
        for range rl.ticker.C {
            select {
            case rl.tokens <- struct{}{}:
            default:
                // Channel full, skip
            }
        }
    }()
    
    return rl
}

func (rl *RateLimiter) Wait() {
    <-rl.tokens
}

Optimizing Concurrency

For applications that need to respect rate limits, limiting goroutines with channels or the semaphore package can be effective. Here's how to implement dynamic concurrency control:

// Adaptive concurrency based on response times
type AdaptiveScraper struct {
    minWorkers     int
    maxWorkers     int
    currentWorkers int
    avgResponseTime time.Duration
    mu             sync.Mutex
}

func (as *AdaptiveScraper) adjustWorkers() {
    as.mu.Lock()
    defer as.mu.Unlock()
    
    // Increase workers if response time is good
    if as.avgResponseTime < 2*time.Second && as.currentWorkers < as.maxWorkers {
        as.currentWorkers++
        log.Printf("Increasing workers to %d", as.currentWorkers)
    }
    
    // Decrease workers if response time is slow
    if as.avgResponseTime > 5*time.Second && as.currentWorkers > as.minWorkers {
        as.currentWorkers--
        log.Printf("Decreasing workers to %d", as.currentWorkers)
    }
}

Advanced Browser Techniques

For sites with complex interactions, implement human-like behavior:

// Simulate human-like mouse movements
func humanLikeInteraction(ctx context.Context) error {
    return chromedp.Run(ctx,
        // Random mouse movements
        chromedp.MouseEvent(100, 100, chromedp.MouseMoved),
        chromedp.Sleep(time.Millisecond * 300),
        chromedp.MouseEvent(250, 200, chromedp.MouseMoved),
        
        // Random delays between actions
        chromedp.Sleep(time.Duration(1000+rand.Intn(2000)) * time.Millisecond),
        
        // Simulate reading time
        chromedp.ActionFunc(func(ctx context.Context) error {
            readingTime := time.Duration(5+rand.Intn(10)) * time.Second
            log.Printf("Simulating reading for %v", readingTime)
            time.Sleep(readingTime)
            return nil
        }),
    )
}

Step 5: Bypass Anti-Bot Protection Like a Pro

Using Smart Proxies: Smart proxies are essential for bypassing Cloudflare as they can rotate IP addresses and manage requests effectively. Here's a comprehensive anti-detection strategy:

1. Implement Smart Header Rotation

type HeaderRotator struct {
    userAgents []string
    languages  []string
    accepts    []string
    mu         sync.Mutex
}

func NewHeaderRotator() *HeaderRotator {
    return &HeaderRotator{
        userAgents: []string{
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
        },
        languages: []string{
            "en-US,en;q=0.9",
            "en-GB,en;q=0.9",
            "en-US,en;q=0.8,es;q=0.6",
        },
        accepts: []string{
            "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
            "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        },
    }
}

func (hr *HeaderRotator) ApplyHeaders(req *http.Request) {
    hr.mu.Lock()
    defer hr.mu.Unlock()
    
    // Randomize headers
    req.Header.Set("User-Agent", hr.userAgents[rand.Intn(len(hr.userAgents))])
    req.Header.Set("Accept-Language", hr.languages[rand.Intn(len(hr.languages))])
    req.Header.Set("Accept", hr.accepts[rand.Intn(len(hr.accepts))])
    
    // Add more realistic headers
    req.Header.Set("Accept-Encoding", "gzip, deflate, br")
    req.Header.Set("DNT", "1")
    req.Header.Set("Connection", "keep-alive")
    req.Header.Set("Upgrade-Insecure-Requests", "1")
}

2. Advanced Proxy Rotation

type ProxyRotator struct {
    proxies    []string
    current    int
    mu         sync.Mutex
    httpClient *http.Client
}

func (pr *ProxyRotator) GetNextProxy() string {
    pr.mu.Lock()
    defer pr.mu.Unlock()
    
    proxy := pr.proxies[pr.current]
    pr.current = (pr.current + 1) % len(pr.proxies)
    return proxy
}

func (pr *ProxyRotator) CreateProxyClient(proxyURL string) (*http.Client, error) {
    proxy, err := url.Parse(proxyURL)
    if err != nil {
        return nil, err
    }
    
    transport := &http.Transport{
        Proxy: http.ProxyURL(proxy),
        DialContext: (&net.Dialer{
            Timeout:   30 * time.Second,
            KeepAlive: 30 * time.Second,
        }).DialContext,
        TLSHandshakeTimeout: 10 * time.Second,
    }
    
    return &http.Client{
        Transport: transport,
        Timeout:   60 * time.Second,
    }, nil
}

Save cookies to maintain sessions. Here's how to implement persistent sessions:

type SessionManager struct {
    sessions map[string]*http.CookieJar
    mu       sync.RWMutex
}

func NewSessionManager() *SessionManager {
    return &SessionManager{
        sessions: make(map[string]*http.CookieJar),
    }
}

func (sm *SessionManager) GetOrCreateSession(domain string) *http.CookieJar {
    sm.mu.Lock()
    defer sm.mu.Unlock()
    
    if jar, exists := sm.sessions[domain]; exists {
        return jar
    }
    
    jar, _ := cookiejar.New(nil)
    sm.sessions[domain] = jar
    return jar
}

// Save and load cookies for session persistence
func (sm *SessionManager) SaveCookies(domain string, filename string) error {
    sm.mu.RLock()
    jar, exists := sm.sessions[domain]
    sm.mu.RUnlock()
    
    if !exists {
        return fmt.Errorf("no session for domain: %s", domain)
    }
    
    // Serialize cookies to JSON
    cookies := jar.Cookies(&url.URL{Scheme: "https", Host: domain})
    data, err := json.Marshal(cookies)
    if err != nil {
        return err
    }
    
    return ioutil.WriteFile(filename, data, 0644)
}

4. TLS Fingerprint Randomization

// Randomize TLS fingerprint to avoid detection
func createStealthTransport() *http.Transport {
    return &http.Transport{
        TLSClientConfig: &tls.Config{
            // Randomize cipher suites
            CipherSuites: []uint16{
                tls.TLS_AES_128_GCM_SHA256,
                tls.TLS_AES_256_GCM_SHA384,
                tls.TLS_CHACHA20_POLY1305_SHA256,
                tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
                tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
            },
            // Randomize TLS version
            MinVersion: tls.VersionTLS12,
            MaxVersion: tls.VersionTLS13,
        },
    }
}

Step 6: Process and Store Your Scraped Data

Efficient data processing is crucial for large-scale scraping. Here's how to handle data extraction and storage:

Structured Data Extraction

type Product struct {
    ID          string    `json:"id"`
    Title       string    `json:"title"`
    Price       float64   `json:"price"`
    Description string    `json:"description"`
    ImageURL    string    `json:"image_url"`
    InStock     bool      `json:"in_stock"`
    ScrapedAt   time.Time `json:"scraped_at"`
}

// Extract and clean data
func extractProduct(selection *goquery.Selection) (*Product, error) {
    product := &Product{
        ScrapedAt: time.Now(),
    }
    
    // Extract with error handling
    product.Title = strings.TrimSpace(selection.Find(".title").Text())
    
    // Parse price with validation
    priceText := selection.Find(".price").Text()
    priceText = regexp.MustCompile(`[^\d.]`).ReplaceAllString(priceText, "")
    if price, err := strconv.ParseFloat(priceText, 64); err == nil {
        product.Price = price
    }
    
    // Extract availability
    stockText := selection.Find(".stock-status").Text()
    product.InStock = strings.Contains(strings.ToLower(stockText), "in stock")
    
    return product, nil
}

Concurrent Data Pipeline

type Product struct {
    ID          string    `json:"id"`
    Title       string    `json:"title"`
    Price       float64   `json:"price"`
    Description string    `json:"description"`
    ImageURL    string    `json:"image_url"`
    InStock     bool      `json:"in_stock"`
    ScrapedAt   time.Time `json:"scraped_at"`
}

// Extract and clean data
func extractProduct(selection *goquery.Selection) (*Product, error) {
    product := &Product{
        ScrapedAt: time.Now(),
    }
    
    // Extract with error handling
    product.Title = strings.TrimSpace(selection.Find(".title").Text())
    
    // Parse price with validation
    priceText := selection.Find(".price").Text()
    priceText = regexp.MustCompile(`[^\d.]`).ReplaceAllString(priceText, "")
    if price, err := strconv.ParseFloat(priceText, 64); err == nil {
        product.Price = price
    }
    
    // Extract availability
    stockText := selection.Find(".stock-status").Text()
    product.InStock = strings.Contains(strings.ToLower(stockText), "in stock")
    
    return product, nil
}

Concurrent Data Pipeline

// Pipeline for processing scraped data
type DataPipeline struct {
    scrapers   int
    processors int
    writers    int
}

func (dp *DataPipeline) Run(urls []string) error {
    // Create channels for pipeline stages
    urlChan := make(chan string, len(urls))
    rawDataChan := make(chan RawData, 100)
    processedChan := make(chan Product, 100)
    
    var wg sync.WaitGroup
    
    // Stage 1: Scraping
    for i := 0; i < dp.scrapers; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for url := range urlChan {
                if data, err := scrapeURL(url); err == nil {
                    rawDataChan <- data
                }
            }
        }()
    }
    
    // Stage 2: Processing
    for i := 0; i < dp.processors; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for raw := range rawDataChan {
                if product, err := processRawData(raw); err == nil {
                    processedChan <- product
                }
            }
        }()
    }
    
    // Stage 3: Storage
    for i := 0; i < dp.writers; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            batch := make([]Product, 0, 100)
            
            for product := range processedChan {
                batch = append(batch, product)
                
                // Write in batches for efficiency
                if len(batch) >= 100 {
                    if err := writeBatch(batch); err != nil {
                        log.Printf("Write error: %v", err)
                    }
                    batch = batch[:0]
                }
            }
            
            // Write remaining items
            if len(batch) > 0 {
                writeBatch(batch)
            }
        }()
    }
    
    // Feed URLs
    for _, url := range urls {
        urlChan <- url
    }
    close(urlChan)
    
    // Wait and close channels
    wg.Wait()
    close(rawDataChan)
    close(processedChan)
    
    return nil
}

Export Options

// Export to multiple formats
type DataExporter struct {
    data []Product
}

func (de *DataExporter) ToJSON(filename string) error {
    file, err := os.Create(filename)
    if err != nil {
        return err
    }
    defer file.Close()
    
    encoder := json.NewEncoder(file)
    encoder.SetIndent("", "  ")
    return encoder.Encode(de.data)
}

func (de *DataExporter) ToCSV(filename string) error {
    file, err := os.Create(filename)
    if err != nil {
        return err
    }
    defer file.Close()
    
    writer := csv.NewWriter(file)
    defer writer.Flush()
    
    // Write header
    header := []string{"ID", "Title", "Price", "In Stock", "Scraped At"}
    if err := writer.Write(header); err != nil {
        return err
    }
    
    // Write data
    for _, product := range de.data {
        record := []string{
            product.ID,
            product.Title,
            fmt.Sprintf("%.2f", product.Price),
            fmt.Sprintf("%t", product.InStock),
            product.ScrapedAt.Format(time.RFC3339),
        }
        if err := writer.Write(record); err != nil {
            return err
        }
    }
    
    return nil
}

Final Thoughts

By now, you know exactly how to build a production-ready Golang scraper that can:

Run multiple tasks at once, without breaking a sweat
Sidestep modern anti-bot protections like Cloudflare
Render JavaScript-heavy content with a headless browser
Scale up to handle millions of pages with minimal fuss

Remember: the secret sauce is Go’s blend of speed, simplicity, and concurrency. When you combine these with smart scraping techniques, you’re ready to tackle scraping projects at any scale.

Ready to put this into practice? Fire up your terminal, spin up your goroutines, and get scraping!

Article by

Marius Bernard

Marius Bernard is a Product Advisor, Technical SEO, & Brand Ambassador at Roundproxies. He was the lead author for the SEO chapter of the 2024 Web and a reviewer for the 2023 SEO chapter.

Marius Bernard

Marius Bernard is a Product Advisor, Technical SEO, & Brand Ambassador at Roundproxies. He was the lead author for the SEO chapter of the 2024 Web and a reviewer for the 2023 SEO chapter.

Get the best
proxies out there

Get Proxies now

What You’ll Learn

Why Golang Excels at Web Scraping

Step 1: Set Up Your Golang Scraping Environment

Step 2: Master Request-Based Scraping (No Framework Needed)

Step 3: Scale Up with Concurrent Scraping

Optimizing Concurrency

Advanced Browser Techniques

Step 5: Bypass Anti-Bot Protection Like a Pro

1. Implement Smart Header Rotation

2. Advanced Proxy Rotation

3. Cookie Management and Session Persistence

4. TLS Fingerprint Randomization

Step 6: Process and Store Your Scraped Data

Structured Data Extraction

Concurrent Data Pipeline

Concurrent Data Pipeline

Export Options

Final Thoughts

Marius Bernard