How to Scale Web Scraping with Puppeteer Pool

By Marius Bernard

June 14, 2025

8 min read

So your Puppeteer scraper works great—until it doesn’t. What seemed like a rock-solid script for scraping 100 URLs starts falling apart when you throw 500,000 pages at it. Suddenly you’re battling memory leaks, browser crashes, proxy bans, and waiting days for something that should take hours.

I’ve been there. When I had to pull product data from over 2 million pages across multiple e-commerce platforms, my basic Puppeteer setup estimated 23 days to finish. That clearly wasn’t going to fly. By introducing Puppeteer Pool (via puppeteer-cluster) and layering on smart optimizations, we slashed that down to under 48 hours—on the exact same hardware.

In this guide, I’ll walk you through how to go from a simple scraper to a production-level system capable of handling scraping jobs at scale.

Why Traditional Puppeteer Breaks at Scale

Let’s start by talking about why your existing Puppeteer script collapses when scaled.

Memory Leaks: Each headless browser instance can eat up 300MB to over 1GB of RAM. Without clean shutdowns, you’ll quickly be overrun with zombie processes.

Network Bottlenecks: Puppeteer is fine for small jobs, but it doesn’t play nice at scale. Once you start juggling multiple browser sessions, your system can easily max out on CPU and memory.

Detection and Blocking: Websites are built to detect non-human behavior. If you’re scraping at scale and not mimicking real user interaction, expect to get flagged—and blocked.

Resource Competition: Even background processes like prefetchers can cause havoc. When they fight for memory and cache, performance suffers across the board.

Step 1: Set Up Puppeteer-Cluster for Browser Pool Management

The first step to scaling efficiently? Offload the heavy lifting to puppeteer-cluster.

This handy library spins up a pool of browsers (or contexts) and handles browser lifecycle management for you. No more babysitting browser restarts or manual error checks.

First, install what you need:

npm install puppeteer puppeteer-cluster

Then, here’s a straightforward example that’s miles ahead of a single-browser setup:

const { Cluster } = require('puppeteer-cluster');

(async () => {
  // Create a cluster with browser context isolation
  const cluster = await Cluster.launch({
    concurrency: Cluster.CONCURRENCY_CONTEXT,
    maxConcurrency: 4,
    monitor: true,
    puppeteerOptions: {
      headless: 'new',
      args: [
        '--no-sandbox',
        '--disable-setuid-sandbox',
        '--disable-dev-shm-usage',
        '--disable-gpu',
        '--no-first-run',
        '--no-zygote',
        '--single-process',
        '--disable-extensions'
      ]
    }
  });

  // Define your scraping task
  await cluster.task(async ({ page, data: url }) => {
    await page.goto(url, { 
      waitUntil: 'domcontentloaded',
      timeout: 30000 
    });
    
    // Your scraping logic here
    const data = await page.evaluate(() => {
      return {
        title: document.title,
        // Add your data extraction
      };
    });
    
    // Process or save the data
    console.log(`Scraped: ${url}`, data);
  });

  // Queue your URLs
  const urls = ['https://example.com/page1', 'https://example.com/page2'];
  urls.forEach(url => cluster.queue(url));

  await cluster.idle();
  await cluster.close();
})();

Pro Tip: Puppeteer Cluster isn’t just for scraping—you can use it for automated testing, site audits, even SEO experiments. Anything that benefits from parallelism can leverage it.

Step 2: Implement Smart Resource Management

When scraping hundreds of thousands of pages, memory becomes a limiting factor fast. That’s why it’s crucial to optimize how your system handles resources.

Here’s an advanced configuration that manages memory more efficiently:

const cluster = await Cluster.launch({
  concurrency: Cluster.CONCURRENCY_PAGE, // Reuse browser, create new pages
  maxConcurrency: 8,
  timeout: 60000,
  puppeteerOptions: {
    headless: 'new',
    args: [
      '--disable-dev-shm-usage',
      '--disable-gpu',
      '--disable-software-rasterizer',
      '--disable-background-timer-throttling',
      '--disable-backgrounding-occluded-windows',
      '--disable-renderer-backgrounding',
      '--disable-features=TranslateUI',
      '--disable-ipc-flooding-protection',
      '--max_old_space_size=4096' // Increase Node.js memory limit
    ]
  },
  skipDuplicateUrls: true,
  retryLimit: 3,
  retryDelay: 1000
});

// Implement resource blocking to save memory and bandwidth
await cluster.task(async ({ page, data }) => {
  // Block unnecessary resources
  await page.setRequestInterception(true);
  
  page.on('request', (request) => {
    const resourceType = request.resourceType();
    const blockResources = ['image', 'stylesheet', 'font', 'media'];
    
    if (blockResources.includes(resourceType)) {
      request.abort();
    } else {
      request.continue();
    }
  });
  
  // Navigate with minimal resource loading
  await page.goto(data.url, {
    waitUntil: 'networkidle0',
    timeout: 30000
  });
  
  // Force garbage collection after processing
  if (global.gc) {
    global.gc();
  }
});

By blocking unnecessary assets like images or stylesheets and forcing garbage collection when possible, you keep memory usage tight and page loads fast.

Step 3: Configure Optimal Concurrency Settings

Concurrency is where scale starts to get tricky. Choosing the right model depends on your hardware, network conditions, and scraping needs.

Here’s a breakdown of your options:

// Option 1: CONCURRENCY_CONTEXT (Recommended for most cases)
// One browser, multiple incognito contexts
const cluster = await Cluster.launch({
  concurrency: Cluster.CONCURRENCY_CONTEXT,
  maxConcurrency: Math.floor(os.cpus().length * 1.5), // 1.5x CPU cores
});

// Option 2: CONCURRENCY_PAGE (Best for memory-constrained environments)
// One browser, multiple pages
const cluster = await Cluster.launch({
  concurrency: Cluster.CONCURRENCY_PAGE,
  maxConcurrency: 20, // Can handle more pages than contexts
});

// Option 3: CONCURRENCY_BROWSER (For complete isolation)
// Multiple browsers
const cluster = await Cluster.launch({
  concurrency: Cluster.CONCURRENCY_BROWSER,
  maxConcurrency: 4, // Limited by system resources
});

// Dynamic concurrency based on system load
const dynamicConcurrency = await (async () => {
  const systemLoad = os.loadavg()[0];
  const freeMemory = os.freemem() / (1024 * 1024 * 1024); // GB
  
  if (freeMemory < 2 || systemLoad > 4) {
    return 2; // Low resources
  } else if (freeMemory < 4 || systemLoad > 2) {
    return 4; // Medium resources
  } else {
    return 8; // High resources
  }
})();

You can also dynamically adjust concurrency based on real-time system metrics like CPU load or available RAM. This allows your scraper to adapt on the fly.

Step 4: Add Request-Based Scraping (Beyond Browser Automation)

Not every page needs a browser.

If you're scraping static pages or sites that don’t rely on JavaScript for rendering content, switching to HTTP-based scraping can be significantly faster.

Check out this hybrid model:

const axios = require('axios');
const cheerio = require('cheerio');

// Intelligent routing: Use browsers only when necessary
await cluster.task(async ({ page, data }) => {
  // First, try with a simple HTTP request
  try {
    const response = await axios.get(data.url, {
      headers: {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
      },
      timeout: 10000
    });
    
    const $ = cheerio.load(response.data);
    
    // Check if content is fully loaded
    if ($('.product-data').length > 0) {
      // Extract data using Cheerio (much faster)
      return extractDataWithCheerio($);
    }
  } catch (error) {
    console.log('Falling back to browser for:', data.url);
  }
  
  // Fall back to Puppeteer for JavaScript-heavy pages
  await page.goto(data.url, { waitUntil: 'networkidle2' });
  return await extractDataWithPuppeteer(page);
});

// Separate queues for different scraping methods
const simplePages = urls.filter(url => !url.includes('dynamic'));
const complexPages = urls.filter(url => url.includes('dynamic'));

// Process simple pages with axios (100x faster)
await Promise.all(simplePages.map(url => processWithAxios(url)));

// Process complex pages with Puppeteer
complexPages.forEach(url => cluster.queue({ url, type: 'complex' }));

Using Axios and Cheerio for static content means fewer resources, quicker response times, and a much smaller system footprint. Save full browser sessions for pages that truly need them.

Step 5: Build Robust Error Handling and Retry Logic

At scale, even a small error rate adds up. A 0.1% failure on a million pages? That’s 1,000 missed data points. You can’t afford that.

This is where structured retries, proper logging, and error isolation come into play.

Here’s a solid foundation for error handling:

const winston = require('winston');
const Redis = require('ioredis');

// Set up structured logging
const logger = winston.createLogger({
  level: 'info',
  format: winston.format.json(),
  transports: [
    new winston.transports.File({ filename: 'scraping-errors.log', level: 'error' }),
    new winston.transports.File({ filename: 'scraping-combined.log' })
  ]
});

// Use Redis for distributed job management
const redis = new Redis();

cluster.on('taskerror', async (err, data, willRetry) => {
  logger.error('Scraping error', {
    url: data.url,
    error: err.message,
    willRetry,
    timestamp: new Date().toISOString()
  });
  
  if (!willRetry) {
    // Save failed URLs for later processing
    await redis.sadd('failed_urls', data.url);
  }
});

// Implement circuit breaker pattern
class CircuitBreaker {
  constructor(threshold = 5, timeout = 60000) {
    this.failures = 0;
    this.threshold = threshold;
    this.timeout = timeout;
    this.state = 'CLOSED';
    this.nextAttempt = Date.now();
  }
  
  async execute(fn) {
    if (this.state === 'OPEN') {
      if (Date.now() < this.nextAttempt) {
        throw new Error('Circuit breaker is OPEN');
      }
      this.state = 'HALF_OPEN';
    }
    
    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }
  
  onSuccess() {
    this.failures = 0;
    this.state = 'CLOSED';
  }
  
  onFailure() {
    this.failures++;
    if (this.failures >= this.threshold) {
      this.state = 'OPEN';
      this.nextAttempt = Date.now() + this.timeout;
    }
  }
}

// Use circuit breaker for each domain
const breakers = new Map();

await cluster.task(async ({ page, data }) => {
  const domain = new URL(data.url).hostname;
  
  if (!breakers.has(domain)) {
    breakers.set(domain, new CircuitBreaker());
  }
  
  const breaker = breakers.get(domain);
  
  return await breaker.execute(async () => {
    await page.goto(data.url);
    return await page.evaluate(() => {
      // Your scraping logic
    });
  });
});

The circuit breaker pattern is especially powerful—it prevents your system from hammering already failing domains, buying you time to recover or investigate.

Step 6: Monitor and Scale Your Infrastructure

Once your scraper is humming at scale, visibility becomes mission-critical. You need to know what’s working, what’s failing, and where your bottlenecks are.

Prometheus and custom metrics let you track all that in real-time:

const prometheus = require('prom-client');

// Set up Prometheus metrics
const scrapingDuration = new prometheus.Histogram({
  name: 'scraping_duration_seconds',
  help: 'Duration of scraping operations',
  labelNames: ['status', 'domain']
});

const activeScrapers = new prometheus.Gauge({
  name: 'active_scrapers',
  help: 'Number of active scraping operations'
});

const memoryUsage = new prometheus.Gauge({
  name: 'scraper_memory_usage_bytes',
  help: 'Memory usage of scraper process'
});

// Track metrics in your scraping task
await cluster.task(async ({ page, data }) => {
  const startTime = Date.now();
  const domain = new URL(data.url).hostname;
  
  activeScrapers.inc();
  
  try {
    const result = await scrapePage(page, data);
    
    scrapingDuration
      .labels('success', domain)
      .observe((Date.now() - startTime) / 1000);
    
    return result;
  } catch (error) {
    scrapingDuration
      .labels('error', domain)
      .observe((Date.now() - startTime) / 1000);
    
    throw error;
  } finally {
    activeScrapers.dec();
    
    // Update memory usage
    const usage = process.memoryUsage();
    memoryUsage.set(usage.heapUsed);
  }
});

// Expose metrics endpoint
const express = require('express');
const app = express();

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', prometheus.register.contentType);
  res.end(await prometheus.register.metrics());
});

app.listen(9090);

With proper monitoring in place, you’ll be able to fine-tune performance, quickly diagnose failures, and scale responsibly without flying blind.

Step 7: Advanced Optimization Techniques

1. Implement Intelligent Proxy Rotation

Proxy bans are inevitable at scale. An intelligent proxy rotator helps you dodge IP blocks and keep scraping uninterrupted.

Here’s how to set that up:

const ProxyChain = require('proxy-chain');

class ProxyRotator {
  constructor(proxies) {
    this.proxies = proxies;
    this.currentIndex = 0;
    this.failCounts = new Map();
  }
  
  async getNext() {
    // Skip failed proxies
    let attempts = 0;
    while (attempts < this.proxies.length) {
      const proxy = this.proxies[this.currentIndex];
      const failures = this.failCounts.get(proxy) || 0;
      
      if (failures < 3) {
        this.currentIndex = (this.currentIndex + 1) % this.proxies.length;
        return proxy;
      }
      
      this.currentIndex = (this.currentIndex + 1) % this.proxies.length;
      attempts++;
    }
    
    throw new Error('All proxies have failed');
  }
  
  reportFailure(proxy) {
    const current = this.failCounts.get(proxy) || 0;
    this.failCounts.set(proxy, current + 1);
  }
  
  reportSuccess(proxy) {
    this.failCounts.set(proxy, 0);
  }
}

const proxyRotator = new ProxyRotator([
  'http://proxy1.com:8080',
  'http://proxy2.com:8080',
  // Add more proxies
]);

// Use proxy in cluster
await cluster.task(async ({ page, data }) => {
  const proxy = await proxyRotator.getNext();
  
  try {
    const proxyUrl = await ProxyChain.anonymizeProxy(proxy);
    
    await page.authenticate({
      username: 'your-username',
      password: 'your-password'
    });
    
    await page.goto(data.url);
    proxyRotator.reportSuccess(proxy);
    
    // Continue scraping...
  } catch (error) {
    proxyRotator.reportFailure(proxy);
    throw error;
  }
});

It monitors proxy health and rotates away from unreliable endpoints without manual intervention.

2. Implement Session Persistence

Some sites track user sessions. Saving cookies lets you simulate continuity between requests and avoid unnecessary blocks or captchas.

Here’s a simple way to persist sessions using Redis:

// Save and reuse cookies across sessions
const cookieManager = {
  async save(page, domain) {
    const cookies = await page.cookies();
    await redis.set(`cookies:${domain}`, JSON.stringify(cookies), 'EX', 3600);
  },
  
  async load(page, domain) {
    const cookiesStr = await redis.get(`cookies:${domain}`);
    if (cookiesStr) {
      const cookies = JSON.parse(cookiesStr);
      await page.setCookie(...cookies);
    }
  }
};

await cluster.task(async ({ page, data }) => {
  const domain = new URL(data.url).hostname;
  
  // Load existing session
  await cookieManager.load(page, domain);
  
  await page.goto(data.url);
  
  // Save session for reuse
  await cookieManager.save(page, domain);
});

This keeps your scraper looking more human—without the complexity of re-authenticating constantly.

3. Implement GPU Acceleration (When Needed)

Most scrapers skip GPU usage—but in some cases, enabling GPU rendering can speed up visual content processing and reduce detection risk.

Here’s a config tweak for that:

const cluster = await Cluster.launch({
  puppeteerOptions: {
    headless: false, // Required for GPU
    args: [
      '--enable-unsafe-webgpu',
      '--enable-features=Vulkan',
      '--use-gl=desktop',
      '--enable-gpu-rasterization',
      '--enable-oop-rasterization'
    ]
  }
});

Not every project will benefit, but it’s worth testing for image-heavy or media-rich targets.

Final Thoughts

Scaling a Puppeteer scraper from a few hundred to millions of pages is a different game entirely. You’re no longer just writing scraping logic—you’re building infrastructure.

The key is to think systemically:

Use browser pools with Puppeteer Cluster
Optimize memory and network usage
Balance concurrency based on system load
Split traffic between HTTP and browser-based scrapers
Add robust logging and circuit breakers
Monitor performance with real-time metrics

By doing all this, we cut scraping time from weeks to under 48 hours—and achieved a 99.8% success rate.

Article by

Marius Bernard

Marius Bernard is a Product Advisor, Technical SEO, & Brand Ambassador at Roundproxies. He was the lead author for the SEO chapter of the 2024 Web and a reviewer for the 2023 SEO chapter.

Marius Bernard

Marius Bernard is a Product Advisor, Technical SEO, & Brand Ambassador at Roundproxies. He was the lead author for the SEO chapter of the 2024 Web and a reviewer for the 2023 SEO chapter.

Get the best
proxies out there

Get Proxies now