Web Scraping in Rust: Complete 2026 Guide

Rust web scraping earns its keep when your scraper is the bottleneck, not the network. A script that pulls a few hundred pages an hour has no business being rewritten in Rust. A crawler that fetches millions of URLs, parses HTML on every one, and keeps a large queue in memory is a different job. That is where compiled code and real threads without a Global Interpreter Lock start paying for themselves.

The cost is the first week. Ownership, borrowing, and lifetime annotations reject code that any other language would happily run, and the compiler will feel like an adversary until the patterns click. What you get back is a class of bug that stops existing: no None showing up where you expected a string three hours into a crawl, no worker thread dying quietly and taking a batch of results with it. If it compiles, it usually runs until the site blocks you.

The crate ecosystem is smaller than Python's, but the pieces you need are settled. reqwest handles HTTP, scraper gives you CSS selector queries over parsed HTML, tokio drives async concurrency, and a headless browser binding takes over when the content only appears after JavaScript runs.

This guide goes from a first request through to the parts most tutorials skip: running hundreds of requests concurrently without hammering a server, structuring extraction so a site redesign breaks one selector instead of the whole scraper, handling the fingerprinting and rate limits that get scrapers banned, and keeping memory flat on long crawls. My honest take: reach for Python on a one-off. Reach for Rust when the thing has to run every day.

Why choose Rust for web scraping?

Rust is the wrong default and the right specialist. For a one-off scrape of a few hundred pages, a Python script will be finished and running before a fresh Cargo project completes its first clean build. Rust web scraping earns its keep at the other end of the scale, when the crawl stops being a script and turns into infrastructure that runs on a schedule, feeds a pipeline, and costs real machine hours.

What Rust actually buys you

• No global interpreter lock. Threads run in parallel for real, so CPU-bound work like HTML parsing and text extraction spreads across cores instead of queuing behind one interpreter.
• Cheap concurrency. On Tokio, an in-flight request costs a task and a socket rather than an OS thread, so one machine can hold thousands of connections open while remote servers take their time answering.
• No garbage collector. Memory stays flat and predictable over a crawl that runs for hours, which matters when you're pinned to a small VM and a stop-the-world pause would stall the fetch loop.
• Shared state checked at compile time. A data race between worker tasks is a build error, not a corrupted output file you notice three days later.

Deployment is the benefit nobody mentions. A release build produces one binary you can drop into a minimal container: no interpreter, no virtualenv, no dependency resolution happening on the production host at 3am.

What it costs

• Iteration speed. Change a CSS selector and you wait for a rebuild. Python hands you a REPL and a live page in the browser console; Rust hands you a compiler.
• Async plus borrows. Spawning tasks that touch shared data teaches you Arc, clone, and 'static bounds faster than any tutorial. Budget for that learning curve if the team is new to the language.
• A thinner browser story. Driving headless Chrome from Rust works, but the tooling around Playwright and Puppeteer is deeper, and stealth patches for anti-bot detection land in the Node and Python ecosystems first.
• Fewer batteries included. Scrapy gives you retries, throttling, deduplication, robots handling, and item pipelines out of the box. In Rust you either assemble that from crates, write it yourself, or adopt a full crawler engine such as spider.
• Fingerprinting. Matching a browser's TLS handshake is close to a solved problem in Python. In Rust you're tuning the TLS stack yourself or routing traffic through a service that handles it.

How I'd actually decide

Prototype in whatever you already know. Once a crawl reaches hundreds of thousands of pages, or you're processing millions of records and the hosting bill has become a line item somebody asks about, port the fetch-and-parse loop to Rust and keep the exploratory work wherever it's fastest to write. That split works well in practice: Python or a browser devtools session to figure out where the data lives, Rust for the loop that hits that page a million times.

Quick recommendation. If the job fits in an afternoon and runs once, use Python. If it runs forever, write it in Rust.

Setting up your Rust scraping environment

Rust web scraping needs three things in place before you write a line of scraping logic: the toolchain, a cargo project, and a handful of crates that handle HTTP, HTML parsing, and serialization. Setup is quick. The first cargo build is the slow part, because cargo compiles every dependency from source.

Install the toolchain

First, install Rust using rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

On Windows, grab rustup-init.exe from rustup.rs instead. You also need the MSVC build tools, since cargo uses the Microsoft linker by default. If you skip that step, your first build fails with a "linker not found" error rather than anything about Rust itself.

Check that it worked with rustc --version and cargo --version. Both should print a version. If your shell can't find them, open a new terminal so it picks up ~/.cargo/bin on your PATH.

Create the project

Create a new project:

cargo new rust_scraper
cd rust_scraper

That gives you Cargo.toml for dependencies and src/main.rs with a hello-world main. Run cargo run once now, before adding anything, so you know a clean build works.

Add the dependencies

Add these dependencies to your Cargo.toml:

[dependencies]
reqwest = { version = "0.11", features = ["blocking", "cookies"] }
scraper = "0.18"
tokio = { version = "1", features = ["full"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"

You can also let cargo edit the manifest for you with cargo add reqwest --features blocking,cookies, which pins whatever version is current on crates.io.

Here's what each does:

• reqwest: HTTP client that handles requests and responses
• scraper: HTML parsing using CSS selectors
• tokio: Async runtime for concurrent scraping
• serde/serde_json: Serialization for saving scraped data

The feature flags matter more than the crate names, so it's worth knowing what you just switched on.

The blocking feature on reqwest gives you the synchronous API, which is the easier way to write your first script. It has one sharp edge: a blocking client panics if you build or call it from inside an async runtime. Pick one style per program. The cookies feature adds a cookie jar to the client, so a session cookie set on your first request is sent back on every request after it. Many sites hand out a session on the landing page and reject requests that arrive without one.

The full feature on tokio pulls in the entire runtime. It's fine while you're learning, but the pieces a scraper actually uses are rt-multi-thread, macros for the #[tokio::main] attribute, and time for delays between requests. Trimming to those three noticeably shortens rebuilds on a small project. Serde's derive feature is what lets you write #[derive(Serialize, Deserialize)] on your output structs instead of writing the impls by hand.

Fixes for the errors you'll hit first

On Linux, reqwest's default TLS backend links against the system OpenSSL, so a missing libssl-dev or pkg-config shows up as a link failure in the openssl-sys build script. Install those packages, or sidestep C dependencies entirely by setting default-features = false on reqwest and enabling rustls-tls alongside the features above. I'd take rustls by default. It builds the same way on every machine, which matters the moment your scraper moves into a container or CI.

The other early stumble is calling an async reqwest method without a runtime, which produces "there is no reactor running, must be called from the context of a Tokio 1.x runtime". That means your main is missing #[tokio::main], or you enabled tokio without the macros feature.

One thing this stack does not do: run JavaScript. scraper parses the HTML that the server sent, nothing more. If the data you want is injected by a script after page load, no combination of reqwest and scraper will see it, and you'll need a headless browser instead.

Your First Rust Scraper: Blocking Mode

Let's scrape some quotes from http://quotes.toscrape.com. We'll start with synchronous (blocking) code before moving to async.

use reqwest::blocking::Client;
use scraper::{Html, Selector};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a client with a realistic user agent
    let client = Client::builder()
        .user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36")
        .build()?;
    
    // Fetch the page
    let response = client.get("http://quotes.toscrape.com").send()?;
    let body = response.text()?;
    
    // Parse HTML
    let document = Html::parse_document(&body);
    
    // Define selectors
    let quote_selector = Selector::parse("span.text").unwrap();
    let author_selector = Selector::parse("small.author").unwrap();
    
    // Extract data
    for (quote_el, author_el) in document
        .select(&quote_selector)
        .zip(document.select(&author_selector))
    {
        let quote = quote_el.inner_html();
        let author = author_el.inner_html();
        println!("{} - {}", quote, author);
    }
    
    Ok(())
}

What's happening here?

The Client::builder() pattern lets you configure headers, timeouts, and connection pooling. Setting a user agent matters, since many sites block requests without one.

The ? operator propagates errors up the call stack. This is Rust's way of handling errors without try-catch blocks. If any operation fails, the function returns early with that error.

Selector::parse() returns a Result, but we call .unwrap() because we control the selector string. In production code, you'd handle this more gracefully.

The .zip() iterator combines quotes with authors. This works because the site's HTML structure guarantees they appear in matching order.

Understanding Rust's Error Handling in Scrapers

One of Rust's biggest advantages is explicit error handling. No silent failures or uncaught exceptions.

use std::error::Error;
use reqwest::blocking::Client;

fn scrape_page(url: &str) -> Result<String, Box<dyn Error>> {
    let client = Client::new();
    let response = client.get(url).send()?;
    
    // Check status code before parsing
    if !response.status().is_success() {
        return Err(format!("HTTP {}: {}", response.status(), url).into());
    }
    
    let text = response.text()?;
    Ok(text)
}

fn main() {
    match scrape_page("http://quotes.toscrape.com") {
        Ok(html) => println!("Scraped {} bytes", html.len()),
        Err(e) => eprintln!("Scraping failed: {}", e),
    }
}

The Result type forces you to handle errors. You can't accidentally ignore a failed request like you might in Python with a bare except: pass.

In production scrapers, implement custom error types:

use std::fmt;

#[derive(Debug)]
enum ScraperError {
    NetworkError(reqwest::Error),
    ParseError(String),
    NotFound,
}

impl fmt::Display for ScraperError {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        match self {
            ScraperError::NetworkError(e) => write!(f, "Network error: {}", e),
            ScraperError::ParseError(msg) => write!(f, "Parse error: {}", msg),
            ScraperError::NotFound => write!(f, "Resource not found"),
        }
    }
}

impl Error for ScraperError {}

This gives you fine-grained control over error handling and recovery strategies.

Async Scraping with Tokio: The Real Power

Blocking I/O works for single pages, but it doesn't scale. Async Rust with Tokio lets you scrape hundreds of pages concurrently without spawning OS threads.

Here's a basic async scraper:

use reqwest::Client;
use scraper::{Html, Selector};
use tokio;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::new();
    
    let urls = vec![
        "http://quotes.toscrape.com/page/1/",
        "http://quotes.toscrape.com/page/2/",
        "http://quotes.toscrape.com/page/3/",
    ];
    
    // Create futures for all requests
    let futures = urls.into_iter().map(|url| {
        let client = client.clone();
        async move {
            let response = client.get(url).send().await?;
            let html = response.text().await?;
            Ok::<String, reqwest::Error>(html)
        }
    });
    
    // Execute concurrently
    let results = futures::future::try_join_all(futures).await?;
    
    println!("Scraped {} pages", results.len());
    Ok(())
}

Key differences from blocking code:

The #[tokio::main] macro transforms your main function into an async runtime. Without it, you can't use .await.

client.clone() is cheap. It's just an Arc pointer under the hood. All clones share the same underlying connection pool.

try_join_all waits for all futures to complete. If any fail, the whole operation fails. Use join_all if you want to continue even when some requests fail.

Building a Proper Concurrent Scraper

Let's build something more realistic: a scraper that processes multiple pages with rate limiting and error recovery.

use reqwest::Client;
use scraper::{Html, Selector};
use tokio::time::{sleep, Duration};
use futures::stream::{self, StreamExt};
use std::sync::Arc;

struct Scraper {
    client: Client,
    concurrent_limit: usize,
}

impl Scraper {
    fn new(concurrent_limit: usize) -> Self {
        let client = Client::builder()
            .user_agent("Mozilla/5.0 (compatible; RustScraper/1.0)")
            .timeout(Duration::from_secs(30))
            .build()
            .unwrap();
        
        Self { client, concurrent_limit }
    }
    
    async fn scrape_page(&self, url: &str) -> Result<Vec<String>, Box<dyn std::error::Error>> {
        // Random delay between 1-3 seconds
        let delay = Duration::from_millis(1000 + (rand::random::<u64>() % 2000));
        sleep(delay).await;
        
        let response = self.client.get(url).send().await?;
        let html = response.text().await?;
        
        let document = Html::parse_document(&html);
        let selector = Selector::parse("span.text").unwrap();
        
        let quotes: Vec<String> = document
            .select(&selector)
            .map(|el| el.inner_html())
            .collect();
        
        Ok(quotes)
    }
    
    async fn scrape_all(&self, urls: Vec<String>) -> Vec<Result<Vec<String>, Box<dyn std::error::Error>>> {
        stream::iter(urls)
            .map(|url| async move {
                self.scrape_page(&url).await
            })
            .buffer_unordered(self.concurrent_limit)
            .collect()
            .await
    }
}

#[tokio::main]
async fn main() {
    let scraper = Scraper::new(5); // Max 5 concurrent requests
    
    let urls: Vec<String> = (1..=10)
        .map(|i| format!("http://quotes.toscrape.com/page/{}/", i))
        .collect();
    
    let results = scraper.scrape_all(urls).await;
    
    let mut total_quotes = 0;
    for result in results {
        match result {
            Ok(quotes) => total_quotes += quotes.len(),
            Err(e) => eprintln!("Error: {}", e),
        }
    }
    
    println!("Total quotes scraped: {}", total_quotes);
}

Why this pattern works:

buffer_unordered is what makes this work. It processes up to N futures concurrently, yielding results as they complete. This gives you concurrency control without manual semaphore management.

The random delay mimics human behavior. Most anti-bot systems look for regular patterns. A consistent 2-second delay between requests is more suspicious than variable timing.

We're building a Scraper struct instead of loose functions. This makes it easy to share configuration and add features like connection pooling or request caching.

Advanced Anti-Detection: User-Agent Rotation

Sites often block scrapers by tracking user agents. Here's a rotation strategy:

use reqwest::Client;
use rand::seq::SliceRandom;
use std::sync::Arc;
use tokio::sync::Mutex;

const USER_AGENTS: &[&str] = &[
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0",
];

struct RotatingClient {
    user_agents: Vec<String>,
    index: Arc<Mutex<usize>>,
}

impl RotatingClient {
    fn new() -> Self {
        let mut user_agents: Vec<String> = USER_AGENTS
            .iter()
            .map(|s| s.to_string())
            .collect();
        
        user_agents.shuffle(&mut rand::thread_rng());
        
        Self {
            user_agents,
            index: Arc::new(Mutex::new(0)),
        }
    }
    
    async fn get_client(&self) -> Client {
        let mut idx = self.index.lock().await;
        let ua = &self.user_agents[*idx];
        *idx = (*idx + 1) % self.user_agents.len();
        
        Client::builder()
            .user_agent(ua)
            .build()
            .unwrap()
    }
}

#[tokio::main]
async fn main() {
    let rotating_client = Arc::new(RotatingClient::new());
    
    for i in 0..10 {
        let client = rotating_client.get_client().await;
        let response = client
            .get("https://httpbin.org/user-agent")
            .send()
            .await
            .unwrap();
        
        println!("Request {}: {}", i, response.text().await.unwrap());
    }
}

The Arc<Mutex<usize>> pattern lets multiple async tasks safely share the rotation index. The mutex ensures only one task modifies the index at a time.

Shuffling the user agent list at startup adds another layer of randomness. Each run of your scraper will use a different rotation order.

Cookie Management for Session-Based Scraping

Some sites require cookies for session management. Here's how to handle them:

use reqwest::{Client, cookie::Jar};
use std::sync::Arc;
use url::Url;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let jar = Arc::new(Jar::default());
    
    let client = Client::builder()
        .cookie_provider(Arc::clone(&jar))
        .build()?;
    
    // First request sets cookies
    let login_url = "http://quotes.toscrape.com/login";
    let response = client.get(login_url).send().await?;
    
    // Check what cookies were set
    let parsed_url = Url::parse(login_url)?;
    let cookies = jar.cookies(&parsed_url);
    println!("Cookies: {:?}", cookies);
    
    // Subsequent requests automatically include cookies
    let profile_url = "http://quotes.toscrape.com/";
    let response = client.get(profile_url).send().await?;
    println!("Status: {}", response.status());
    
    Ok(())
}

The cookie jar persists across requests automatically. You don't need to manually extract and reinsert cookies as in some other languages.

For more complex scenarios, save cookies to disk:

use reqwest::cookie::Jar;
use std::sync::Arc;
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize)]
struct CookieData {
    name: String,
    value: String,
    domain: String,
}

fn save_cookies(jar: &Jar, url: &url::Url) -> Result<(), Box<dyn std::error::Error>> {
    let cookies_header = jar.cookies(url).unwrap();
    let cookies_str = cookies_header.to_str()?;
    
    // Parse and save to JSON
    // Implementation depends on your needs
    
    Ok(())
}

Memory-Efficient Scraping with Rust's Ownership

Rust's ownership model lets you avoid allocations that a GC language would hide. Here's an approach for memory-efficient scraping.

Instead of storing all HTML in memory, process it as a stream:

use reqwest::Client;
use scraper::{Html, Selector};
use tokio::fs::File;
use tokio::io::AsyncWriteExt;

async fn scrape_to_file(
    client: &Client,
    url: &str,
    output_path: &str,
) -> Result<(), Box<dyn std::error::Error>> {
    let response = client.get(url).send().await?;
    let html = response.text().await?;
    
    // Parse and extract data
    let document = Html::parse_document(&html);
    let selector = Selector::parse("span.text").unwrap();
    
    // Open file for writing
    let mut file = File::create(output_path).await?;
    
    // Write quotes one at a time instead of collecting
    for element in document.select(&selector) {
        let quote = element.inner_html();
        file.write_all(quote.as_bytes()).await?;
        file.write_all(b"\n").await?;
    }
    
    Ok(())
}

This approach uses constant memory regardless of how many quotes are on the page. The iterator processes elements one at a time without allocating a vector.

For truly massive datasets, use the bytes crate for zero-copy operations:

use bytes::Bytes;
use reqwest::Client;

async fn fetch_binary(client: &Client, url: &str) -> Result<Bytes, reqwest::Error> {
    let response = client.get(url).send().await?;
    let bytes = response.bytes().await?;
    Ok(bytes)
}

Bytes is a reference-counted byte buffer. Multiple parts of your code can hold references to the same data without copying.

Building a Custom Rate Limiter

Most tutorials use simple delays. Let's build a proper token bucket rate limiter:

use tokio::time::{sleep, Duration, Instant};
use std::sync::Arc;
use tokio::sync::Mutex;

struct RateLimiter {
    tokens: Arc<Mutex<f64>>,
    capacity: f64,
    refill_rate: f64,
    last_refill: Arc<Mutex<Instant>>,
}

impl RateLimiter {
    fn new(requests_per_second: f64) -> Self {
        Self {
            tokens: Arc::new(Mutex::new(requests_per_second)),
            capacity: requests_per_second,
            refill_rate: requests_per_second,
            last_refill: Arc::new(Mutex::new(Instant::now())),
        }
    }
    
    async fn acquire(&self) {
        loop {
            let mut tokens = self.tokens.lock().await;
            let mut last_refill = self.last_refill.lock().await;
            
            // Refill tokens based on elapsed time
            let elapsed = last_refill.elapsed().as_secs_f64();
            let new_tokens = (*tokens + elapsed * self.refill_rate).min(self.capacity);
            *tokens = new_tokens;
            *last_refill = Instant::now();
            
            if *tokens >= 1.0 {
                *tokens -= 1.0;
                break;
            }
            
            drop(tokens);
            drop(last_refill);
            sleep(Duration::from_millis(100)).await;
        }
    }
}

#[tokio::main]
async fn main() {
    let limiter = Arc::new(RateLimiter::new(5.0)); // 5 requests per second
    
    let mut handles = vec![];
    
    for i in 0..20 {
        let limiter = Arc::clone(&limiter);
        let handle = tokio::spawn(async move {
            limiter.acquire().await;
            println!("Request {} started at {:?}", i, Instant::now());
        });
        handles.push(handle);
    }
    
    for handle in handles {
        handle.await.unwrap();
    }
}

This rate limiter smooths out bursts. Instead of sleeping for a fixed interval, it refills tokens continuously. This yields higher throughput at the same request rate.

Structured Data Extraction with Serde

Most scrapers need to save data. Serde makes this trivial:

use serde::{Serialize, Deserialize};
use scraper::{Html, Selector};
use std::fs::File;
use std::io::Write;

#[derive(Serialize, Deserialize, Debug)]
struct Quote {
    text: String,
    author: String,
    tags: Vec<String>,
}

fn extract_quotes(html: &str) -> Vec<Quote> {
    let document = Html::parse_document(html);
    let quote_selector = Selector::parse("div.quote").unwrap();
    
    let mut quotes = Vec::new();
    
    for quote_el in document.select(&quote_selector) {
        let text_selector = Selector::parse("span.text").unwrap();
        let author_selector = Selector::parse("small.author").unwrap();
        let tag_selector = Selector::parse("a.tag").unwrap();
        
        let text = quote_el
            .select(&text_selector)
            .next()
            .map(|el| el.inner_html())
            .unwrap_or_default();
        
        let author = quote_el
            .select(&author_selector)
            .next()
            .map(|el| el.inner_html())
            .unwrap_or_default();
        
        let tags: Vec<String> = quote_el
            .select(&tag_selector)
            .map(|el| el.inner_html())
            .collect();
        
        quotes.push(Quote { text, author, tags });
    }
    
    quotes
}

fn save_to_json(quotes: &[Quote], path: &str) -> std::io::Result<()> {
    let json = serde_json::to_string_pretty(quotes)?;
    let mut file = File::create(path)?;
    file.write_all(json.as_bytes())?;
    Ok(())
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = reqwest::Client::new();
    let response = client.get("http://quotes.toscrape.com").send().await?;
    let html = response.text().await?;
    
    let quotes = extract_quotes(&html);
    save_to_json(&quotes, "quotes.json")?;
    
    println!("Saved {} quotes", quotes.len());
    Ok(())
}

The #[derive(Serialize)] macro generates all serialization code at compile time. No runtime overhead, no reflection.

For CSV output, add csv = "1.3" to your dependencies:

use csv::Writer;

fn save_to_csv(quotes: &[Quote], path: &str) -> Result<(), Box<dyn std::error::Error>> {
    let mut wtr = Writer::from_path(path)?;
    
    for quote in quotes {
        wtr.serialize(quote)?;
    }
    
    wtr.flush()?;
    Ok(())
}

JavaScript-Rendered Content: Headless Chrome

For sites that require JavaScript, use headless_chrome:

use headless_chrome::{Browser, LaunchOptions};
use headless_chrome::protocol::cdp::Page;

fn scrape_js_page(url: &str) -> Result<String, Box<dyn std::error::Error>> {
    let browser = Browser::new(LaunchOptions::default())?;
    let tab = browser.new_tab()?;
    
    // Navigate and wait for load
    tab.navigate_to(url)?;
    tab.wait_until_navigated()?;
    
    // Wait for specific element to ensure JS has run
    tab.wait_for_element("div.quote")?;
    
    // Get rendered HTML
    let html = tab.get_content()?;
    Ok(html)
}

Warning: headless browsers are slow and resource-intensive. Only use them when you actually need JavaScript execution. For most sites, reqwest + scraper is roughly 10× faster.

Production Tips: Error Recovery and Retry Logic

Real scrapers need to recover from errors. Here's a retry wrapper:

use tokio::time::{sleep, Duration};

async fn retry_with_backoff<F, T, E>(
    mut f: F,
    max_retries: u32,
) -> Result<T, E>
where
    F: FnMut() -> futures::future::BoxFuture<'static, Result<T, E>>,
{
    let mut retries = 0;
    
    loop {
        match f().await {
            Ok(result) => return Ok(result),
            Err(e) if retries >= max_retries => return Err(e),
            Err(_) => {
                let backoff = Duration::from_secs(2u64.pow(retries));
                sleep(backoff).await;
                retries += 1;
            }
        }
    }
}

// Usage example
async fn scrape_with_retry(url: &str) -> Result<String, reqwest::Error> {
    let client = reqwest::Client::new();
    let url = url.to_string();
    
    retry_with_backoff(
        || {
            let client = client.clone();
            let url = url.clone();
            Box::pin(async move {
                client.get(&url).send().await?.text().await
            })
        },
        3, // max 3 retries
    )
    .await
}

Exponential backoff prevents hammering a failing server. The first retry waits 2 seconds, the second waits 4, the third waits 8.

Performance Benchmarking

Measure your scraper's performance with Criterion:

// benches/scraper_bench.rs
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn benchmark_selector_parse(c: &mut Criterion) {
    let html = r#"<div class="quote">...</div>"#;
    
    c.bench_function("parse html", |b| {
        b.iter(|| {
            let document = scraper::Html::parse_document(black_box(html));
            let selector = scraper::Selector::parse("div.quote").unwrap();
            document.select(&selector).count()
        })
    });
}

criterion_group!(benches, benchmark_selector_parse);
criterion_main!(benches);

Run with cargo bench. This shows you where parse time or allocation is spent before a long crawl makes it expensive.

Common Pitfalls and How to Avoid Them

1. Not handling Result properly

Don't use .unwrap() everywhere. It will panic on errors. Use ? or pattern matching:

// Bad
let html = response.text().await.unwrap();

// Good
let html = response.text().await?;

2. Creating too many HTTP clients

Each Client::new() creates a new connection pool. Reuse clients:

// Bad - creates 100 connection pools
for url in urls {
    let client = Client::new();
    client.get(url).send().await?;
}

// Good - one connection pool
let client = Client::new();
for url in urls {
    client.get(url).send().await?;
}

3. Not setting timeouts

Always set timeouts to prevent hanging:

let client = Client::builder()
    .timeout(Duration::from_secs(30))
    .build()?;

4. Forgetting to respect robots.txt

Check a site's robots.txt before scraping at scale. Most commercial sites publish their scraping policies there.

When Rust isn't the right tool for scraping

Rust pays off when scraping is performance-critical. For a large share of projects, it adds compile times, a steeper learning curve, and slower iteration without a matching gain. Before you commit to a Rust scrape pipeline, check where your time actually goes.

On most small jobs, the bottleneck is the network. A scraper spends most of its run waiting on HTTP responses, and a faster language doesn't make a remote server answer sooner. Rust's speed shows up in parsing, concurrency, and memory use, and those only dominate once the volume is large.

Use Python instead if

• You're scraping < 1000 pages. At that scale the job finishes quickly in any language, and the hours you'd spend on Rust setup outweigh any runtime savings.
• Your team doesn't know Rust. The borrow checker, lifetimes, and async runtimes take time to learn. A scraper that only one person can maintain becomes a liability when selectors break.
• You need to prototype quickly. Python lets you test a selector in a REPL, tweak it, and rerun in seconds. In Rust, each change goes through a compile step, which slows exploratory work.
• The scraping logic is complex and changes frequently. Sites redesign, class names shift, and fields move. When you're rewriting extraction code every few weeks, iteration speed matters more than execution speed, and Python's mature ecosystem (Requests, Beautiful Soup, Scrapy) covers most of these cases out of the box.

Rust also gives you little extra against two common obstacles. JavaScript-heavy sites need a headless browser, and the browser becomes the slowest part of the stack whatever language drives it. Anti-bot systems react to your fingerprint, IP reputation, and request patterns, and the language your client is written in doesn't change those.

Use Rust when

• Performance is actually a bottleneck. Profile first. If CPU time on parsing or data processing is what slows you down, Rust's compiled native code helps directly.
• You're processing millions of pages. At this volume, efficient async concurrency and low per-request overhead let a single machine handle more work, which reduces the infrastructure you need.
• Memory usage matters (embedded systems, cloud costs). Rust has no garbage collector, so memory stays predictable on constrained hardware, and smaller instances can lower your cloud bill.
• You're building a long-term production system. Strong typing catches malformed data and missing fields at compile time, and Rust compiles to a single binary that's easy to deploy and keep running for months.

A practical middle path is to prototype in Python until the selectors and data model settle, then move to Rust once you know the scraper will run at scale and its logic has stopped changing every week.

Wrapping Up

Rust gives you compiled speed and compile-time safety for web scraping, but it adds friction that isn't always worth paying. The language's steep learning curve means you'll spend more time fighting the borrow checker than writing selectors, at least initially.

That said, once you internalize Rust's patterns, you'll build scrapers that are both fast and maintainable. The type system catches bugs at compile time that would be runtime errors in Python. The memory model lets you process huge datasets without GC pauses. And the async ecosystem scales to thousands of concurrent connections without spawning OS threads.

Start with the blocking examples in this guide. Once you're comfortable with Rust's basics, move to async code with Tokio. Add anti-detection features only when you need them. And always remember: the best scraper is one that respects the sites you're scraping.