Botasaurus is a Python framework that hides a browser driver, a browser-like HTTP client, caching, parallelism, and multi-format output behind three decorators: @browser, @request, and @task. You write an ordinary function, decorate it, and the framework handles driver setup, stealth patches, retries, and where the data lands. This Botasaurus tutorial goes from install to a working scraper, then into the anti-detection settings that decide whether Cloudflare or DataDome lets you through.
I reach for it because most scraping tools solve half the problem. Selenium runs JavaScript and gets fingerprinted within a handful of requests. Requests is fast and blind to anything rendered client-side. Scrapy scales beautifully once you know its conventions, but you still bolt a browser onto it when a site renders its content in the client. Botasaurus ships the browser, the fingerprint patches, and the ChromeDriver management as one install, which is why it clears protections that stop a stock Selenium script cold.
Two things are worth knowing before you type the install command. First, pip install botasaurus is a meta-package: it pulls in a family of first-party wheels plus a wide transitive dependency tree. Independent benchmarking puts the installed footprint at roughly 122 MB, with the driver itself around 4 MB. That is heavier than a plain requests project, so plan for it in containers and CI images. Second, the sub-millisecond performance figures in the project's marketing do not survive independent benchmarking. Treat them as promotion, not as capacity planning numbers.
Botasaurus also goes further than scripts. It can wrap a finished scraper in a desktop application or a hosted web service with its own UI, drives human-like mouse movement, and includes proxy-saving optimizations that the project claims cut proxy spend by as much as 97%. Everything below assumes Python 3 and a machine that can run Chrome.
What is Botasaurus and why should you use it?
Botasaurus is an all-in-one Python web scraping framework that wraps browser automation and HTTP requests behind the same simple interface. You write a plain Python function, decorate it, and the framework wires up a browser driver, a browser-like HTTP client, caching, parallelism, and output files around it. It builds on Selenium and patches the fingerprint leaks that get stock Selenium blocked, which is the reason most people find their way to a botasaurus tutorial in the first place.
The three decorators
Everything in Botasaurus starts with one of three decorators, and picking the right one is the single biggest performance decision you'll make:
@browserlaunches a real Chrome instance. Use it for JavaScript-rendered pages, login flows, and sites that fingerprint the client.@requestsends HTTP requests with browser-like headers and TLS behaviour. Far faster and lighter than a browser, and enough for any site that ships its data in the raw HTML or a JSON endpoint.@taskruns anything that isn't a fetch: parsing a local file, transforming records, calling an API. It still gets the caching, parallelism, and output handling.
The decorators share the same conventions, so switching a scraper from @browser to @request once you discover the data is in the HTML is mostly a one-line change.
What you get for free
Anti-detection out of the box. Real browser fingerprints, human-like mouse movement, and user-agent handling are on by default rather than something you bolt on with a stack of patches.
No driver management. Botasaurus downloads and version-matches ChromeDriver itself, which removes the most common reason a working scraper breaks after a Chrome update.
Caching. Results are stored on disk, so a re-run during development replays the cached response instead of hitting the site again. That saves your IP reputation while you're still fixing selectors.
Parallelism. Set parallel=5 and five workers run at once, with the framework handling the process pool and merging results.
Structured output. Scraped records land in JSON and CSV files without you writing serialisation code.
The alternative is assembling this yourself: undetected-chromedriver for the fingerprint work, requests for HTTP calls, multiprocessing for parallelism, plus your own cache layer and writers.
Beyond a script: desktop apps and web services
Botasaurus also ships a productization layer most scraping libraries don't have. A finished scraper can be packaged as a desktop application with a UI, or run as a hosted web service where someone enters an input and downloads the results. If you build scrapers for people who don't run Python, that's the difference between handing over a repository and handing over an installer. Botasaurus also advertises proxy-saving optimizations, with claimed savings of up to 97% on proxy spend.
What to expect before you install
Independent benchmarking has pulled apart what pip install botasaurus actually does: it's a meta-package that pulls in a family of first-party wheels plus a wide transitive dependency tree, landing at roughly 122 MB installed, with the browser driver itself around 4 MB. Budget accordingly for slim containers and CI images. The same benchmarking work found the marketed sub-millisecond performance figures don't survive scrutiny, so treat those numbers as marketing rather than something to plan capacity around.
When another tool fits better
Running a real browser costs memory and time. Each Chromium instance carries meaningful overhead, so crawling thousands of static pages per minute is work for Scrapy's async HTTP engine, which also gives you Items, pipelines, and feed exports for heavier data-processing needs. Botasaurus earns its keep on the sites that fight back: fingerprinting, CAPTCHAs, JavaScript-rendered content, and anything behind a bot wall where a raw HTTP request never gets a usable response.
Getting started: your first Botasaurus scraper
The shortest path through this Botasaurus tutorial is three steps: install the package, decorate a plain Python function, run the file. There is no project scaffold to generate, no settings module to fill in, and no driver binary to download by hand.
Install Botasaurus
Python 3.7 or higher is the only prerequisite:
pip install botasaurus
That single command installs more than one library. Botasaurus is a meta-package: it pulls in a family of first-party wheels plus a wide transitive dependency tree, which lands at roughly 122 MB on disk once everything resolves. The browser driver itself is small, around 4 MB. Expect the install to take a few minutes, and budget for the footprint if you are baking Botasaurus into a Docker image or a CI cache where layer size matters.
One expectation to set early: independent benchmarking has not reproduced the sub-millisecond timings that appear in Botasaurus marketing, so don't size a project around them. Judge the framework on what it actually saves you, which is browser plumbing and anti-detection work.
On first run, Botasaurus downloads ChromeDriver automatically and matches it to your installed Chrome. Driver-version mismatches are the single most common way a plain Selenium setup breaks after a Chrome update, and this removes that failure mode entirely.
Write and run the scraper
Create a file called main.py:
from botasaurus.browser import browser, Driver
@browser
def scrape_heading(driver: Driver, data):
# Visit the website
driver.get("https://www.omkar.cloud/")
# Extract the heading text
heading = driver.get_text("h1")
# Return the data (Botasaurus saves it automatically)
return {
"heading": heading
}
# Run the scraper
scrape_heading()
Run it:
python main.py
Chrome launches, the page loads, the heading is extracted, and the result is written to output/scrape_heading.json. No driver setup, no configuration file, no boilerplate around opening and closing the browser.
What the decorator is doing for you
- The
@browserdecorator tells Botasaurus to run this function inside a real, patched Chrome session rather than an HTTP client. - Botasaurus creates and injects the
Driverinstance, then closes it when the function returns, including when the function raises. driver.get_text("h1")collapses the usual find-element-then-read-text pair into one call, and there are similar shorthands for links, attributes and lists of elements.- Whatever the function returns is serialised and saved into the
output/folder, named after the function. Caching, parallelism and other output formats hang off the same decorator as arguments rather than as separate machinery.
The second parameter, data, is the input Botasaurus hands to your function. In the example above nothing is passed, so it arrives empty. Pass a value when you call the scraper and it shows up there, which is how you move from a hardcoded URL to a function that takes a list of targets. The equivalent decorators for other workloads are @request for a browser-like HTTP client with no Chrome overhead, and @task for plain logic that still wants the caching and output handling.
Starting from the official template instead
For a new project rather than a one-file experiment, the maintainers recommend cloning omkarcloud/botasaurus-starter into your project directory, installing from its requirements.txt, then running python run.py install followed by python main.py. The template ships a working scraper whose results land in output/finished.json, and you adapt it by editing task.py rather than wiring a project from scratch.
The same template also exposes a UI dashboard: run python run.py and you get a browser interface for triggering the scraper and viewing results, which is useful when a non-developer needs to run the job without touching a terminal.
The design goal running through all of this is to make the common case a single decorated function, while leaving proxies, caching, parallel workers and stealth settings available as arguments when you need them.
Understanding the three decorators
Botasaurus gives you three decorators for different scraping scenarios. Picking the right one matters for performance and reliability.
@browser: For JavaScript-heavy sites
Use this when you need a real browser - JavaScript rendering, dynamic content, or sites that detect headless browsers.
from botasaurus.browser import browser, Driver
@browser
def scrape_product(driver: Driver, data):
"""
Scrape a product page that loads content via JavaScript.
"""
driver.get("https://example.com/product/12345")
# Wait for JavaScript to load the price
driver.wait_for_element(".price", wait=10)
# Extract data
name = driver.get_text("h1.product-name")
price = driver.get_text(".price")
description = driver.get_text(".description")
# Get all image URLs
images = driver.links(".product-images img", attribute="src")
return {
"name": name,
"price": price,
"description": description,
"images": images
}
scrape_product()
The Driver object is a patched Selenium WebDriver with convenience shortcuts like get_text(), links(), and wait_for_element() that make scraping cleaner.
@request: For fast HTTP scraping
When you don't need JavaScript rendering, use @request. It's 10-20x faster than browser automation because it skips the browser entirely.
from botasaurus.request import request, Request
from bs4 import BeautifulSoup
@request
def scrape_api(request: Request, data):
"""
Scrape a site using HTTP requests (no browser).
"""
response = request.get("https://api.example.com/products")
# Parse JSON response
products = response.json()
return products
# Or scrape HTML with BeautifulSoup
@request
def scrape_html(request: Request, data):
response = request.get("https://example.com/products")
soup = BeautifulSoup(response.text, 'html.parser')
products = []
for item in soup.select('.product'):
products.append({
"name": item.select_one('.name').text,
"price": item.select_one('.price').text
})
return products
scrape_html()
Use @request when the site's HTML contains all the data you need. If you inspect a page and see the data in the source HTML (not loaded by JavaScript), requests are the way to go.
@task: For non-scraping work
Sometimes you need to process files, call APIs, or perform data transformations. Use @task for anything that doesn't involve web scraping.
from botasaurus.task import task
import pandas as pd
@task
def process_csv(data):
"""
Process a CSV file and extract insights.
"""
df = pd.read_csv("data.csv")
# Perform analysis
summary = {
"total_rows": len(df),
"average_price": df['price'].mean(),
"top_products": df.nlargest(10, 'sales')['name'].tolist()
}
return summary
process_csv()
The @task decorator gives you the same benefits (automatic output saving, error handling) without browser or HTTP overhead.
Scraping JavaScript-heavy websites
Modern websites load content dynamically with JavaScript. If you see a page with loading spinners or infinite scroll, you need browser automation.
Here's how to handle common JavaScript patterns:
Waiting for elements to load
Don't blindly wait 5 seconds. Wait for specific elements:
@browser
def scrape_dynamic_content(driver: Driver, data):
driver.get("https://example.com/products")
# Wait for the product list to appear
driver.wait_for_element(".product-card", wait=15)
# Extract products
products = []
for card in driver.select_all(".product-card"):
name = card.get_text(".product-name")
price = card.get_text(".product-price")
products.append({"name": name, "price": price})
return products
wait_for_element() is smart: it waits up to 15 seconds for the element to appear, but returns immediately once it does.
Handling infinite scroll
Many sites load more content as you scroll. Here's how to scrape them:
@browser
def scrape_infinite_scroll(driver: Driver, data):
driver.get("https://example.com/feed")
items = []
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
# Scroll to bottom
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
# Wait for new content to load
driver.sleep(2)
# Calculate new scroll height
new_height = driver.execute_script("return document.body.scrollHeight")
if new_height == last_height:
break # No more content
last_height = new_height
# Now extract all items
for item in driver.select_all(".feed-item"):
items.append({
"title": item.get_text(".title"),
"content": item.get_text(".content")
})
return items
This pattern scrolls until no new content loads, then extracts everything at once.
Clicking buttons and interacting
Sometimes you need to click "Load More" buttons or fill forms:
@browser
def scrape_with_interaction(driver: Driver, data):
driver.get("https://example.com/search")
# Fill search form
driver.type("input[name='query']", "python books")
# Click search button
driver.click("button[type='submit']")
# Wait for results
driver.wait_for_element(".search-results", wait=10)
# Click "Load More" until no more items
while True:
try:
load_more = driver.select("button.load-more")
load_more.click()
driver.sleep(2) # Wait for new items to load
except:
break # Button disappeared, no more items
# Extract all results
results = []
for result in driver.select_all(".search-result"):
results.append({
"title": result.get_text(".title"),
"author": result.get_text(".author")
})
return results
Botasaurus's click(), type(), and select() methods make interaction simple.
Bypassing Cloudflare and anti-bot systems
Anti-bot bypass is Botasaurus's strongest feature. Most scraping frameworks get blocked by Cloudflare within seconds. Botasaurus makes it look easy.
The google_get() trick
Instead of visiting a site directly, simulate arriving from a Google search result:
@browser
def scrape_cloudflare_protected(driver: Driver, data):
# Don't use driver.get() - it gets blocked
# Use google_get() instead
driver.google_get("https://example.com/protected-page")
# Wait for Cloudflare challenge to pass (if any)
driver.wait_for_element("h1", wait=30)
# Now extract data normally
title = driver.get_text("h1")
content = driver.get_text(".main-content")
return {
"title": title,
"content": content
}
scrape_cloudflare_protected()
google_get() routes your request through Google, making it appear like organic traffic. Cloudflare is far less likely to block traffic that originates from search engines.
Using real browser fingerprints
Botasaurus can use realistic browser configurations:
from botasaurus import bt
@browser(
user_agent=bt.UserAgent.REAL, # Use real browser fingerprint
window_size=bt.WindowSize.REAL, # Realistic window size
block_resources=True # Still block images for speed
)
def scrape_with_fingerprint(driver: Driver, data):
driver.get("https://example.com")
# Your scraping logic here
return {"data": driver.get_text("body")}
The bt.UserAgent.REAL and bt.WindowSize.REAL options make your scraper mimic a real user's browser. This defeats fingerprinting techniques that check screen resolution, user agent, and browser capabilities.
Headless vs headful browsers
Some sites detect headless browsers. If you're getting blocked, try running with a visible browser:
@browser(
headless=False, # Show the browser window
user_agent=bt.UserAgent.REAL
)
def scrape_tough_site(driver: Driver, data):
driver.get("https://tough-site.com")
# Scraping logic
return {"data": "..."}
Running headless=False means the browser window appears on your screen. It's slower and requires a display, but some sites can't detect it.
Adding delays to appear human
Real users don't scrape at lightning speed. Add random delays:
import random
@browser
def scrape_humanlike(driver: Driver, data):
driver.get("https://example.com/products")
products = []
for link in driver.links(".product-link"):
# Random delay between 1-3 seconds
driver.sleep(random.uniform(1, 3))
driver.get(link)
products.append({
"name": driver.get_text("h1"),
"price": driver.get_text(".price")
})
return products
The random.uniform(1, 3) creates a delay between 1 and 3 seconds, making your scraper's timing unpredictable like a human's.
Parallel scraping for speed
Scraping one page at a time is slow. If each page takes 3 seconds, scraping 100 pages takes 5 minutes. With parallel scraping, you can do it in under a minute.
Botasaurus makes parallelization trivial:
@browser(
parallel=5, # Run 5 browsers simultaneously
headless=True
)
def scrape_products(driver: Driver, url):
"""
This function runs in parallel for each URL.
"""
driver.get(url)
driver.wait_for_element("h1", wait=10)
return {
"name": driver.get_text("h1"),
"price": driver.get_text(".price"),
"url": url
}
# Pass a list of URLs
urls = [
"https://example.com/product/1",
"https://example.com/product/2",
"https://example.com/product/3",
# ... 100 more URLs
]
# Botasaurus automatically distributes URLs across 5 browsers
results = scrape_products(urls)
Pass a list of URLs, and Botasaurus handles the rest. It opens 5 browser instances, distributes URLs among them, and collects results. You don't write threading code.
How many parallel browsers?
More isn't always better. Each browser consumes ~200-300MB of RAM. On a machine with 8GB RAM, parallel=10 is reasonable. Beyond that, you risk memory issues.
Start with parallel=3, measure performance, and increase gradually:
from botasaurus import bt
@browser(
parallel=bt.calc_max_parallel_browsers, # Auto-calculate based on system
headless=True
)
def scrape_auto_parallel(driver: Driver, url):
# Scraping logic
pass
bt.calc_max_parallel_browsers automatically determines the optimal number based on your CPU and RAM.
Reusing browsers for better performance
Opening a new browser for every page is expensive. If you're scraping many pages from the same site, reuse the browser:
@browser(
parallel=3,
reuse_driver=True, # Keep browsers alive between pages
headless=True
)
def scrape_efficiently(driver: Driver, url):
driver.get(url)
return {"data": driver.get_text("h1")}
# These 100 URLs will use just 3 browser instances
urls = ["https://example.com/page-{}".format(i) for i in range(100)]
results = scrape_efficiently(urls)
With reuse_driver=True, Botasaurus keeps browsers open and reuses them. This cuts overhead dramatically.
Using proxies and browser profiles
Proxy configuration
Proxies are essential for large-scale scraping. They distribute requests across multiple IP addresses, avoiding rate limits and blocks.
Basic proxy usage:
@browser(proxy="http://proxy-server.com:8080")
def scrape_with_proxy(driver: Driver, url):
driver.get(url)
# Verify proxy is working
ip = driver.get_text("body") # On ipinfo.io or similar
print(f"Scraping from IP: {ip}")
return {"data": driver.get_text(".content")}
For authenticated proxies (username/password):
@browser(
proxy={
"http": "http://user:[email protected]:8080",
"https": "https://user:[email protected]:8080"
}
)
def scrape_with_auth_proxy(driver: Driver, url):
driver.get(url)
return {"data": "..."}
Botasaurus supports SSL-authenticated proxies, which is rare. Most frameworks force you into complex workarounds, but Botasaurus handles it natively.
Browser profiles for logged-in scraping
Need to scrape content behind a login? Use browser profiles to save cookies and session data:
@browser(
profile="my-linkedin-profile" # Reuse this profile across scrapes
)
def scrape_linkedin(driver: Driver, data):
driver.get("https://linkedin.com/in/someone")
# First time: manually log in
# Botasaurus saves cookies automatically
# Next time: you're already logged in
name = driver.get_text(".profile-name")
headline = driver.get_text(".profile-headline")
return {
"name": name,
"headline": headline
}
scrape_linkedin()
The first time you run this, manually log in to LinkedIn in the browser window. Botasaurus saves the cookies. Every subsequent run reuses those cookies - you stay logged in.
Installing Chrome extensions
Sometimes you need browser extensions (like CAPTCHA solvers or ad blockers):
@browser(
extensions=[
"path/to/extension-1.crx",
"path/to/extension-2.crx"
]
)
def scrape_with_extensions(driver: Driver, url):
driver.get(url)
# Extensions are pre-installed and active
return {"data": driver.get_text(".content")}
Download any Chrome extension as a .crx file, and Botasaurus loads it automatically.
Caching to avoid re-scraping
Development involves running the same scraper dozens of times. Re-scraping wastes time and risks getting blocked. Botasaurus's caching solves this:
@browser(
cache=True, # Enable caching
cache_duration=3600 # Cache for 1 hour (seconds)
)
def scrape_cached(driver: Driver, url):
driver.get(url)
# First run: scrapes the page and caches result
# Subsequent runs: returns cached data instantly
return {
"title": driver.get_text("h1"),
"content": driver.get_text(".article-body")
}
# First call: takes 5 seconds (actual scraping)
result1 = scrape_cached("https://example.com/article")
# Second call: takes 0.1 seconds (cached)
result2 = scrape_cached("https://example.com/article")
Caching is per-URL. If you scrape page-1.html and then page-2.html, both are cached separately.
Want to force a fresh scrape? Delete the cache:
from botasaurus.cache import Cache
# Clear all caches
Cache.clear()
# Clear cache for specific function
Cache.clear("scrape_cached")
Custom cache keys
Sometimes URLs aren't unique enough. Add custom cache logic:
@browser(
cache=True,
cache_key=lambda data: f"{data['url']}-{data['user_id']}"
)
def scrape_personalized(driver: Driver, data):
url = data['url']
user_id = data['user_id']
# Different users see different content
driver.get(url)
return {"data": driver.get_text(".personalized-content")}
# These create separate cache entries
scrape_personalized({"url": "example.com", "user_id": "123"})
scrape_personalized({"url": "example.com", "user_id": "456"})
The cache_key function determines how cache entries are keyed. Here, we key by both URL and user ID.
Handling authentication and cookies
Many valuable sites require login. Botasaurus makes authenticated scraping straightforward.
Manual login with profile persistence
The simplest approach: log in manually once, save the profile:
@browser(
profile="amazon-profile",
headless=False # Show browser for manual login
)
def scrape_amazon_orders(driver: Driver, data):
driver.get("https://amazon.com/orders")
# First run: you'll see the login page
# Log in manually, then press Enter in terminal to continue
input("Logged in? Press Enter to continue...")
# Botasaurus saves cookies automatically
# Next runs: you're already logged in
orders = []
for order in driver.select_all(".order-card"):
orders.append({
"order_id": order.get_text(".order-id"),
"date": order.get_text(".order-date"),
"total": order.get_text(".order-total")
})
return orders
scrape_amazon_orders()
The profile parameter saves all cookies and local storage. Run this once, log in manually, and you're authenticated forever (until cookies expire).
Programmatic login
For automation, log in programmatically:
@browser(profile="github-profile")
def scrape_github_private(driver: Driver, data):
driver.get("https://github.com/login")
# Fill login form
driver.type("#login_field", "[email protected]")
driver.type("#password", "your-password")
# Click login button
driver.click("input[name='commit']")
# Wait for redirect
driver.wait_for_element(".dashboard", wait=10)
# Now scrape private repos
driver.get("https://github.com/your-org/private-repo")
readme = driver.get_text(".markdown-body")
return {"readme": readme}
Combine this with profiles, and you only log in once. Subsequent runs reuse the session.
Handling CAPTCHA
Some sites show CAPTCHAs during login. For these, use manual solving or a CAPTCHA service:
@browser(
profile="protected-site",
headless=False # Show browser to solve CAPTCHA
)
def scrape_with_captcha(driver: Driver, data):
driver.get("https://protected-site.com/login")
# Fill credentials
driver.type("#email", "[email protected]")
driver.type("#password", "password123")
# Click login - CAPTCHA appears
driver.click("button[type='submit']")
# Wait for manual CAPTCHA solving
input("Solve the CAPTCHA, then press Enter...")
# Proceed with scraping
driver.get("https://protected-site.com/data")
return {"data": driver.get_text(".protected-content")}
For automated CAPTCHA solving, Botasaurus supports services like CapSolver:
@browser(
captcha_solver="capsolver",
capsolver_api_key="your-api-key-here"
)
def scrape_with_auto_captcha(driver: Driver, data):
driver.get("https://site-with-captcha.com")
# Botasaurus automatically solves CAPTCHAs
driver.solve_captcha()
return {"data": driver.get_text(".content")}
Note: CAPTCHA solving services cost money (typically $1-3 per 1000 CAPTCHAs). Manual solving is free but doesn't scale.
Building production-ready scrapers
Development scrapers are quick and dirty. Production scrapers need error handling, logging, and reliability.
Error handling and retries
Scraping fails. Networks timeout, selectors change, servers return errors. Handle failures gracefully:
@browser(
max_retry=3, # Retry 3 times on failure
retry_wait=5 # Wait 5 seconds between retries
)
def scrape_with_retries(driver: Driver, url):
try:
driver.get(url)
driver.wait_for_element("h1", wait=15)
title = driver.get_text("h1")
# Validate data before returning
if not title or title == "":
raise Exception("Empty title - page might not have loaded")
return {"title": title}
except Exception as e:
print(f"Error scraping {url}: {str(e)}")
raise # Re-raise to trigger retry
max_retry=3 automatically retries on exceptions. If it fails 3 times, Botasaurus returns None instead of crashing your entire scrape.
Blocking resources for speed
Images, CSS, and fonts slow down scraping. You don't need them for data extraction:
@browser(
block_resources=True, # Block images, CSS, fonts
headless=True
)
def scrape_fast(driver: Driver, url):
# Loads in 1 second instead of 5
driver.get(url)
return {"data": driver.get_text(".content")}
This can reduce page load time by 70-80%. Your scraper extracts text 4-5x faster.
For finer control, specify what to block:
@browser(
block_resources=["image", "stylesheet", "font"]
)
def scrape_selective(driver: Driver, url):
# Blocks images, CSS, fonts but loads JavaScript
driver.get(url)
return {"data": "..."}
Handling pagination properly
Most sites paginate results. Don't hardcode page numbers - follow "next" links:
@browser
def scrape_all_pages(driver: Driver, data):
driver.get("https://example.com/products?page=1")
all_products = []
while True:
# Scrape current page
for product in driver.select_all(".product"):
all_products.append({
"name": product.get_text(".name"),
"price": product.get_text(".price")
})
# Look for "next" button
try:
next_button = driver.select("a.next-page")
next_button.click()
driver.sleep(2) # Wait for new page to load
except:
break # No more pages
return all_products
This pattern works for any paginated site. It clicks "next" until the button disappears.
Saving data incrementally
For large scrapes, don't wait until the end to save data. Save as you go:
import json
@browser
def scrape_large_dataset(driver: Driver, urls):
for i, url in enumerate(urls):
driver.get(url)
data = {
"url": url,
"title": driver.get_text("h1"),
"content": driver.get_text(".content")
}
# Save every 10 items
if i % 10 == 0:
with open(f"output/batch_{i}.json", "w") as f:
json.dump(data, f)
# Or append to a single file
with open("output/all_data.jsonl", "a") as f:
f.write(json.dumps(data) + "\n")
return {"scraped": len(urls)}
If your scraper crashes after scraping 500 pages, you'll still have the data. Don't lose hours of work to a crash.
Common mistakes and how to avoid them
After hundreds of hours with Botasaurus, here are the traps I see developers fall into:
1. Not reusing browsers
Opening a new browser for every scrape is expensive:
# Bad: Opens new browser each time (slow)
@browser
def scrape_inefficient(driver: Driver, url):
driver.get(url)
return {"data": driver.get_text("h1")}
# Call this 100 times = 100 browser launches
for url in urls:
scrape_inefficient(url)
Instead, pass a list of URLs and use reuse_driver:
# Good: Reuses same browser (fast)
@browser(reuse_driver=True)
def scrape_efficient(driver: Driver, url):
driver.get(url)
return {"data": driver.get_text("h1")}
# Single browser handles all 100 URLs
scrape_efficient(urls)
This cuts execution time by 80-90% on large scrapes.
2. Ignoring cache during development
Running the same scraper repeatedly while debugging wastes time:
# Turn on cache during development
@browser(
cache=True,
cache_duration=3600 # 1 hour
)
def scrape_dev(driver: Driver, url):
driver.get(url)
return {"data": "..."}
First run scrapes the page. Subsequent runs return cached data instantly. When you're ready for production, remove cache=True.
3. Using headless browsers when headful works
Some developers always use headless=True for speed. But headless browsers are easier to detect:
# Gets blocked on some sites
@browser(headless=True)
def scrape_detected(driver: Driver, url):
driver.get(url)
return {"data": "..."}
# Works better
@browser(headless=False, user_agent=bt.UserAgent.REAL)
def scrape_stealthy(driver: Driver, url):
driver.get(url)
return {"data": "..."}
If you're getting blocked, try headless=False first before adding proxies or other complexity.
4. Not handling missing elements
Selectors change. Elements disappear. Your scraper should handle this:
# Bad: Crashes if price is missing
@browser
def scrape_fragile(driver: Driver, url):
driver.get(url)
name = driver.get_text("h1") # Always present
price = driver.get_text(".price") # Sometimes missing!
return {"name": name, "price": price}
Add fallbacks:
# Good: Handles missing elements
@browser
def scrape_robust(driver: Driver, url):
driver.get(url)
name = driver.get_text("h1")
# Use try/except for optional elements
try:
price = driver.get_text(".price")
except:
price = "N/A"
return {"name": name, "price": price}
Or use Botasaurus's built-in default values:
@browser
def scrape_with_defaults(driver: Driver, url):
driver.get(url)
name = driver.get_text("h1", default="Unknown")
price = driver.get_text(".price", default="N/A")
return {"name": name, "price": price}
5. Scraping too fast
Real humans don't navigate at computer speed. Add delays:
import random
@browser
def scrape_humanlike(driver: Driver, urls):
products = []
for url in urls:
# Random delay between pages
driver.sleep(random.uniform(2, 5))
driver.get(url)
products.append({"name": driver.get_text("h1")})
return products
Scraping 100 pages in 30 seconds looks suspicious. Spread it over 5-10 minutes with random delays.
6. Not using parallel when you should
Scraping one page at a time is unnecessarily slow:
# Slow: 100 pages × 3 seconds = 5 minutes
@browser
def scrape_sequential(driver: Driver, url):
driver.get(url)
return {"data": "..."}
for url in urls:
scrape_sequential(url)
# Fast: 100 pages ÷ 5 parallel = 1 minute
@browser(parallel=5, reuse_driver=True)
def scrape_parallel(driver: Driver, url):
driver.get(url)
return {"data": "..."}
scrape_parallel(urls) # Pass list, not loop
If you're scraping more than 10 pages, use parallel.
7. Forgetting to clean data
Scraped data is messy. Always clean it:
@browser
def scrape_with_cleaning(driver: Driver, url):
driver.get(url)
price_raw = driver.get_text(".price") # "$ 19.99 "
# Clean the price
price = price_raw.strip() # " $19.99 " → "$19.99"
price = price.replace("$", "") # "$19.99" → "19.99"
price = float(price) # "19.99" → 19.99
return {"price": price}
Botasaurus has built-in cleaners for common cases:
from botasaurus import cl
@browser
def scrape_auto_clean(driver: Driver, url):
driver.get(url)
price_raw = driver.get_text(".price") # "$ 19.99 "
price = cl.extract_numbers(price_raw) # 19.99
return {"price": price}
The cl module has cleaners for prices, dates, phone numbers, emails, and more.
Using Botasaurus in production
Ready to deploy your scraper? Here are some practical approaches:
Command-line scraper
The simplest deployment: run it as a script on a schedule (cron job):
# scraper.py
from botasaurus.browser import browser, Driver
@browser(
headless=True,
parallel=5,
cache=True
)
def daily_price_scrape(driver: Driver, urls):
# Scraping logic
pass
if __name__ == "__main__":
urls = [...] # Load from database or file
results = daily_price_scrape(urls)
# Save to database
save_to_db(results)
Run daily with cron:
0 2 * * * cd /path/to/scraper && python scraper.py
Flask API wrapper
For on-demand scraping, wrap Botasaurus in a simple Flask API:
from flask import Flask, request, jsonify
from botasaurus.browser import browser, Driver
app = Flask(__name__)
@browser(headless=True, reuse_driver=True)
def scrape_product(driver: Driver, url):
driver.get(url)
return {
"name": driver.get_text("h1"),
"price": driver.get_text(".price")
}
@app.route('/scrape', methods=['POST'])
def scrape_endpoint():
url = request.json.get('url')
result = scrape_product(url)
return jsonify(result)
if __name__ == '__main__':
app.run(port=5000)
Now anyone can hit your API to scrape on demand.
Docker for consistency
Containerize your scraper for reliable deployments:
FROM python:3.11-slim
# Install Chrome dependencies
RUN apt-get update && apt-get install -y \
chromium \
chromium-driver \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "scraper.py"]
Build and run:
docker build -t my-scraper .
docker run my-scraper
This ensures your scraper works identically on any machine.
Wrapping up
Botasaurus is what web scraping should have been all along: powerful but simple, and opinionated where it matters.
The key takeaways:
- Pick the right decorator:
@browserfor JavaScript sites,@requestfor fast HTTP scraping,@taskfor everything else - Use anti-detection by default:
google_get(), real user agents, and profiles make your scraper undetectable - Parallelize everything: The
parallelparameter turns single-threaded scrapes into parallel operations - Cache aggressively: Never re-scrape the same page twice during development
- Handle errors gracefully:
max_retry, fallback values, and proper exception handling keep your scraper running - Think like a human: Random delays, headful browsers, and gradual pagination make scraping sustainable
Most web scraping tutorials focus on extracting data. Botasaurus goes further - it solves the real problems of modern web scraping: detection, scale, and reliability.
You don't need a PhD in browser fingerprinting or an army of proxies. You need Botasaurus, a clear understanding of the three decorators, and the patience to let your scraper run at a human pace.
The web has evolved. Bot detection has evolved. Botasaurus keeps you one step ahead.
