Web scraping in 2026 means fighting anti-bot systems at every turn. Cloudflare, DataDome, and Akamai block most automation tools before you grab a single byte of data.
Zendriver solves this problem by using Chrome DevTools Protocol instead of WebDriver. This makes it virtually undetectable to anti-bot systems while remaining blazing fast.
In this guide, you'll learn how to install Zendriver, navigate pages, extract data, handle dynamic content, and bypass common anti-bot protections. You'll walk away with working code you can adapt for your own projects.
What is Zendriver?
Zendriver is a Python library that controls Chrome browsers using the Chrome DevTools Protocol (CDP). Unlike Selenium or Playwright, it bypasses WebDriver entirely—the primary detection vector for anti-bot systems.
The library started as a fork of nodriver to fix critical bugs and enable community contributions. Today it offers first-class Docker support, automatic cookie management, and typed CDP bindings for advanced use cases.
In benchmark tests, Zendriver successfully bypassed Cloudflare, CloudFront, and Akamai protections out of the box—a 75% success rate compared to 25% for Selenium and Playwright.
Prerequisites
Before starting, make sure you have:
- Python 3.8 or higher installed
- Google Chrome or Chromium browser on your system
- Basic understanding of async/await in Python
Zendriver handles Chrome installation detection automatically. You don't need to download separate drivers like with Selenium.
Step 1: Install Zendriver
Installation is straightforward using pip or any Python package manager.
pip install zendriver
If you use uv or poetry, the commands are similar:
# Using uv
uv add zendriver
# Using poetry
poetry add zendriver
Verify the installation by checking the version:
import zendriver
print(zendriver.__version__)
You should see a version number like 0.15.2 or higher.
Windows users: Add this asyncio policy fix at the top of your scripts for better compatibility:
import asyncio
import sys
if sys.platform == "win32":
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
This prevents event loop errors that can occur on Windows systems.
Step 2: Launch Your First Browser Session
Zendriver uses async/await syntax throughout. Every script starts with launching a browser instance.
Here's the minimal working example:
import asyncio
import zendriver as zd
async def main():
# Start a new browser instance
browser = await zd.start()
# Navigate to a website
page = await browser.get("https://example.com")
# Always close the browser when done
await browser.stop()
if __name__ == "__main__":
asyncio.run(main())
Let me break down what each line does.
The zd.start() function launches Chrome with anti-detection settings enabled by default. It returns a Browser object you'll use throughout your script.
The browser.get() method navigates to a URL and returns a Tab (page) object. This Tab object contains all methods for interacting with page content.
The browser.stop() method cleanly shuts down Chrome and releases resources.
Pro tip: By default, Zendriver creates a fresh browser profile for each session. This profile is automatically deleted when your script ends.
Configuring Browser Options
You can customize the browser launch with various options:
import zendriver as zd
browser = await zd.start(
headless=False, # Show the browser window
user_data_dir="/path/to/profile", # Use persistent profile
browser_args=["--disable-gpu"], # Custom Chrome arguments
lang="en-US" # Set browser language
)
For more complex configurations, use the Config object:
import zendriver as zd
config = zd.Config()
config.headless = True
config.browser_args = ["--no-sandbox", "--disable-dev-shm-usage"]
browser = await zd.start(config=config)
The --no-sandbox and --disable-dev-shm-usage flags are essential for running in Docker or cloud environments.
Step 3: Navigate and Extract Page Content
Once you have a page loaded, extracting content is simple.
Get Full HTML
async def scrape_html():
browser = await zd.start()
page = await browser.get("https://example.com")
# Get the entire page HTML as a string
html_content = await page.get_content()
print(html_content)
await browser.stop()
The get_content() method returns the complete DOM including any JavaScript-rendered content.
Save Screenshots
Screenshots are useful for debugging and visual verification:
async def capture_screenshot():
browser = await zd.start()
page = await browser.get("https://example.com")
# Save screenshot to current directory
await page.save_screenshot("example_screenshot.png")
await browser.stop()
You can specify the full path if needed. The screenshot captures exactly what you'd see in the browser window.
Open Multiple Tabs
Zendriver handles multiple tabs efficiently:
async def multi_tab_scraping():
browser = await zd.start()
# First tab
page1 = await browser.get("https://example.com")
# Open second tab
page2 = await browser.get("https://github.com", new_tab=True)
# Open in new window
page3 = await browser.get("https://twitter.com", new_window=True)
# Switch between pages
await page1.bring_to_front()
# Close specific tabs
await page2.close()
await browser.stop()
Each tab operates independently. You can scrape multiple sites concurrently using asyncio.gather().
Step 4: Find and Interact with Elements
Element selection in Zendriver works two ways: CSS selectors and text search.
CSS Selectors
Use select() for single elements and select_all() for multiple:
async def select_elements():
browser = await zd.start()
page = await browser.get("https://example.com")
# Select single element by CSS selector
heading = await page.select("h1")
print(heading.text)
# Select multiple elements
links = await page.select_all("a")
for link in links:
print(link.attrs.get("href"))
await browser.stop()
CSS selectors follow standard syntax. Use classes with .classname, IDs with #id, and attributes with [attribute=value].
Text-Based Search
The find() method searches for elements by visible text:
async def find_by_text():
browser = await zd.start()
page = await browser.get("https://example.com")
# Find element containing "More information"
link = await page.find("More information", best_match=True)
if link:
await link.click()
await browser.stop()
The best_match=True parameter is important. Without it, Zendriver returns the first matching element.
With best_match=True, it compares text length to find the most relevant match. This prevents clicking on a parent container when you want a specific button.
Interacting with Elements
Once you have an element, you can click, type, or extract data:
async def interact_with_elements():
browser = await zd.start()
page = await browser.get("https://example.com")
# Click an element
button = await page.find("Submit", best_match=True)
await button.click()
# Type into an input field
search_box = await page.select("input[type=search]")
await search_box.send_keys("search query")
# Get element text
heading = await page.select("h1")
title_text = heading.text
# Get element attributes
link = await page.select("a")
href = link.attrs.get("href")
await browser.stop()
The send_keys() method simulates real keyboard input. For file uploads, use send_file() with the full file path.
Flash Elements for Debugging
During development, visually highlight elements to verify your selectors:
async def debug_selectors():
browser = await zd.start(headless=False)
page = await browser.get("https://example.com")
elements = await page.select_all("a")
for elem in elements:
await elem.flash() # Briefly highlights the element
await browser.stop()
The flash() method adds a visual indicator, making it easy to see exactly which elements your selector matches.
Step 5: Handle Dynamic Content and Infinite Scrolling
Many modern websites load content dynamically. Zendriver provides several ways to handle this.
Waiting for Elements
Use timeouts and polling to wait for elements to appear:
import asyncio
import zendriver as zd
async def wait_for_content():
browser = await zd.start()
page = await browser.get("https://example.com")
# Wait for a specific element to load
while True:
element = await page.select("#dynamic-content")
if element:
break
await asyncio.sleep(0.5)
print(element.text)
await browser.stop()
You can create a reusable wait function:
async def wait_for_selector(page, selector, timeout=10):
"""Wait for an element to appear on the page."""
start_time = asyncio.get_event_loop().time()
while True:
element = await page.select(selector)
if element:
return element
if asyncio.get_event_loop().time() - start_time > timeout:
raise TimeoutError(f"Element {selector} not found")
await asyncio.sleep(0.5)
Scrolling Pages
For infinite scroll pages, combine scrolling with content detection:
async def scrape_infinite_scroll():
browser = await zd.start()
page = await browser.get("https://example.com/feed")
all_items = []
previous_count = 0
while True:
# Get current items
items = await page.select_all(".feed-item")
current_count = len(items)
# Check if we've reached the end
if current_count == previous_count:
break
previous_count = current_count
# Scroll down
await page.scroll_down(500)
# Wait for new content to load
await asyncio.sleep(1)
# Extract data from all items
items = await page.select_all(".feed-item")
for item in items:
all_items.append(item.text)
await browser.stop()
return all_items
The scroll_down() method accepts a pixel value. You can also use scroll_up() for navigating back.
Waiting for Page Events
Await the page object itself to wait for events to be processed:
async def wait_for_page_events():
browser = await zd.start()
page = await browser.get("https://example.com")
# Wait for page events to settle
await page
# Now the page is fully loaded
content = await page.get_content()
await browser.stop()
This is useful after actions that trigger JavaScript execution or AJAX requests.
Step 6: Manage Cookies and Browser Profiles
Persistent sessions require cookie and profile management.
Using Persistent Profiles
To maintain login state across sessions, specify a user data directory:
async def persistent_session():
browser = await zd.start(
user_data_dir="/path/to/chrome/profile"
)
page = await browser.get("https://example.com")
# First run: login manually or programmatically
# Subsequent runs: you'll already be logged in
await browser.stop()
When you specify user_data_dir, Zendriver won't delete the profile on exit. This preserves cookies, localStorage, and other session data.
Programmatic Login Example
Here's how to automate a login flow:
async def login_to_site():
browser = await zd.start(user_data_dir="./my_profile")
page = await browser.get("https://example.com/login")
# Fill login form
username_field = await page.select("input[name=username]")
await username_field.send_keys("your_username")
password_field = await page.select("input[name=password]")
await password_field.send_keys("your_password")
# Submit form
submit_button = await page.find("Log in", best_match=True)
await submit_button.click()
# Wait for redirect
await asyncio.sleep(2)
await page
# Verify login success
profile_element = await page.select(".user-profile")
if profile_element:
print("Login successful!")
await browser.stop()
After the first successful login, future runs will skip the login process because cookies are preserved in the profile directory.
Step 7: Intercept Network Requests and API Responses
Zendriver can capture network traffic, which is useful for extracting API data.
Capturing API Responses
Use response expectations to capture specific API calls:
import json
import zendriver as zd
from zendriver.cdp.network import get_response_body
async def capture_api_response():
browser = await zd.start()
page = browser.tabs[0]
# Set up expectation BEFORE navigating
async with page.expect_response("*/api/data.json") as response_expectation:
await page.get("https://example.com/dashboard")
# Get the captured response
response = await response_expectation
# Fetch the response body
body = await page.send(get_response_body(response.request_id))
# Parse JSON data
data = json.loads(body.body)
print(data)
await browser.stop()
The pattern */api/data.json matches any URL ending with that path. You can use wildcards for flexible matching.
Handling Downloads
Capture file downloads with expect_download():
async def handle_download():
browser = await zd.start()
page = await browser.get("https://example.com")
async with page.expect_download() as download_expectation:
# Click download button
download_btn = await page.find("Download", best_match=True)
await download_btn.click()
download = await download_expectation
print(f"Downloaded file: {download.suggested_filename}")
await browser.stop()
This captures the download metadata without actually saving the file to disk.
Advanced: Using CDP Commands Directly
Zendriver provides full access to Chrome DevTools Protocol for advanced use cases.
Sending CDP Commands
Import CDP modules and send commands directly:
import zendriver as zd
from zendriver.cdp import runtime
async def execute_javascript():
browser = await zd.start()
page = await browser.get("https://example.com")
# Enable the runtime domain
await page.send(runtime.enable())
# Execute JavaScript and get result
result = await page.send(
runtime.evaluate(expression="document.title")
)
print(f"Page title: {result.result.value}")
await browser.stop()
CDP commands give you access to 30+ domains including Network, DOM, Input, and Performance.
Listening to CDP Events
Register handlers for browser events:
from zendriver.cdp import console
async def listen_to_console():
browser = await zd.start()
page = await browser.get("https://example.com")
# Enable console domain
await page.send(console.enable())
# Register event handler
def handle_console_message(event):
print(f"Console: {event.message.text}")
page.add_handler(console.MessageAdded, handle_console_message)
# Navigate and interact
await page.get("https://example.com/app")
await browser.stop()
Event handlers run asynchronously. Always enable the domain before registering handlers.
Common Pitfalls and Solutions
Timing Issues
Zendriver is fast—sometimes faster than the JavaScript on the page. Always wait for elements before interacting:
# Bad: Element might not exist yet
element = await page.select(".dynamic-element")
await element.click() # May fail
# Good: Wait for element to appear
element = None
for _ in range(20):
element = await page.select(".dynamic-element")
if element:
break
await asyncio.sleep(0.5)
if element:
await element.click()
Headless Detection
Some sites still detect headless browsers. Run with headless=False when testing:
browser = await zd.start(headless=False)
For production, Zendriver's default settings work for most anti-bot systems. If blocked, try adding random delays between actions to appear more human-like.
Memory Leaks
Always close the browser when done:
try:
browser = await zd.start()
# Your scraping code here
finally:
await browser.stop()
For long-running scripts, periodically restart the browser to clear accumulated memory.
Using Proxies
For large-scale scraping, rotate through residential proxies like those from Roundproxies.com to avoid IP-based blocks:
browser = await zd.start(
browser_args=[
"--proxy-server=http://proxy.example.com:8080"
]
)
Residential proxies from providers like Roundproxies work best because datacenter IPs are often flagged by anti-bot systems.
Final Thoughts
Zendriver gives you a powerful, undetectable web automation framework that's faster than Selenium and more stealth-focused than Playwright.
You learned how to install Zendriver, navigate pages, extract content, handle dynamic loading, manage sessions, and intercept network traffic. These fundamentals cover 90% of scraping use cases.
For protected sites, Zendriver bypasses Cloudflare, CloudFront, and Akamai out of the box. Combine it with proxy rotation for maximum reliability at scale.
The async-first design makes it perfect for concurrent scraping. You can run dozens of tabs simultaneously without the overhead of multiple browser instances.
Start with the basic examples and gradually incorporate advanced features as needed. The official documentation at zendriver.dev has additional tutorials on form handling, account creation, and direct CDP manipulation.
FAQ
Is Zendriver better than Selenium?
Yes, for web scraping Zendriver is superior. It uses Chrome DevTools Protocol instead of WebDriver, making it undetectable by most anti-bot systems. It's also faster due to direct browser communication and native async support.
Can Zendriver bypass Cloudflare?
Zendriver successfully bypasses Cloudflare, CloudFront, and Akamai protections in testing. However, no tool works 100% of the time. Combine it with residential proxies and random delays for the best results against heavily protected sites.
Does Zendriver work in Docker?
Yes. The official zendriver-docker project provides GPU-accelerated Chrome in Docker containers with VNC debugging support. Use --no-sandbox and --disable-dev-shm-usage flags when running in containers.
How is Zendriver different from nodriver?
Zendriver is a community-maintained fork of nodriver. It incorporates unmerged bug fixes, adds static analysis tools, and maintains active community engagement. The API is nearly identical, so migration is straightforward.