Cloudflare blocks roughly 40% of the websites you want to scrape. Your Python requests hit a wall, returning 403 errors or challenge pages instead of data.
Cloudscraper solves this problem without launching heavy browser instances. It works like Python Requests but automatically handles Cloudflare's JavaScript challenges behind the scenes.
In this guide, you'll learn how to install Cloudscraper, configure browser profiles, integrate proxies, and handle the edge cases that trip up most scrapers.
What Is Cloudscraper and How Does It Bypass Cloudflare?
Cloudscraper is a Python library that bypasses Cloudflare's anti-bot protection by impersonating a real web browser. It uses optimized HTTP headers and a JavaScript interpreter to solve Cloudflare's "I'm Under Attack Mode" challenges automatically. The library waits approximately 5 seconds on your first request to complete the challenge, then reuses session cookies for subsequent requests without delay.
This approach works because Cloudflare's basic protection relies on JavaScript execution to verify real browsers. Cloudscraper runs these scripts using interpreters like Node.js or js2py, making your requests appear legitimate.
However, Cloudscraper has limitations. It struggles with Cloudflare's newer Bot Detection v2 and v3 systems that use advanced fingerprinting beyond simple JavaScript challenges.
How Cloudflare Detects and Blocks Scrapers
Before diving into Cloudscraper configuration, understanding Cloudflare's detection layers helps you avoid common mistakes.
Cloudflare assigns each request a "bot score" based on multiple signals. Lower scores mean higher suspicion. Here's what triggers blocks:
IP Reputation: Requests from data center IPs, VPNs, or previously flagged addresses get scrutinized heavily. Residential IPs pass more easily.
TLS Fingerprinting: Cloudflare examines how your client establishes secure connections. Python's default TLS handshake differs from Chrome's, which raises red flags.
HTTP Headers: Missing or mismatched browser headers signal automation. A Chrome User-Agent with Firefox-specific headers gets blocked immediately.
JavaScript Challenges: Cloudflare injects scripts that require execution. Standard HTTP libraries can't run JavaScript, triggering the infamous "Checking your browser" page.
Request Patterns: Scraping at machine speeds or hitting endpoints in unusual sequences flags your session.
Cloudscraper addresses the JavaScript challenge and HTTP header issues directly. For other detection methods, you'll need proxies and careful request timing.
Installing Cloudscraper
Getting Cloudscraper running takes one command:
pip install cloudscraper
The library depends on Python Requests and js2py for JavaScript execution. Both install automatically.
For better JavaScript performance, install Node.js separately. Cloudscraper detects and uses it automatically:
# Ubuntu/Debian
sudo apt-get install nodejs
# macOS
brew install node
# Windows
# Download from nodejs.org
Node.js handles complex challenges faster than the pure Python js2py interpreter.
Verify your installation works:
import cloudscraper
scraper = cloudscraper.create_scraper()
response = scraper.get("https://httpbin.org/headers")
print(response.status_code)
A 200 status confirms everything installed correctly.
Basic Usage: Your First Cloudscraper Request
Cloudscraper mirrors Python Requests syntax exactly. If you've used Requests before, you already know how to use Cloudscraper.
Here's a minimal example:
import cloudscraper
# Create scraper instance
scraper = cloudscraper.create_scraper()
# Make request exactly like requests.get()
response = scraper.get("https://example-cloudflare-site.com")
print(response.status_code)
print(response.text[:500])
The create_scraper() function returns a session object. Use it for all requests to maintain cookies and avoid repeating Cloudflare challenges.
For POST requests, the syntax stays identical:
scraper = cloudscraper.create_scraper()
data = {"username": "user", "query": "search term"}
response = scraper.post("https://example.com/api", data=data)
JSON payloads work the same way:
import json
payload = {"key": "value"}
response = scraper.post(
"https://example.com/api",
json=payload,
headers={"Content-Type": "application/json"}
)
The first request to any Cloudflare-protected site triggers a ~5 second delay while Cloudscraper solves the challenge. Subsequent requests use cached session cookies and complete instantly.
Configuring Browser Profiles and User-Agents
Default settings work for many sites, but some require specific browser configurations to bypass Cloudflare successfully.
Configure browser emulation when creating your scraper:
import cloudscraper
scraper = cloudscraper.create_scraper(
browser={
'browser': 'chrome',
'platform': 'windows',
'desktop': True,
'mobile': False
}
)
Available configuration options:
| Parameter | Values | Default |
|---|---|---|
| browser | chrome, firefox |
Random |
| platform | linux, windows, darwin, android, ios |
Random |
| desktop | True, False |
True |
| mobile | True, False |
False |
Mobile emulation sometimes bypasses stricter desktop protections:
scraper = cloudscraper.create_scraper(
browser={
'browser': 'chrome',
'platform': 'android',
'desktop': False,
'mobile': True
}
)
For complete control, set a custom User-Agent:
scraper = cloudscraper.create_scraper(
browser={
'custom': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
}
)
Cloudscraper attempts to match your custom string against known browser signatures. Matching signatures get appropriate headers and ciphers applied automatically.
Choosing the Right JavaScript Interpreter
Cloudscraper supports multiple JavaScript engines for solving Cloudflare challenges. Your choice affects speed and compatibility.
js2py (Default): Pure Python implementation. No external dependencies but slower on complex challenges. Comes installed with Cloudscraper.
Node.js (Recommended): Fastest option. Handles modern JavaScript challenges better. Requires separate installation.
ChakraCore: Microsoft's JavaScript engine. Alternative when Node.js isn't available.
V8: Google's engine via v8eval Python module. Powerful but complex setup.
Specify your interpreter explicitly:
scraper = cloudscraper.create_scraper(interpreter='nodejs')
Test different interpreters if one fails on your target site:
interpreters = ['nodejs', 'js2py', 'native']
for interp in interpreters:
try:
scraper = cloudscraper.create_scraper(interpreter=interp)
response = scraper.get(target_url)
if response.status_code == 200:
print(f"Success with {interp}")
break
except Exception as e:
print(f"Failed with {interp}: {e}")
Node.js handles most challenges best, but js2py occasionally works when Node.js doesn't.
Using Proxies with Cloudscraper
Proxy integration follows standard Python Requests patterns:
import cloudscraper
scraper = cloudscraper.create_scraper()
proxies = {
'http': 'http://user:pass@proxy.example.com:8080',
'https': 'http://user:pass@proxy.example.com:8080'
}
response = scraper.get("https://target-site.com", proxies=proxies)
Critical rule: Keep the same proxy throughout your session. Cloudflare ties challenge solutions to specific IP addresses. Switching proxies mid-session triggers new challenges or blocks.
For rotating proxies, create new scraper instances:
proxy_list = [
'http://proxy1.example.com:8080',
'http://proxy2.example.com:8080',
'http://proxy3.example.com:8080'
]
for proxy in proxy_list:
scraper = cloudscraper.create_scraper()
proxies = {'http': proxy, 'https': proxy}
try:
response = scraper.get(target_url, proxies=proxies)
if response.status_code == 200:
# Use this scraper for subsequent requests
break
except Exception:
continue
Residential proxies from providers like Roundproxies.com work best for Cloudflare-protected sites. Data center IPs often get blocked regardless of your Cloudscraper configuration.
SOCKS proxies work too:
proxies = {
'http': 'socks5://user:pass@proxy.example.com:1080',
'https': 'socks5://user:pass@proxy.example.com:1080'
}
Install PySocks first: pip install pysocks
Handling CAPTCHAs with Third-Party Solvers
When Cloudflare escalates to CAPTCHA challenges, Cloudscraper integrates with solving services:
scraper = cloudscraper.create_scraper(
captcha={
'provider': '2captcha',
'api_key': 'your_2captcha_api_key'
}
)
Supported CAPTCHA services:
- 2captcha
- anticaptcha
- CapSolver
- CapMonster Cloud
- deathbycaptcha
- 9kw
Each service charges per solve. Configure credentials according to their documentation:
# anticaptcha example
scraper = cloudscraper.create_scraper(
captcha={
'provider': 'anticaptcha',
'api_key': 'your_anticaptcha_key'
}
)
# CapSolver example
scraper = cloudscraper.create_scraper(
captcha={
'provider': 'capsolver',
'api_key': 'your_capsolver_key'
}
)
CAPTCHA challenges indicate heightened protection. Consider whether the site's terms allow automated access before proceeding.
Adjusting Challenge Delays
Cloudflare's standard challenge requires approximately 5 seconds before submission. Cloudscraper handles this automatically, but you can override the timing:
scraper = cloudscraper.create_scraper(delay=10)
Increase delays when sites use extended challenge timers. Some implementations wait 10-15 seconds.
Shorter delays risk premature submission and failed challenges:
# Risky - might fail on some sites
scraper = cloudscraper.create_scraper(delay=3)
Debug mode helps identify timing issues:
scraper = cloudscraper.create_scraper(debug=True)
This logs challenge detection and solving attempts to help troubleshoot failures.
Session Management and Cookie Extraction
For integration with other tools, extract Cloudscraper's session cookies:
scraper = cloudscraper.create_scraper()
response = scraper.get("https://cloudflare-protected-site.com")
# Get cookies as dictionary
cookies = scraper.cookies.get_dict()
print(cookies)
# Get specific Cloudflare cookies
cf_clearance = cookies.get('cf_clearance')
The cf_clearance cookie proves you passed the challenge. Use it with the matching User-Agent in other HTTP clients:
import requests
# Use cookies from cloudscraper in regular requests
session = requests.Session()
session.cookies.update(cookies)
session.headers.update({'User-Agent': scraper.headers['User-Agent']})
# Now requests work without Cloudscraper
response = session.get("https://cloudflare-protected-site.com/api/data")
Important: Cloudflare validates that the cookie's User-Agent matches subsequent requests. Always pair extracted cookies with the exact User-Agent string.
For programmatic cookie extraction:
tokens, user_agent = cloudscraper.get_tokens("https://target-site.com")
print(f"Cookies: {tokens}")
print(f"User-Agent: {user_agent}")
Or as a formatted cookie header:
cookie_string, user_agent = cloudscraper.get_cookie_string("https://target-site.com")
print(f"Cookie header: {cookie_string}")
Troubleshooting Common Cloudscraper Errors
CloudflareChallengeError: Detected a Cloudflare version 2 challenge
This error means the site uses Bot Detection v2, which Cloudscraper's open-source version can't handle. Options:
- Try different browser profiles
- Use residential proxies
- Switch to headless browser tools like Playwright
# Sometimes mobile profiles bypass v2
scraper = cloudscraper.create_scraper(
browser={
'browser': 'chrome',
'platform': 'android',
'mobile': True
}
)
403 Forbidden despite apparent success
The challenge solved but subsequent requests fail. Usually a cookie or User-Agent mismatch:
# Ensure consistent headers across requests
scraper = cloudscraper.create_scraper()
scraper.headers.update({
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5'
})
Connection timeouts during challenges
Increase timeout values:
response = scraper.get(url, timeout=30)
JavaScript interpreter errors
Switch interpreters or ensure Node.js is properly installed:
node --version # Should return version number
If Node.js isn't found, Cloudscraper falls back to js2py automatically.
When Cloudscraper Won't Work
Cloudscraper fails against:
Cloudflare Bot Management v2/v3: Advanced fingerprinting beyond JavaScript challenges. Sites like G2.com use this level of protection.
Turnstile CAPTCHAs: Cloudflare's newer invisible challenge system requires real browser execution.
Heavy TLS fingerprinting: Some configurations detect Python's TLS stack regardless of other protections.
Rate limiting: Too many requests from one IP triggers blocks that Cloudscraper can't bypass.
For these cases, consider:
- Playwright with stealth plugins: Full browser automation with anti-detection
- FlareSolverr: Proxy server that uses Puppeteer to solve challenges
- Commercial scraping APIs: Services that maintain their own bypass infrastructure
Cloudscraper works best against standard "I'm Under Attack Mode" challenges, which still protect millions of websites.
Complete Working Example
Here's a production-ready scraper combining best practices:
import cloudscraper
from bs4 import BeautifulSoup
import time
def create_scraper_with_fallback():
"""Try multiple configurations until one works."""
configs = [
{'browser': 'chrome', 'platform': 'windows', 'desktop': True},
{'browser': 'chrome', 'platform': 'android', 'mobile': True},
{'browser': 'firefox', 'platform': 'linux', 'desktop': True}
]
for config in configs:
try:
scraper = cloudscraper.create_scraper(
browser=config,
interpreter='nodejs'
)
return scraper
except Exception:
continue
# Final fallback with defaults
return cloudscraper.create_scraper()
def scrape_with_retry(url, max_retries=3):
"""Scrape URL with automatic retry on failure."""
scraper = create_scraper_with_fallback()
for attempt in range(max_retries):
try:
response = scraper.get(url, timeout=30)
if response.status_code == 200:
return response
print(f"Attempt {attempt + 1}: Status {response.status_code}")
time.sleep(5)
except Exception as e:
print(f"Attempt {attempt + 1} failed: {e}")
time.sleep(5)
return None
# Usage
url = "https://example-cloudflare-site.com"
response = scrape_with_retry(url)
if response:
soup = BeautifulSoup(response.text, 'html.parser')
# Extract your data here
title = soup.find('title')
print(f"Page title: {title.text if title else 'Not found'}")
else:
print("All attempts failed")
This implementation tries multiple browser profiles, uses Node.js for better challenge solving, and includes retry logic for transient failures.
Conclusion
Cloudscraper handles Cloudflare's basic JavaScript challenges effectively without the overhead of browser automation. Install it with pip, create a scraper instance, and make requests like you would with Python Requests.
For best results, use residential proxies, match browser profiles to your target site, and maintain consistent sessions. When sites deploy advanced Bot Management, consider browser automation tools instead.
The library saves significant development time compared to building challenge solvers from scratch. Just remember its limitations against newer Cloudflare protections.
FAQ
Does Cloudscraper work with all Cloudflare-protected sites?
Cloudscraper works against standard Cloudflare "I'm Under Attack Mode" and basic JavaScript challenges. It fails against Cloudflare Bot Management v2/v3, Turnstile CAPTCHAs, and sites with heavy TLS fingerprinting. Test your target site before building extensive scrapers around it.
Why does my first request take 5 seconds?
Cloudflare requires browsers to wait approximately 5 seconds before submitting challenge answers. Cloudscraper mimics this behavior on your first request, then reuses the session cookie for instant subsequent requests.
Can I rotate proxies with Cloudscraper?
You must keep the same proxy throughout a session because Cloudflare ties challenge solutions to IP addresses. Create new scraper instances when switching proxies. Each new instance solves the challenge fresh with its assigned IP.
Is using Cloudscraper legal?
The library itself is legal software. However, scraping websites may violate their terms of service. Review target site policies before automated access. Copyright laws and computer access regulations vary by jurisdiction.
