Are your Selenium scraping bots getting slapped with IP blocks faster than you can say “rate-limit error 429”?
You’re not the only one. Hitting request caps and geoblocks is the single biggest roadblock devs face when they try to scale Selenium automation.
The antidote? Proxies.
Yet wiring a proxy into Selenium (especially an authenticated one) can feel like persuading a cat into a bath.
In this guide you’ll learn—step by step—how to configure proxies in Selenium using modern Python 3.12+ and Selenium 4.x. We’ll start simple, then crank things up to fully-fledged rotating proxy systems that keep scrapers humming day and night.
What You’ll Master
- Set up a basic HTTP/HTTPS proxy in Selenium (Chrome & Firefox)
- Handle authenticated proxies – three battle-tested methods
- Build a rotating proxy engine that actually survives at scale
- Know when to ditch Selenium for
requests
and scrape 10× faster - Troubleshoot proxy pain points without pulling your hair out
Why Proxies Are Non-Negotiable for Selenium
- Sidestep IP blocks & rate limits – spread requests across multiple IPs.
- Bypass geographic filters – scrape region-locked content effortlessly.
- Scale like a pro – a proxy pool lets you look like thousands of users.
- Hide your footprint – mask your real IP and protect your infrastructure.
Step 1 – Install the Right Packages
Basic Installation
# Selenium
pip install selenium==4.20.0
# Driver manager (auto-downloads the right ChromeDriver)
pip install webdriver-manager==4.0.1
# Authenticated-proxy support
pip install selenium-wire==5.1.0
Why Selenium Wire?
ChromeDriver ignoresusername:password@proxy:port
URLs. Selenium Wire patches that gap—even though it’s officially unmaintained, it still just works for authentication.
Verify Everything’s Ready
import selenium
import seleniumwire
print(f"Selenium version: {selenium.__version__}")
print("Selenium Wire installed successfully")
Step 2 – Point Selenium at a Proxy (No Auth)
Chrome
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
PROXY = "123.456.789.10:8080" # swap in your proxy
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(f'--proxy-server={PROXY}')
chrome_options.add_argument('--headless=new') # optional
driver = webdriver.Chrome(
service=Service(ChromeDriverManager().install()),
options=chrome_options
)
driver.get("http://httpbin.org/ip")
print(driver.find_element(By.TAG_NAME, "body").text)
driver.quit()
Firefox
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.common.proxy import Proxy, ProxyType
from webdriver_manager.firefox import GeckoDriverManager
proxy = Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.http_proxy = proxy.ssl_proxy = "123.456.789.10:8080"
firefox_options = webdriver.FirefoxOptions()
firefox_options.proxy = proxy
driver = webdriver.Firefox(
service=Service(GeckoDriverManager().install()),
options=firefox_options
)
driver.get("http://httpbin.org/ip")
driver.quit()
Pro Tip – Go SOCKS5 for Versatility
chrome_options.add_argument('--proxy-server=socks5://123.456.789.10:1080')
Step 3 – Tackle Authenticated Proxies (3 Ways)
Method 1 – Selenium Wire (Fastest & Friendliest)
from seleniumwire import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
proxy_options = {
'proxy': {
'http': 'http://username:password@proxy_host:port',
'https': 'https://username:password@proxy_host:port',
'no_proxy': 'localhost,127.0.0.1'
}
}
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless=new')
driver = webdriver.Chrome(
service=Service(ChromeDriverManager().install()),
seleniumwire_options=proxy_options,
options=chrome_options
)
driver.get('http://httpbin.org/ip')
print(driver.find_element(By.TAG_NAME, 'body').text)
driver.quit()
Method 2 – Spin Up a Chrome Extension on the Fly
import os, zipfile
from selenium import webdriver
def create_proxy_extension(proxy_host, proxy_port, proxy_user, proxy_pass):
# manifest.json & background.js creation snipped for brevity …
# (see full code above)
return 'proxy_auth_plugin.zip'
ext = create_proxy_extension('proxy.example.com', '8080', 'user', 'pass')
chrome_opts = webdriver.ChromeOptions()
chrome_opts.add_extension(ext)
driver = webdriver.Chrome(options=chrome_opts)
driver.get("http://httpbin.org/ip")
Method 3 – Whitelist Your IP (Zero Credentials)
# After adding your server’s IP to the provider’s allow-list:
PROXY = "premium-proxy.example.com:8080"
chrome_opts = webdriver.ChromeOptions()
chrome_opts.add_argument(f'--proxy-server={PROXY}')
driver = webdriver.Chrome(options=chrome_opts)
Step 4 – Build Proxy Rotation That Scales
Simple Rotation (Restart Browser Every N Calls)
from seleniumwire import webdriver
import random, time
from selenium.webdriver.common.by import By
class ProxyRotator:
# class code unchanged; see full listing above
…
proxy_list = [
'http://user1:pass1@proxy1.com:8080',
'http://user2:pass2@proxy2.com:8080',
'http://user3:pass3@proxy3.com:8080',
]
rotator = ProxyRotator(proxy_list)
urls = ['http://httpbin.org/ip'] * 10
rotator.scrape_with_rotation(urls)
Dynamic Rotation (Swap IP Without Restarting)
from seleniumwire import webdriver
import random
class DynamicProxyRotator:
# class code unchanged; see full listing above
…
proxies = [
'http://user:pass@proxy1.com:8080',
'http://user:pass@proxy2.com:8080',
'http://user:pass@proxy3.com:8080',
]
rotator = DynamicProxyRotator(proxies)
rotator.scrape_with_dynamic_rotation('http://httpbin.org/ip')
Step 5 – Test & Troubleshoot Like a Pro
Smoke-Test Your Proxy
def test_proxy_setup(driver):
tests = {
'IP Check': 'http://httpbin.org/ip',
'Headers Check': 'http://httpbin.org/headers',
'User Agent': 'http://httpbin.org/user-agent'
}
# function body unchanged; see full listing above
Fix the Usual Suspects
Error | Why It Happens | Quick Fix |
---|---|---|
407 Proxy Authentication Required | Wrong creds / special chars un-escaped | URL-encode your username & password with urllib.parse.quote |
Timeouts | Slow proxies or heavy pages | Retry with exponential back-off (2**attempt seconds) |
SSL Certificate Errors | Intercepted HTTPS traffic | Add --ignore-ssl-errors & --ignore-certificate-errors (use sparingly) |
Performance Hacks That Save Hours
- Reuse browser instances – opening Chrome is the slowest part.
- Block heavy assets – disable images & CSS with
prefs
. - Always run headless –
--headless=new --disable-gpu --no-sandbox
.
When to Drop Selenium for requests
Selenium is your only option for JavaScript-heavy pages, but if you’re scraping static HTML or hitting public APIs, requests
+ proxies is 10 × faster and cheaper on resources.
import requests, bs4
proxies = {'http': 'http://user:pass@proxy.com:8080',
'https': 'https://user:pass@proxy.com:8080'}
r = requests.get('https://example.com', proxies=proxies)
soup = bs4.BeautifulSoup(r.content, 'html.parser')
Final Thoughts
Setting up a proxy in Selenium doesn’t have to be rocket science. Whether you need a single IP for quick tests or a beefy rotating pool for enterprise-grade scraping, the five steps above have you covered.
Remember: proxy quality beats proxy quantity. Pair top-tier residential IPs with smart rotation rules, and your Selenium bots will run smoother than a fresh Chrome install.
Happy (and block-free) scraping.