Apify's built-in proxy service works great for most scraping jobs. But sometimes you need more control.
Maybe you already pay for premium proxies from another provider. Maybe you need specific IPs for geo-targeting. Or maybe you just want to cut costs on a tight budget.
Whatever the reason, Apify lets you plug in your own proxy servers. This guide shows you exactly how to do it—both through the console and programmatically with the SDK.
What Are Custom Proxies in Apify?
Custom proxies in Apify are third-party proxy servers you own or subscribe to, separate from Apify's native proxy service. Instead of relying on Apify Proxy's managed pool, you feed your own proxy URLs directly into your Actors.
This gives you full control over which IPs handle your requests.
Apify supports any HTTP/HTTPS proxy that follows the standard URL format:
http://username:password@host:port
The platform rotates through your proxy list automatically. Each request goes through a different proxy from your pool, mimicking real user behavior.
Why Use Your Own Proxies Instead of Apify Proxy?
Apify Proxy is solid. It handles rotation, health checks, and session management automatically. But here's when your own proxies make more sense:
Cost optimization. If you already pay for a proxy service (like Roundproxies residential or datacenter plans), why pay twice?
Specific IP requirements. Some targets whitelist certain IPs. Your own dedicated proxies can stay on that whitelist.
Geographic precision. Your provider might offer better coverage in specific regions than Apify's pool.
Privacy concerns. Keeping proxy traffic on infrastructure you control adds a layer of data security.
Method 1: Configure Custom Proxies in Apify Console
The fastest way to use custom proxies with Apify is through the web interface. No coding required.
Step 1: Open Your Actor
Log into Apify Console and navigate to your Actor. Click on Input and options in the left sidebar.
Step 2: Find Proxy Configuration
Scroll down until you see Proxy and browser configuration. Click to expand this section.
Step 3: Select Custom Proxies
You'll see options for Apify Proxy and custom proxies. Click on Custom proxies or Own proxies (depending on the Actor).
Step 4: Enter Your Proxy URLs
Add your proxy URLs in the standard format:
http://username:password@proxy1.example.com:8080
http://username:password@proxy2.example.com:8080
http://username:password@proxy3.example.com:8080
Each URL goes on a new line. The Actor will rotate through them automatically.
Step 5: Save and Run
Hit Save & Start to run your Actor with the custom proxy configuration.
That's it. Your scraper now routes all traffic through your own proxies.
Method 2: Use Custom Proxies With the JavaScript SDK
For programmatic control, the Apify SDK gives you more flexibility. Here's how to set up custom proxies in JavaScript.
Basic Setup
The proxyUrls option accepts an array of your proxy URLs:
import { Actor } from 'apify';
import { CheerioCrawler } from 'crawlee';
await Actor.init();
const proxyConfiguration = await Actor.createProxyConfiguration({
proxyUrls: [
'http://user:pass@proxy1.example.com:8080',
'http://user:pass@proxy2.example.com:8080',
'http://user:pass@proxy3.example.com:8080',
],
});
const crawler = new CheerioCrawler({
proxyConfiguration,
async requestHandler({ $, request }) {
const title = $('title').text();
console.log(`Page title: ${title}`);
},
});
await crawler.run(['https://example.com']);
await Actor.exit();
The SDK rotates through your proxy list in round-robin order. Each request gets a different proxy from the pool.
Works With Any Crawler
This same pattern works with PuppeteerCrawler and PlaywrightCrawler:
import { PuppeteerCrawler } from 'crawlee';
const crawler = new PuppeteerCrawler({
proxyConfiguration,
async requestHandler({ page }) {
await page.screenshot({ path: 'screenshot.png' });
},
});
Just pass the proxyConfiguration object and you're set.
Method 3: Use Custom Proxies With the Python SDK
The Python SDK follows a similar pattern. Here's the equivalent setup.
Basic Configuration
import asyncio
from apify import Actor
async def main():
async with Actor:
proxy_configuration = await Actor.create_proxy_configuration(
proxy_urls=[
'http://user:pass@proxy1.example.com:8080',
'http://user:pass@proxy2.example.com:8080',
'http://user:pass@proxy3.example.com:8080',
],
)
if not proxy_configuration:
raise RuntimeError('Proxy configuration failed')
proxy_url = await proxy_configuration.new_url()
Actor.log.info(f'Using proxy: {proxy_url}')
if __name__ == '__main__':
asyncio.run(main())
Using With HTTP Libraries
Once you have a proxy URL, you can use it with any HTTP library:
import httpx
from apify import Actor
async def main():
async with Actor:
proxy_cfg = await Actor.create_proxy_configuration(
proxy_urls=[
'http://user:pass@proxy1.example.com:8080',
'http://user:pass@proxy2.example.com:8080',
],
)
proxy_url = await proxy_cfg.new_url()
async with httpx.AsyncClient(proxy=proxy_url) as client:
response = await client.get('https://httpbin.org/ip')
Actor.log.info(f'Response: {response.json()}')
if __name__ == '__main__':
asyncio.run(main())
This works identically with requests, aiohttp, or any other HTTP library that supports proxies.
Advanced: Dynamic Proxy Generation With Custom Functions
What if your proxy provider uses a special URL format? Or you need to generate proxies dynamically based on the request?
Both SDKs support custom URL functions.
JavaScript Custom Function
const proxyConfiguration = await Actor.createProxyConfiguration({
newUrlFunction: (sessionId) => {
// Generate proxy URL dynamically
const baseUrl = 'http://gate.myproxy.com';
const port = 10000 + Math.floor(Math.random() * 100);
if (sessionId) {
return `${baseUrl}:${port}?session=${sessionId}`;
}
return `${baseUrl}:${port}`;
},
});
Python Custom Function
async def custom_proxy_generator(session_id=None, request=None):
base = 'http://gate.myproxy.com'
if session_id:
return f'{base}:10000?session={session_id}'
return f'{base}:10000'
proxy_cfg = await Actor.create_proxy_configuration(
new_url_function=custom_proxy_generator,
)
This approach is perfect for providers with rotating gateway endpoints or session-based authentication.
Session Management With Custom Proxies
Sessions let you maintain the same IP across multiple requests. This is critical for scraping sites that track user sessions.
How Sessions Work
When you call new_url() with a session ID, the SDK maps that ID to a specific proxy:
# First call with session 'user-123'
proxy_url = await proxy_cfg.new_url('user-123') # Returns proxy1.example.com
# Second call with same session
proxy_url = await proxy_cfg.new_url('user-123') # Returns proxy1.example.com (same)
# Different session
proxy_url = await proxy_cfg.new_url('user-456') # Returns proxy2.example.com
The same session ID always returns the same proxy. Different IDs rotate through your pool.
Enable Sessions in Crawlers
For CheerioCrawler and similar crawlers, enable the session pool:
const crawler = new PuppeteerCrawler({
proxyConfiguration,
useSessionPool: true,
persistCookiesPerSession: true,
async requestHandler({ page, session }) {
// Same session = same proxy = same cookies
console.log(`Session ID: ${session.id}`);
},
});
This keeps cookies and IP addresses consistent across a scraping session.
Debugging Your Custom Proxy Setup
Things don't always work the first time. Here's how to verify your proxies are actually being used.
Check proxyInfo in Your Crawler
The crawler context includes a proxyInfo object with details about the current proxy:
const crawler = new CheerioCrawler({
proxyConfiguration,
async requestHandler({ $, request, proxyInfo }) {
console.log('Proxy URL:', proxyInfo.url);
console.log('Proxy hostname:', proxyInfo.hostname);
console.log('Proxy port:', proxyInfo.port);
},
});
These logs confirm which proxy handled each request.
Test Proxy Connection
Before scraping your target, test against a service that returns your IP:
await crawler.run(['https://httpbin.org/ip']);
The response shows the IP address of the proxy, not your server's IP. If you see your real IP, the proxy configuration isn't working.
Common Issues
Authentication errors. Double-check your username and password. Special characters might need URL encoding.
Connection timeouts. The proxy server might be down. Test it directly with curl or a browser extension.
Wrong format. Make sure your URLs include the protocol (http:// or https://).
Blocked proxies. Your target site might have already blacklisted those IPs. Try different proxies.
Proxy URL Format Reference
Different proxy providers use different formats. Here's a quick reference:
Standard HTTP Proxy
http://username:password@host:port
IP Authentication (No Password)
http://host:port
Your IP must be whitelisted in the proxy provider's dashboard.
Rotating Gateway With Sessions
http://username-session-abc123:password@gate.provider.com:10000
Session identifiers are often embedded in the username.
Country-Specific Proxies
http://username-country-us:password@gate.provider.com:10000
Check your provider's documentation for their exact format.
When to Use Apify Proxy vs Custom Proxies
Here's a quick decision framework:
Use Apify Proxy when:
- You want automatic rotation and health monitoring
- You need access to millions of residential IPs
- You don't have existing proxy infrastructure
- You prefer a single vendor solution
Use custom proxies when:
- You already pay for premium proxy services
- You need specific dedicated IPs
- Your target requires whitelisted addresses
- You want to avoid additional proxy costs
Both approaches work well. The best choice depends on your existing setup and budget.
Final Thoughts
Using custom proxies with Apify isn't complicated. The console method takes about two minutes. The SDK approach adds maybe ten lines of code.
The hardest part is usually getting proxy URLs from your provider in the right format. Once that's sorted, Apify handles the rotation automatically.
Start with a small pool of 3-5 proxies for testing. Verify they work with httpbin or a similar service. Then scale up once everything runs smoothly.
If you're looking for reliable residential or datacenter proxies to use with Apify, check out our own proxies for affordable options with solid geographic coverage.
Error Handling for Custom Proxies
Custom proxies don't have Apify's built-in health monitoring. You need to handle failures yourself.
Retry Failed Requests
Wrap your requests in try-catch blocks and implement retries:
const crawler = new CheerioCrawler({
proxyConfiguration,
maxRequestRetries: 3,
async requestHandler({ $, request }) {
// Your scraping logic
},
async failedRequestHandler({ request, error }) {
console.log(`Request failed: ${request.url}`);
console.log(`Error: ${error.message}`);
},
});
The crawler automatically retries failed requests up to the limit you set.
Validate Proxy Before Scraping
Test your proxy pool before starting long scraping jobs:
async def validate_proxies(proxy_urls):
working = []
for url in proxy_urls:
try:
async with httpx.AsyncClient(proxy=url, timeout=10) as client:
response = await client.get('https://httpbin.org/ip')
if response.status_code == 200:
working.append(url)
except Exception:
Actor.log.warning(f'Proxy failed: {url}')
return working
This filters out dead proxies before they cause problems.
Performance Optimization Tips
Getting the most out of your custom proxy setup requires some tuning.
Match Pool Size to Concurrency
If your crawler runs 10 concurrent requests but you only have 3 proxies, each proxy handles multiple simultaneous connections. Some providers throttle this.
Rule of thumb: Have at least as many proxies as your max concurrency setting.
Monitor Proxy Response Times
Slow proxies drag down your entire scraping operation. Log response times and remove consistently slow proxies:
async requestHandler({ request, proxyInfo }) {
const start = Date.now();
// ... your scraping code
const duration = Date.now() - start;
if (duration > 5000) {
console.log(`Slow proxy: ${proxyInfo.hostname} took ${duration}ms`);
}
}
Rotate Proxy Providers
Don't put all your eggs in one basket. Mix proxies from different providers to reduce the risk of complete pool failure.
Proxy Types Explained
Different proxy types work better for different targets. Here's what to consider.
Datacenter Proxies
Fast and cheap. IPs come from cloud data centers.
Good for: High-volume scraping of non-protected sites. Bad for: Sites with advanced bot detection.
Residential Proxies
IPs from real home internet connections. Much harder to detect.
Good for: Bypassing strict anti-bot systems. Bad for: Budget-conscious projects (they cost more).
ISP Proxies
Datacenter IPs registered as residential. The best of both worlds.
Good for: Balancing speed and detection resistance. Bad for: Extremely tight budgets.
Mobile Proxies
IPs from mobile carrier networks. Highest trust level.
Good for: The most aggressive anti-bot systems. Bad for: Everything else (expensive and often slow).
Match your proxy type to your target's protection level. Don't waste expensive residential proxies on sites that barely check for bots.
FAQ
Can I mix Apify Proxy with my own custom proxies?
No, not in the same Actor run. The ProxyConfiguration class operates in either Apify Proxy mode or custom proxy mode—not both simultaneously. You'd need separate Actor runs for each proxy source.
How many custom proxies should I use?
It depends on your target site and request volume. For light scraping, 5-10 proxies often work fine. For aggressive scraping at scale, you might need hundreds to avoid rate limits.
Do custom proxies cost less than Apify Proxy?
Sometimes. If you already have a proxy subscription, using those proxies with Apify costs nothing extra. But if you're starting fresh, Apify's per-request pricing might be cheaper than maintaining your own proxy infrastructure.
What happens if one of my custom proxies goes down?
The SDK will eventually cycle to the next proxy in your list. But there's no automatic health checking like Apify Proxy provides. Dead proxies might cause some requests to fail before rotation kicks in.
Can I use SOCKS5 proxies with Apify?
The standard ProxyConfiguration expects HTTP/HTTPS proxies. For SOCKS5 support, you'd need to handle proxy connections manually using a library like socks or run a local HTTP-to-SOCKS5 bridge.
How do I format proxies with special characters in passwords?
URL-encode special characters. For example, p@ss becomes p%40ss. Most programming languages have built-in URL encoding functions.
