Proxy authentication: IP whitelisting vs username:password

Proxy authentication is the gate a provider runs before it carries your traffic, and it works one of two ways: the gateway recognises the IP your connection came from, or your client hands over a username and password on every request. The proxy IP whitelist vs user:pass auth decision turns on a single fact about your setup: whether the machine sending the request keeps a stable public IP.

Your scraper worked from your laptop for three weeks. You deployed it to a VPS on Friday, and the first request came back 407 Proxy Authentication Required. Same code, same proxy, same credentials. The only thing that changed was the machine it runs on.

That bug is the entire topic in miniature. Whitelisting ties access to a place, credentials tie access to a secret, and each one breaks in the situation the other handles fine. A whitelist locks you out the moment your egress IP moves, whether that's a DHCP lease renewing overnight, a cloud instance getting a new address, or a coffee shop network. A password keeps working for anyone who finds it, including whoever reads the Git history, the ticket comment, or the CI log you pasted it into.

Short answer. Whitelist the IPs of fixed hosts you control: production servers, a dedicated VPS, an office egress. Use username:password wherever the source IP moves, which covers laptops, CI runners, serverless jobs, and browser automation on other people's machines. Plenty of teams run both on the same account, and neither one is inherently safer than the other. The right pick depends on which risk is actually yours: a leaked credential or a network you don't control.

The details that bite later are the lookalikes and the tooling. A provider's proxy authentication is a different gate from the IP allowlist a target site keeps, and clients handle credentials unevenly: some read them straight from the proxy URL, others expect a header or a separate configuration step.

IP whitelisting vs username:password: what's the difference?

The main difference between IP whitelisting and username:password proxy authentication is what the proxy checks. IP whitelisting trusts the source address of the connection and sends no credentials at all, while username:password sends a Proxy-Authorization header the proxy checks on every connection. Use whitelisting for servers with a fixed public IP; use credentials anywhere the IP can change.

That's the compressed version. The longer one is that both methods answer the same question, "is this client allowed to use me?", and they just look at different evidence.

Whitelisting looks at the packet. The proxy reads the source IP from the TCP connection, checks it against a list you maintain in a dashboard or an API, and either serves the request or drops it. Nothing about you travels in the HTTP request.

Credentials look at the request. Your client adds a header, the proxy decodes it and compares it against your account. Where the connection came from doesn't matter.

Most providers support both, and a few let you require both at once. Which one you should reach for depends almost entirely on whether the IP your traffic leaves from is something you control.

Terms that get lumped in with proxy authentication

Four things get called "proxy auth" or "whitelisting" in forum threads, and only two of them are the subject of this article. Sorting them out first saves a lot of misdirected debugging.

The third row is the one that causes the most wasted afternoons. "Whitelist my IP" means the opposite thing depending on which side you're standing on. On the proxy side, you're telling the gateway which of your machines may use it. On the target side, someone is telling their server which IPs may reach it, and the proxy's exit IP is usually not on that list.

The fourth row is a different beast again. Providers pack rotation settings, sticky session IDs and country codes into the username string because it's the one field every HTTP client already knows how to send. It rides on top of username:password auth; it isn't a third method. If you're on whitelisting, you typically lose that channel and have to pick the port or gateway host instead. (The full mechanics of session parameters are in the rotating proxies explainer.)

Say you're scraping 10,000 product pages from a fixed VPS. IP whitelisting lets the VPS reach the proxy gateway. Username:password would do the same job if you wanted a sticky session ID in the username. Target-side allowlisting is irrelevant unless the store is somehow only reachable from specific IPs. And the username parameters are how you'd ask for a German exit IP if the store geo-blocks.

How the two proxy authentication methods work on the wire

Understanding the handshake tells you why each method breaks where it does. Two components, two subsections.

1. The 407 challenge and the Proxy-Authorization header

Username:password auth for HTTP proxies is defined in RFC 7235, and the Basic scheme most providers use is RFC 7617. The client is supposed to send a request, get a 407 back with a Proxy-Authenticate: Basic header, and retry with credentials. Most clients skip the first round trip and send the header up front.

Here's what a plain HTTP request through a credentialed proxy looks like at the socket level.

GET http://example.com/ HTTP/1.1
Host: example.com
Proxy-Authorization: Basic dXNlcjpwYXNz
User-Agent: curl/8.6.0

dXNlcjpwYXNz is user:pass in base64. Base64 is an encoding, not encryption, so anyone who can read this request can read your password. Over plain HTTP, that includes every hop between you and the gateway.

HTTPS changes where the header goes, and this is the part almost every explainer skips. For an https:// target, the client first sends a CONNECT request to the proxy, and the credentials ride on that one line.

CONNECT example.com:443 HTTP/1.1
Host: example.com:443
Proxy-Authorization: Basic dXNlcjpwYXNz

After the proxy answers 200 Connection established, the client does its TLS handshake through the tunnel and the proxy can't see anything after that. Two consequences follow. The target site never sees your proxy credentials, so they can't leak that way. And a 407 on an HTTPS request shows up in your logs as a tunnel failure, which is why curl prints Received HTTP code 407 from proxy after CONNECT instead of a normal status line.

Where you'll hit this:

• Any request library that takes a proxy URL with user:pass@ in it
• Browser automation that supports page.authenticate() or a proxy username option
• SOCKS5, which has its own username:password sub-negotiation in RFC 1929 (also cleartext, and sent before any tunnel exists)

How it works in practice: in Python you'd normally let requests build the header for you, but building it by hand makes it obvious what's being sent and lets you retry on 407 deliberately.

import base64, os, requests

user, pw = os.environ["PROXY_USER"], os.environ["PROXY_PASS"]
token = base64.b64encode(f"{user}:{pw}".encode()).decode()

session = requests.Session()
session.proxies = {"http": "http://gw.example:8080", "https": "http://gw.example:8080"}
# requests forwards this header to the proxy, including on the CONNECT for https targets
session.headers["Proxy-Authorization"] = f"Basic {token}"

r = session.get("https://httpbin.org/ip", timeout=15)
print(r.status_code, r.json())

Pull the credentials from the environment, never from a string literal; a proxy URL in source ends up in git history and in the traceback of every exception. If you get ProxyError ... Tunnel connection failed: 407 here, the credentials are wrong or the account expects whitelisting on this port. Nothing about the target is involved yet.

2. Source-IP checks and why your "static" IP isn't

IP whitelisting has no protocol. The proxy accepts the TCP connection, looks at the source address the kernel hands it, and matches it against your list. There's no header to get wrong, no encoding, no 407. That's also why it fails silently: an unlisted IP typically gets a connection reset or a 407 with no useful detail, because from the proxy's point of view you're an anonymous stranger.

The catch is that the address the proxy sees is your egress IP, and that is often not the IP you think it is. A Kubernetes pod leaves through the node or a NAT gateway. A serverless function leaves from a pool. A home connection behind CGNAT shares one public address with hundreds of neighbors, all of whom would then be "whitelisted" too.

After moving a scraper fleet between two cloud regions last year, the thing that bit me wasn't the proxy at all: the new region routed egress through a NAT gateway with a different address than the instance's own public IP, and the whitelist entry I'd copied over was for the instance. Twenty minutes of staring at 407s from a machine that had "the right IP".

Where you'll hit this:

• VPS or dedicated servers with a reserved public IP (the happy path)
• Containers and CI runners, where the egress IP is shared or rotates
• Office networks and residential lines, where the public IP can change on a modem reboot

How it works in practice: before adding anything to a whitelist, ask a third party what address your traffic arrives from, and ask from the exact process that will talk to the proxy.

# Run this from the same machine, container, or CI job that will use the proxy.
# ifconfig.me echoes back the source IP it received the request from.
curl -s https://ifconfig.me
echo

# Check whether that IP is stable: repeat a few times over a few minutes.
for i in 1 2 3; do curl -s https://ifconfig.me; echo; sleep 60; done

If those three lines don't match, whitelisting isn't an option for that environment and you should stop fighting it. If they do match and the proxy still rejects you, check whether the provider uses a separate port for whitelisted traffic; several do, and 407 on the credentials port is the expected answer.

Side by side

Neither column is "more secure" in the abstract. Whitelisting removes the secret and adds a dependency on network topology; credentials remove the topology dependency and add a secret. You're choosing which failure you'd rather manage.

How each tool handles proxy credentials

This table is the reason a lot of people end up on whitelisting even when their infrastructure would allow credentials. The tooling around browsers is uneven.

The Chrome line is the one that matters. --proxy-server="http://user:pass@gw:8080" looks like it should work. Chromium strips the userinfo and never sends it. With Puppeteer you fix that with one call per page; with raw Selenium you write an extension. Playwright is the only one of the three that behaves the way you'd expect.

Here's the Puppeteer version, because it's the one that people search for at 2am.

import puppeteer from "puppeteer";

const browser = await puppeteer.launch({
  args: ["--proxy-server=http://gw.example:8080"],
});
const page = await browser.newPage();

// Must be called before navigation, on every page you create.
await page.authenticate({
  username: process.env.PROXY_USER,
  password: process.env.PROXY_PASS,
});

await page.goto("https://httpbin.org/ip");
console.log(await page.evaluate(() => document.body.innerText));
await browser.close();

If you forget the authenticate call, Chrome reports net::ERR_TUNNEL_CONNECTION_FAILED on HTTPS pages rather than anything mentioning auth, which sends people off to debug TLS. For the full per-framework setup, see the Playwright proxy guide and the Selenium proxy guide, both of which walk through the extension route.

How to choose a proxy authentication method

Four steps. Most people are done after the first two.

1. Find out whether your egress IP is yours

Run the ifconfig.me check from the environment that will actually make requests, not from your laptop. If the address is reserved to you and doesn't share with other tenants, whitelisting is on the table. If it's a NAT pool, CGNAT, or anything that changes on redeploy, it isn't. Don't guess this; the cost of guessing wrong is a production outage the next time the platform reshuffles addresses.

2. Check whether your tooling can send credentials cleanly

Requests, httpx, curl, Playwright, Node agents: yes. Raw Chrome, Selenium, some desktop apps, a lot of mobile proxy settings: painful. If you're in the second group and step 1 said your IP is fixed, whitelist and move on. If step 1 said your IP isn't fixed, budget an hour for the Puppeteer or extension approach rather than fighting the CLI flag.

3. Decide who else needs access

Whitelisting has no concept of "who". A whitelisted office IP means the intern's laptop is authenticated. Credentials can be issued per project or per person and rotated independently, so if you're a team, or you're running several scrapers you'd like to shut off separately, credentials win even on a static IP.

4. Start with the simpler one and don't build a vault yet

If you're one person on one VPS, whitelist the VPS and write the proxy URL with no secrets in it. That's the whole setup. Don't stand up a secrets manager, a credential rotation job and a 407 retry wrapper for a scraper that runs on a cron. Add credentials when you get a second environment, and add a secrets manager when you get a second person. The Python requests proxy tutorial has the minimal version of both setups if you want to copy one.

In practice: three setups

A single VPS running a nightly scraper

Whitelist. The VPS has a reserved IP, one process talks to the proxy, and the code contains only a hostname and port. If you ever need a sticky session, most providers let you select it by port or a separate gateway hostname instead of a username parameter.

A scraper fleet on Kubernetes

Credentials, stored as a Kubernetes Secret and mounted as env vars. Pod egress goes through nodes or a cloud NAT and can change without notice; a whitelist would need every node IP and would break on autoscale. One 407 retry in the client is worth adding here because some gateways challenge the first request after a credential rotation.

A team of three sharing a residential pool

Credentials, one set per person, rotated when someone leaves. Everyone works from home behind ISP-assigned addresses that change on a modem restart, so whitelisting would mean weekly dashboard edits. If one person also runs a fixed server, whitelist that server separately and keep it out of the shared credential.

Pitfalls and how to avoid them

Keep the proxy URL out of the process list

curl -x http://user:pass@gw:8080 ... puts the password in ps aux for every user on that box, and in your shell history. Use -U with --proxy-user read from a file, or the https_proxy environment variable set in a non-logged step, or ~/.curlrc with proxy-user = "user:pass" and chmod 600.

Whitelist the egress address, not the interface address

On cloud instances the two can differ, most often when a NAT gateway or an egress-only load balancer sits in the path. Trust ifconfig.me over ip addr.

Treat a shared public IP as a public proxy

Whitelisting an office, coworking space or CGNAT address means everyone behind that address can route through your account and burn your bandwidth. If you can't get a dedicated egress IP, use credentials, even if the network "feels" private.

Don't send Basic auth over an unencrypted hop you don't control

The header is base64, readable by anything in the path. Between your own server and the gateway over a provider's network it's usually fine; over public Wi-Fi to an http:// gateway it isn't. Prefer an HTTPS proxy endpoint if the provider offers one, or tunnel through SSH. The differences between the transports are in the HTTP vs HTTPS vs SOCKS5 proxies explainer.

Neither method helps once the target blocks you

Proxy authentication decides whether the proxy accepts you. It says nothing about whether the target will. A 403 from the site, a Cloudflare challenge, or a rate-limit page means your exit IP, headers or fingerprint got flagged; switching from whitelisting to credentials or back changes none of that. Debug those with the guide to why scrapers get blocked, not with the auth settings.

FAQ

Is IP whitelisting more secure than username and password?

Neither one is safer across the board. Whitelisting can't leak a password because there isn't one, but it authenticates every machine behind the listed IP, including ones you don't own if the address is shared. Credentials can leak through logs and commits but let you scope and revoke access per user. Use whitelisting for a dedicated server IP and credentials for anything shared or mobile.

Can I use IP whitelisting and username:password at the same time?

Many providers allow it, and some let you require both, so a request has to come from a listed IP and carry valid credentials. That's a reasonable default for production scrapers that handle anything sensitive. Check the provider's docs, because some treat the two as either/or per port rather than as layers.

Why does my proxy return 407 even with the right password?

Four common causes, in order of how often I've seen them: the credentials port and the whitelist port are different and you're on the wrong one; the client is sending the header on the wrong request (Chrome ignoring user:pass@ in the flag is the classic); a special character in the password wasn't URL-encoded; or the account is set to whitelist-only and ignores the header entirely.

What happens if my IP changes while using a whitelist?

Access stops until you add the new address. There's no grace period and usually no descriptive error. If your IP changes more than once a month, move that environment to credentials rather than automating dashboard edits.

Does SOCKS5 support username and password authentication?

Yes, via a sub-negotiation defined in RFC 1929. The credentials are sent in cleartext before any application traffic, so the same transport cautions as HTTP Basic apply. Most providers that offer SOCKS5 also let you whitelist for it.

Wrapping up

The mental model that survives contact with production: whitelisting authenticates a place, credentials authenticate a thing. If the place is fixed and yours alone, whitelist it and enjoy a config with no secrets. The moment the place moves or gets shared, switch to credentials and manage the secret properly.

First thing to go do: run the ifconfig.me check from the environment you're about to deploy to. That single number decides most of the rest. When you're ready to wire it up, the Python requests proxy tutorial has both variants ready to paste.