> ## Content Index > Fetch the complete content index at: https://roundproxies.com/blog/llms.txt > Use this file to discover other available public pages before exploring further. # Why Your Web Scraper Gets Blocked (The Dirty Truth) - URL: https://roundproxies.com/blog/why-your-web-scraper-gets-blocked/ - Published: 2026-01-30T19:21:46.000Z - Updated: 2026-10-22T15:30:31.000Z - Description: Rotating proxies isn't enough. The real reasons scrapers fail: TLS fingerprints, behavioral analysis, IP reputation. Here's what works. - Author: Marius Bernard - Tags: Expert Article, Web Scraping, #dated-697d03d426f439f88a95b6af Everyone wants to talk about the code. The libraries. The frameworks. Nobody wants to talk about what actually makes scrapers work. I've been building production scrapers for six years. I've extracted data from sites that supposedly "can't be scraped." The difference between scrapers that work and ones that fail has almost nothing to do with your technical skills. It comes down to how good you are at deception. ## The lie we tell ourselves Browse any scraping tutorial and you'll see the same advice regurgitated endlessly. Use rotating proxies. Set a User-Agent header. Add random delays. All technically correct. All completely insufficient. I watched a junior developer follow every "best practice" guide on the internet. His scraper worked perfectly in testing. The moment he pointed it at a real target with actual bot detection? Blocked within 47 requests. Here's what those guides don't tell you: modern anti-bot systems don't just look at what you send. They look at *how* you send it. They look at *when* you send it. They look at patterns you didn't even know existed. And they're getting smarter every month. ## The fingerprint problem nobody wants to explain Your browser is a snitch. Every time you visit a website, your browser broadcasts hundreds of signals. Screen resolution. Installed fonts. WebGL renderer. Audio context hash. Canvas fingerprint. Timezone. Language preferences. Combined, these create a fingerprint more unique than your actual fingerprint. When you fire up Selenium or Puppeteer with default settings, you might as well be wearing a sign that says "I AM A BOT." The `navigator.webdriver` flag is set to true. Your User-Agent says "HeadlessChrome." Your plugin list is empty. Your screen resolution is some weird dimension that no real human would use. Anti-bot systems don't even need to look hard. You're practically screaming at them. The dirty truth? Working scrapers don't just fake a User-Agent string. They fake everything. Canvas rendering. WebGL signatures. Audio fingerprints. They build coherent device profiles where every attribute matches what you'd see on a real Windows 10 machine running Chrome 131 in Central Time. If your fingerprint says you're on Windows but your font list contains macOS system fonts, you're done. If you claim to be Chrome but your TLS handshake looks like Python Requests, you're done. If your browser says it has a screen resolution of 1920x1080 but your WebGL reports a GPU that doesn't support that resolution, you're done. Consistency is everything. And consistency requires lying comprehensively. ## TLS and HTTP/2 fingerprinting: the silent killer A blocked scraper often gets identified before it sends a single HTTP request. Your headers, your User-Agent string and your cookies haven't left your machine yet, and the site already has a fingerprint of your client. ### How the TLS handshake gives you away The fingerprint comes from the TLS handshake. When your client opens an HTTPS connection, it announces the TLS version it supports, the encryption methods it offers and the extensions it includes. Anti-bot systems hash those parameters into a fingerprint called JA3, or the newer JA4. Each HTTP client produces its own hash. Python Requests has a JA3 fingerprint. Selenium has one. Puppeteer has one. Real Chrome has a different one. The hash comes from the TLS library underneath your code, so changing anything at the HTTP level leaves it untouched. When a site sees a JA3 fingerprint that says "Python" but headers that say "Chrome," the jig is up. The mismatch is a stronger signal than either value on its own, because a real Chrome browser can't produce that combination. You've been identified before you even said hello. ### HTTP/2 is a second fingerprint Matching the TLS hash doesn't close the gap. Anti-bot systems also fingerprint the HTTP/2 layer separately. They look at the frame settings your client sends when the connection opens and at the order your headers arrive in. A Python client that copies Chrome's header values but sends them in a different sequence, or with different HTTP/2 settings, still looks like a script. A scraper with a clean TLS impersonation can get caught at this layer. ### Fixing it Bypassing this requires tools that impersonate Chrome's network stack at a low level, covering both the TLS handshake and HTTP/2 behavior. Libraries like `curl_cffi` exist specifically for this. Setting a Chrome User-Agent in Requests won't help, because Python's TLS library is still the one generating the handshake. ### How to tell if fingerprinting is your problem Most developers don't know this layer of detection exists. They get blocked, rotate IPs, change User-Agents, and nothing works, because the fingerprint that mattered was taken before any of that. Read the response before you spend money on a fix. Capture the status code, every response header, any Set-Cookie values and the first 500 bytes of the body, then match what you see: - A 403 that returns an HTML challenge page, which on Cloudflare sites arrives with a cf-mitigated header, means your client failed a score check. It isn't an IP ban. Check your TLS and HTTP/2 fingerprints first. - A 403 with a short JSON or plain-text body usually points to IP reputation or an edge deny rule. That's a proxy question, and a better fingerprint won't change the result. - If the connection dies before any HTTP response comes back, or plain Requests fails on a URL where curl with browser impersonation succeeds, suspect the TLS or HTTP/2 fingerprint. No proxy upgrade fixes that. - A 200 with markup your selectors can't match means the page needs JavaScript rendering, which is a separate problem. Matching the response signature to the layer keeps you from buying better proxies when the real fault was header ordering. ## The behavioral layer where scrapers die A blocked scraper with a perfect fingerprint is usually failing on behavior. Behavioral scoring is its own detection layer, separate from IP reputation and fingerprinting, and it judges how your client moves through the site. A browser that is technically indistinguishable from Chrome still gets flagged if it browses the way only a script would. "Solved fingerprinting" also covers more than TLS. Anti-bot systems fingerprint the HTTP/2 layer separately from the JA3/JA4 hash, including frame settings and the order your headers arrive in. If you only impersonated TLS, you may be getting caught there and blaming behavior for it. Close that gap before you spend time on anything in this section. ### What behavioral systems watch Real visitors rarely request 50 pages in 10 seconds or move through a catalog in perfectly sequential order. They stop to read. They scroll past what they wanted and come back up. They move the mouse while deciding what to click. Modern behavioral analysis records these traces and compares them against statistical models of normal sessions: - Scroll velocity and how it changes over a page - Mouse movement paths, including the drift between clicks - Click timing relative to page load and scroll position - Keyboard cadence on search boxes and forms - Viewport focus events, such as the tab losing and regaining focus A typical scraper produces none of these signals, or produces them with machine regularity. Both stand out against the model. ### Why random delays still get flagged Waiting a random 1–3 seconds between requests is the most common fix, and it still fails. Uniformly distributed delays have a recognizable statistical shape. Human timing comes in bursts of quick activity followed by longer pauses, and the pause depends on the content: someone glancing at a short product title moves on faster than someone reading a long product description. A delay that ignores what is on the page reads as synthetic, however random it is. ### Confirm behavior is the layer that failed Before writing behavior simulation, check that behavior is actually the problem. Teams regularly buy better proxies for problems caused by header ordering, or spend weeks on browser automation for problems that were a plain rate limit. The response tells you which layer rejected you, so capture all of it: the status code, every response header, any Set-Cookie values, and the first 500 bytes of the body. - **403 with an HTML challenge page** and a header like `cf-mitigated`: your client failed a scoring check. This is a score problem rather than a ban, and the layer where behavioral fixes can help. - **403 with a short JSON or plain-text body:** usually IP reputation or an edge deny rule. That is a proxy question, and mouse movements will not change the outcome. - **429:** you hit a rate limit. Slow down and back off before adding anything more elaborate. Change one variable per test and keep a known-good request to compare against. If you alter headers, pacing, and proxies at once, you will not know which change mattered. ### Simulating behavioral noise Once the evidence points at behavior, the practitioners who get through add deliberate noise to their sessions. They inject mouse movements between actions instead of jumping straight to targets. They scroll an element into view before clicking it. They add micro-pauses that track how long a human would plausibly spend on the visible content, so timing varies with the page rather than with a random number generator. Writing code that imitates a distracted reader feels absurd, and it is often what separates a scraper that runs for months from one that gets challenged after a few dozen pages. ## The proxy lie Everyone tells you to use proxies. Rotate IPs. Don't send too many requests from one address. What they don't tell you: the type of proxy matters more than the rotation strategy. Datacenter proxies are cheap. They're also largely useless against serious protection. Anti-bot services maintain databases of known datacenter IP ranges. Your fancy proxy rotation means nothing when every IP in your pool is pre-flagged as suspicious. Residential proxies come from real ISPs, assigned to real homes. They're expensive, sometimes 10x the cost of datacenter alternatives. But they work because they look like regular people browsing from regular homes. Mobile proxies are even harder to detect. They come from cellular networks where thousands of users share the same IP through CGNAT. Blocking a mobile IP means potentially blocking legitimate customers. The money you spend on infrastructure affects your success rate. Sites with heavy protection are essentially pay-to-play. And the math often works out. If you're scraping data worth $10,000/month to your business, spending $2,000/month on residential proxies is a requirement. ## The honeypot test Smart site owners don't just try to detect bots. They try to trap them. Honeypots are invisible links planted in the HTML. Hidden via CSS - `display: none` or `visibility: hidden` or simply colored to match the background. A human visitor never sees them. A scraper that blindly follows every link in the DOM walks right into it. The moment your scraper visits a honeypot URL, you're marked. Your fingerprint is flagged. Your IP goes on a watchlist. And often, that watchlist is shared across anti-bot services used by thousands of sites. You didn't just get blocked from one site. You got blocked from an ecosystem. Working scrapers check element visibility before following links. They verify that a link has actual dimensions, isn't hidden by CSS, doesn't have a nofollow tag, and isn't positioned off-screen. The extra code adds complexity. But skipping it is how you get burned in ways that take weeks to diagnose. ## The rate limiting you don't see Obvious rate limiting returns 429 errors or blocks your IP after too many requests. Subtle rate limiting does something worse: it feeds you bad data. Some sites detect scraper-like behavior and don't block you. Instead, they serve different content. Prices are wrong. Product availability is incorrect. Reviews are shuffled or omitted. You keep scraping, happily collecting data, not realizing it's garbage. I've seen scrapers run for months before someone noticed the data didn't match what real users saw. Months of bad business decisions based on poisoned information. This is why verification matters. Compare your scraped data against manual spot-checks. Use different IP ranges and compare results. If the data differs significantly, you're probably being fed a shadow version of the site. ## What actually works in 2026 After everything I've described, here's what a working scraper stack actually looks like: **Residential or mobile proxies.** Not because rotation is magic, but because IP reputation matters. Datacenter IPs are dead weight against real protection. **Stealth browsers.** Tools like Camoufox, nodriver, or SeleniumBase's UC Mode. They handle fingerprint spoofing at multiple layers: JavaScript APIs, browser attributes, and even WebGL rendering. Out of the box, they pass tests that vanilla Puppeteer fails instantly. **TLS impersonation.** Libraries like `curl_cffi` that mimic real browser TLS signatures. Without this, you're detectable before your request is even processed. **Behavioral noise.** Random delays aren't enough. You need variable scroll patterns, mouse movement injection, realistic viewport focus. The goal is to look bored, distracted, and human. **Element visibility checks.** Never follow a link without verifying it's actually visible to users. Same for form fields and buttons. **Data validation.** Assume you're being fed garbage until proven otherwise. Build verification into your pipeline. **Failure recovery.** Exponential backoff on errors. Automatic proxy rotation when you hit blocks. Session persistence where appropriate. All of this is baseline. ## The uncomfortable conclusion The scraping industry runs on a fundamental tension: sites don't want you there, but they also can't block everyone without collateral damage. The scrapers that work exploit this tension. They invest enough in deception that blocking them would risk blocking legitimate users. They spend money on infrastructure that makes their traffic indistinguishable from organic visitors. Thoroughness matters more than cleverness. Most scrapers fail because their creators treat it as a coding problem. Write the script, extract the data, move on. But working scrapers are engineering projects. They require investment in infrastructure, ongoing maintenance as detection evolves, and a realistic budget for the proxies that make success possible. The dirty truth is that scraping at scale is expensive. The dirtier truth is that companies doing it successfully don't talk about their methods. You don't publish a detailed breakdown of how you circumvent LinkedIn's bot detection, not if you want to keep doing it. So the tutorials stay shallow. The guides recycle the same insufficient advice. And developers keep building scrapers that work fine in testing and fail immediately in production. Now you know why.