> ## Content Index
> Fetch the complete content index at: https://roundproxies.com/blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# Roundproxies Insight: How we test scraping guides
- URL: https://roundproxies.com/blog/how-we-test-scraping-guides/
- Published: 2026-11-21T20:32:10.000Z
- Updated: 2026-11-21T20:32:10.000Z
- Description: How Roundproxies tests its scraping guides: live runs, dated results, and how to report a break.
- Author: Marius Bernard
- Tags: #dated-6b01ff542dcd0f00014b9d63, How the Roundproxies team works

A scraping guide is correct on the day someone last ran it.

After that, the site renames a class or switches anti-bot vendor, and the code returns a 403 while the article still says it works.

I write and test the scraping guides on this blog, and this page is the process I hold them to. Guides that have been through it carry a dated "Last tested" line.

A guide without that line hasn't been through it yet. We're working backwards through the older ones.

## How do we test our scraping guides?

We test our scraping guides by running each method against the live site before publishing, in a clean Python environment and at low volume, from a [residential IP](https://roundproxies.com/blog/how-residential-proxies-work/) in the site's home country. The date, country, and library versions go into a "Last tested" line. Each guide gets a rerun every six months, or sooner after a break report.

## What a test run covers

A test run is five steps, done in the same order for each guide.

### 1\. Identify the protection

Before writing any code, I run [antibot-detect](https://github.com/mmewni/antibot-detect) against the target. It sends one browser-like request and reports which bot-management product answered.

It's passive, so a clean result doesn't prove the site is unprotected. Checks that only run in JavaScript can stay hidden from it.

### 2\. Read robots.txt

I read the site's robots.txt and list the paths the guide touches. If a method needs a disallowed path, the guide says so next to that method.

[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) is the reference. It describes robots rules as requests to crawlers and not as access authorization, so the site's terms still apply either way.

### 3\. Run each method in a clean environment

Each run starts in a new virtual environment with only the packages from the guide's prerequisites list. I record the versions.

Then I run the code blocks in page order against the live site, at one request every 3 to 6 seconds.

A method passes when it returns the fields the guide promises on at least 10 different pages.

I also note which client ran the test: curl\_cffi, primp, tls-client, or a browser.

Sites read the [TLS handshake](https://roundproxies.com/blog/what-is-tls-fingerprint/), so a method can pass with one client and fail with another.

### 4\. Repeat from two networks

I run the guide once from a datacenter IP and once from a residential IP in the site's home country, and record the status codes from both.

That's where a line like "datacenter IPs get a 403 here" comes from. If I didn't see it myself, the guide names whoever did.

### 5\. Record what happened

The result goes into the guide as a "Last tested" line. Where a guide shows output, the screenshot is from that run.

## What the "Last tested" line means

The line says one person ran the code on one day under the stated conditions. It follows this format:

| What you see                        | What it means                                                  |
| ----------------------------------- | -------------------------------------------------------------- |
| "Last tested" with a date           | The guide's methods passed on that date under those conditions |
| "Method 4 broken as of" with a date | That method failed a rerun and isn't fixed yet                 |
| No line                             | The guide hasn't been through this process yet                 |

A fact I couldn't check myself is attributed in the text. You'll see the source named, such as another write-up or the site's own docs, with the month it was published.

## What we don't test

A pass is narrower than it sounds, and four things sit outside it.

- **Volume.** A test run is tens to low hundreds of requests. A method that passes at 100 pages can still get blocked at 10,000; [avoiding blocks at volume](https://roundproxies.com/blog/web-scraping-without-getting-blocked/) is its own job.
- **Logged-in pages.** I don't test anything behind an account.
- **Other country versions.** If a guide was tested on a site's .de domain, it says so, and the .fr domain may behave differently.
- **Shelf life.** The date tells you when the code worked. It can't tell you whether the site changed last week.

## What's free and what isn't

Guides with a "Last tested" line use free, open-source tools that run on your machine. That rules out paid scraping APIs, hosted scrapers, and anything you need an account for.

Proxies are the one cost. Some of the sites we cover refuse datacenter IPs outright, and residential proxies cost money from any provider.

Roundproxies sells proxies, so we have a stake in you needing them. Each guide says that once, and the code takes any provider's proxy URL.

For a first run, your home connection is often enough.

## How to report a break or suggest a tool

A useful break report has:

- The guide's URL and the method number
- The full error message or status code
- The date, your country, and whether the IP was home, datacenter, or residential
- The client library and its version

Tool suggestions are welcome on one condition: the tool is open source and you run it yourself. Paid APIs and hosted scrapers don't go in, even when they'd be the easier answer.

I try a suggested tool against a guide's target before it goes into the guide.

### What happens after you report

I rerun the method under the conditions you describe.

If I can fix it, the guide gets new code, a new "Last tested" date, and a changelog entry that credits you if you'd like.

If I can't, the method gets marked as broken with the date. It stays on the page so the next reader doesn't lose an afternoon to it.

## FAQ

### Are your scraping guides free to follow?

Yes. The code and tools in tested guides are free and open source. The one cost is proxies at volume, which cost money from any provider, including us.

### How often do you retest a scraping guide?

Every six months, and sooner when a reader reports a break. The "Last tested" line on the guide shows the date of the most recent run.

### What happens when a method stops working?

I rerun it and either fix the code or mark the method as broken with the date. A fix comes with a new "Last tested" date and a changelog entry.

### Can I suggest a tool or a site for a guide?

Yes, through the GitHub repo or the report form. Tools need to be open source and self-run; paid APIs and hosted scrapers aren't added.

### Do you test scraping guides at scale?

No. A test run is tens to low hundreds of requests, enough to show the method works. It doesn't show how the method holds up at 10,000 pages a day.

## If a guide fails for you

Send the error, the date, and your country. That's enough for me to rerun the method.

If you're new to scraping and can't tell whether the guide or your setup is at fault, [web scraping with Python](https://roundproxies.com/blog/web-scraping-python/) covers the basics.

Our [HTTP 429 explainer](https://roundproxies.com/blog/http-error-429/) covers the most common rate-limit error.