Web Scraping with R in 2026: Build a scraper that works

Your script pulls 20 pages fine. On page 300 it hits a 429, the loop dies, and you lose the whole run because the results only ever existed in a variable in memory.

That's the gap between an rvest tutorial and a scraper you can walk away from. Most guides stop at html_table() and hand you off to a paid API for everything hard.

This one doesn't. You'll build a scraper in R that retries, throttles, runs requests in parallel, survives a crash, and rotates proxies. Every line of it is yours.

What is web scraping with R?

Web scraping with R means pulling data off web pages with R packages instead of an API. httr2 sends the HTTP request, rvest parses the returned HTML into nodes you select with CSS, and the result lands in a data frame. Use it when a site displays data it refuses to export.

The split matters more than beginners expect. rvest is a parser with a convenience fetcher bolted on; httr2 is a real HTTP client with retries, rate limiting, and proxy support.

Tutorials that only use read_html("https://…") are quietly using that bolted-on fetcher. It works until the site pushes back, then you have no controls to reach for.

So we'll separate the two jobs from the first line of code.

Prerequisites

• R 4.4 or newer (R.version.string to check)
• RStudio, Positron, or any editor you like
• Google Chrome installed, if you want the JavaScript section to run
• Enough CSS to know that .price is a class and #main is an ID

You don't need prior rvest experience. You do need to be comfortable with pipes and functions.

Step 1: Install the four packages that matter

R has a lot of scraping packages. Four of them cover about 95% of real work.

install.packages(c("httr2", "rvest", "dplyr", "chromote", "polite"))

httr2 makes the requests. rvest parses HTML (and pulls in xml2 underneath). chromote drives a headless Chrome for JavaScript pages. polite reads robots.txt and enforces a crawl delay if you want that handled for you.

dplyr is optional but you'll want it the moment you have more than one column.

One name you'll see in older guides: RSelenium. It's still on CRAN and got a maintenance release in early 2026, but it talks to browsers over the deprecated JsonWireProtocol and needs a separate Selenium server. For new R code, chromote is less machinery for the same result.

Step 2: Fetch with httr2, not read_html()

read_html(url) performs the request for you. That's convenient and it's also why your scraper has no timeout, no user agent, and no retry policy.

Build the request explicitly instead:

library(httr2)

resp <- request("https://books.toscrape.com/") |>
  req_user_agent("research-bot/1.0 (contact: [email protected])") |>
  req_timeout(20) |>                          # fail fast instead of hanging
  req_retry(max_tries = 3, backoff = ~ 2 ^ .x) |>  # 2s, 4s, 8s
  req_perform()

resp_status(resp)   # 200

req_retry() already knows to retry on 429 and 503, so you don't have to write that logic. The formula ~ 2 ^ .x is exponential backoff, where .x is the attempt number.

The user agent is worth setting honestly. A real contact address gets you an email asking you to slow down; a fake Chrome string gets you a silent IP ban.

Now hand the response to rvest:

library(rvest)
page <- resp_body_html(resp)

resp_body_html() returns exactly what read_html() would have returned. Every rvest function works on it from here.

Step 3: Parse the HTML with rvest

Open the target page, right-click the thing you want, and hit Inspect. On books.toscrape.com each product is an <article class="product_pod"> containing a title link, a price, and a rating class.

titles <- page |>
  html_elements("article.product_pod h3 a") |>
  html_attr("title")

head(titles, 3)
#> [1] "A Light in the Attic" "Tipping the Velvet" "Soumission"

html_elements() (plural) returns every match. html_attr("title") pulls an attribute; html_text2() pulls the visible text with whitespace collapsed the way a browser renders it.

Use html_text2() over html_text() unless you specifically want the raw source whitespace. It saves you a trimws() call on almost every field.

Step 4: Build a data frame that can't silently misalign

Here's the bug that eats hours, and I have yet to see a tutorial mention it.

The obvious approach is to grab each column separately:

# DON'T DO THIS
books <- tibble(
  title = page |> html_elements(".product_pod h3 a") |> html_attr("title"),
  price = page |> html_elements(".product_pod .price_color") |> html_text2()
)

That works right up until one product card is missing a price. Then you get 20 titles and 19 prices, and every price from that row down belongs to the wrong book.

If the vectors happen to differ in length, R errors out and you notice. If a sold-out item is missing a price and an ad card adds a stray one, the lengths match and you never notice.

Select the card first, then read fields off each card:

cards <- html_elements(page, "article.product_pod")

books <- tibble::tibble(
  title  = cards |> html_element("h3 a")        |> html_attr("title"),
  price  = cards |> html_element(".price_color") |> html_text2(),
  rating = cards |> html_element("p.star-rating")|> html_attr("class")
)

Note the singular html_element(). Applied to a node set, it returns exactly one result per node and pads misses with NA, so your columns can't drift out of sync.

Clean up the types and you have real data:

library(dplyr)

books <- books |>
  mutate(
    price  = as.numeric(gsub("[^0-9.]", "", price)),   # "£51.77" -> 51.77
    rating = sub("star-rating ", "", rating)           # "star-rating Three" -> "Three"
  )

Two things to watch. gsub("[^0-9.]", "") also strips thousands separators, which is what you want for 1,299.00 and wrong for European 1.299,00. And ratings arrive as words, so map them to integers before you try to average anything.

Step 5: Crawl pagination without getting banned

Fifty pages of results, one request at a time, no delay, is the fastest way to meet a block page. httr2 gives you a rate limiter that works across your whole run.

Start with a base request and a parse function:

base_req <- request("https://books.toscrape.com/") |>
  req_user_agent("research-bot/1.0 (contact: [email protected])") |>
  req_retry(max_tries = 3) |>
  req_throttle(capacity = 30, fill_time_s = 60)   # 30 requests per minute

parse_books <- function(resp) {
  cards <- resp_body_html(resp) |> html_elements("article.product_pod")
  tibble::tibble(
    title = cards |> html_element("h3 a")         |> html_attr("title"),
    price = cards |> html_element(".price_color") |> html_text2()
  )
}

req_throttle() uses a token bucket: the bucket refills to capacity over fill_time_s, and each request spends one token. You get a burst when you start and a steady rate after that.

Now fan out across pages:

urls <- sprintf("https://books.toscrape.com/catalogue/page-%d.html", 1:50)
reqs <- lapply(urls, \(u) req_url(base_req, u))

resps <- req_perform_parallel(reqs, on_error = "continue", max_active = 5)

books <- resps_data(resps_successes(resps), parse_books)
nrow(books)   # 1000

on_error = "continue" is the important argument. Without it, one dead page kills the batch and you throw away 49 good responses.

resps_successes() filters out the failures, and resps_data() runs your parser over what's left and row-binds the result.

One catch the httr2 docs are explicit about: throttling and retries apply across all requests in a parallel batch, not per request. So parallelism only helps if your throttle allows the extra volume.

Those are arithmetic, not stopwatch numbers, so check them against your own target. The lesson holds either way: raising max_active while leaving the throttle alone changes nothing except your CPU usage.

If you'd rather not think about rates at all, polite::bow() reads the site's robots.txt and applies its declared crawl delay for you. Slower, and much harder to get yelled at.

Step 6: Scrape JavaScript pages with read_html_live()

Sometimes html_elements() returns an empty node set for something you can plainly see in the browser. That usually means the content is rendered by JavaScript after the HTML loads.

rvest ships an answer. read_html_live() runs a real Chrome in the background via chromote, so the DOM you parse is the DOM the browser built.

library(rvest)

sess <- read_html_live("https://quotes.toscrape.com/js/")

quotes <- sess |> html_elements(".quote .text") |> html_text2()
head(quotes, 2)

sess$session$close()   # always close the tab, or Chrome piles up

The returned object is an R6 LiveHTML, and it has methods a static parse can't offer: $click(), $type(), $scroll_to(), and $view() to watch the page in a live viewer while you debug selectors.

Infinite scroll is a loop of scroll, wait, re-read:

sess <- read_html_live("https://example.com/feed")

for (i in 1:5) {
  sess$scroll_to(top = 1e6)   # jump to the bottom
  Sys.sleep(1.5)              # let the fetch land
}

items <- sess |> html_elements(".feed-item h2") |> html_text2()
sess$session$close()

Before you reach for any of this, open DevTools and check the Network tab. Half the "JavaScript sites" I've hit are just calling a JSON endpoint, and req_perform() plus resp_body_json() gets the same data in a tenth of the time with none of the browser overhead.

Browsers are the expensive option. Treat them as the fallback, not the default.

Step 7: Route requests through a proxy pool

One IP making a thousand requests looks like one IP making a thousand requests, no matter how polite your delay is. Spreading traffic across addresses is what keeps long runs alive.

httr2 has proxy support built in:

resp <- request("https://books.toscrape.com/") |>
  req_proxy(
    url      = "gw.your-proxy-host.net",
    port     = 7000,
    username = Sys.getenv("PROXY_USER"),
    password = Sys.getenv("PROXY_PASS")
  ) |>
  req_perform()

Credentials go in .Renviron, never in the script. usethis::edit_r_environ() opens the right file, and everything in it is available through Sys.getenv() after a restart.

Rotation is just a function that picks an endpoint per request:

proxies <- c("gw.host.net:7000", "gw.host.net:7001", "gw.host.net:7002")

with_proxy <- function(req, endpoint = sample(proxies, 1)) {
  parts <- strsplit(endpoint, ":")[[1]]
  req_proxy(req, parts[1], as.integer(parts[2]),
            Sys.getenv("PROXY_USER"), Sys.getenv("PROXY_PASS"))
}

reqs <- lapply(urls, \(u) req_url(base_req, u) |> with_proxy())
resps <- req_perform_parallel(reqs, on_error = "continue", max_active = 5)

Random sampling is fine for stateless page fetches. If the site tracks a session (a cart, a login, a multi-step filter), you need the same exit IP for the whole sequence, which most providers expose as a session ID appended to the username.

Full disclosure: we sell residential proxies at Roundproxies, so weigh that accordingly. The R code above is provider-agnostic, and if you're deciding what kind of IP you need, our breakdown of residential vs. datacenter proxies covers the tradeoff without the sales pitch.

Make the run resumable

This is the section that separates a script from a scraper, and it's the one every guide skips because it isn't glamorous.

A 5,000-page job will fail somewhere. The question is whether failure costs you five minutes or five hours.

Write results to disk as you go, keyed by URL, and skip anything already on disk:

dir.create("cache", showWarnings = FALSE)

fetch_cached <- function(url) {
  key  <- file.path("cache", paste0(substr(rlang::hash(url), 1, 16), ".html"))
  if (file.exists(key)) return(read_html(key))

  resp <- req_url(base_req, url) |> with_proxy() |> req_perform()
  writeLines(resp_body_string(resp), key)
  read_html(key)
}

Re-running the job now costs one file.exists() check per cached page. Kill it, restart it, change your parser and run it again against local HTML at zero network cost.

That last part is the real win. Selector bugs are found by iterating, and iterating against a live site gets you rate limited for a typo.

httr2 does have req_cache(), but it obeys HTTP cache headers, and scraping targets rarely send useful ones. The 12 lines above are more predictable for this job.

Append parsed rows to a CSV in the same loop and a crash costs you one page:

readr::write_csv(page_rows, "books.csv", append = file.exists("books.csv"))

Troubleshooting: real R errors and what they mean

Error in doc_namespaces(x) : external pointer is not valid

Why: rvest documents are C pointers from xml2, not R objects. saveRDS() an xml_document, restart R, load it back, and the pointer is dangling. Same thing happens when you pass one to a future/parallel worker.

Fix: persist the HTML string, not the parsed object. as.character(page) on the way out, read_html() on the way in. Or parse to a data frame first and save that.

HTTP 403 Forbidden.

Why: usually a missing or default user agent. httr2 announces itself as httr2/1.x r-curl/… unless you say otherwise, and plenty of WAFs block on that alone.

Fix: set req_user_agent(), add req_headers(Accept = "text/html", \Accept-Language` = "en-US,en;q=0.9")`, and if it persists, get a residential IP in front of it. Persistent 403s on the first request are a fingerprinting problem, not a rate problem.

html_elements() returns {xml_nodeset (0)}

Why: either the selector is wrong or the content is JavaScript-rendered.

Fix: run xml2::write_html(page, "debug.html") and open that file in a browser. If the data isn't in there, it never arrived over HTTP, and you need Step 6. If it is in there, your selector is off, and html_structure(page) will show you the real shape.

Error: Chrome debugging port closed (or chromote hangs forever)

Why: no Chrome installed where chromote looks, an old zombie process holding the port, or a container without --no-sandbox.

Fix: chromote::find_chrome() to confirm the binary. Kill stale processes with chromote::default_chromote_object()$get_browser()$close(). In Docker, set options(chromote.args = c("--no-sandbox", "--disable-dev-shm-usage")).

Text comes back as é instead of é

Why: the server declared no charset and R guessed Latin-1.

Fix: force it at parse time with read_html(x, encoding = "UTF-8"), or resp_body_html(resp, encoding = "UTF-8"). On Windows, also confirm your session isn't in a legacy locale.

FAQ

Is R good for web scraping, or should I use Python?

R is good at it, and better than Python at the part that happens next. If the scraped data ends up in a dplyr pipeline, a ggplot, or a Shiny app, staying in R saves you an entire handoff. Python has a deeper ecosystem for browser automation and anti-bot work, so heavy JavaScript targets are easier there. For most tabular targets, the difference is preference.

Is web scraping with R legal?

The language you use has no bearing on legality. What matters is what you collect, from where, and what you do with it: public data, personal data, copyrighted content, and terms-of-service violations are all separate questions with different answers by jurisdiction. Read the robots.txt, don't scrape behind logins you agreed not to scrape behind, and talk to a lawyer before anything commercial.

Can rvest handle JavaScript?

Yes, through read_html_live(), which runs a headless Chrome via chromote and returns the rendered DOM. The plain read_html() only sees the server's HTML source. Check the Network tab for a JSON endpoint before committing to a browser.

How do I scrape a table from Wikipedia in R?

read_html(url) |> html_element("table.wikitable") |> html_table() gets you a tibble in one line. html_table() handles colspan and rowspan reasonably well, though merged header cells still produce duplicate column names you'll want to rename.

How many requests per second is safe?

There's no universal number. Start at one request per second, watch for 429s and rising response times, and back off if either appears. req_throttle(capacity = 30, fill_time_s = 60) is a defensible default for a site you don't own.

Wrapping up

You now have the pieces that survive contact with a real site: httr2 for the request with retries and throttling, rvest for parsing, html_element() on card nodes so your columns can't drift, chromote for the JavaScript minority, proxies for volume, and a disk cache so a crash costs minutes instead of hours.

Build it in that order. The mistake I keep watching people make with web scraping with R is jumping straight to a headless browser because one selector came back empty, when a Network-tab check would have found a JSON endpoint in 30 seconds.

Start with the static path, cache everything, and add machinery only when the site forces you to.

Next: the rvest selector reference, the httr2 documentation for the request functions used here, and Hadley Wickham's web scraping chapter in R for Data Science for the HTML fundamentals. If you're porting this pattern to another stack, our Python proxy rotation guide covers the same ideas with requests, and how rotating proxies work explains what's happening on the wire.