> ## Content Index > Fetch the complete content index at: https://roundproxies.com/blog/llms.txt > Use this file to discover other available public pages before exploring further. # How to Do Requests with R - URL: https://roundproxies.com/blog/requests-r/ - Published: 2025-10-06T22:17:08.000Z - Updated: 2026-09-23T13:04:36.000Z - Description: Learn modern HTTP requests in R with httr2. Send requests, authenticate, retry failures, and run parallel API calls with clean, efficient code. - Author: Marius Bernard - Tags: Knowledgebase, #dated-68e43d6d4fa498a2c6ee1f84 If you work in R and need anything from the web, you end up making HTTP requests. Pulling rows out of a REST API, [scraping a website](https://roundproxies.com/blog/why-your-web-scraper-gets-blocked/), or writing a package that wraps someone else's service all reduce to the same two steps: build a request, read the response. The reason people get stuck isn't the protocol. It's that search results for requests in R still point at RCurl and httr, and neither is where I'd start a new project today. httr2 is. It's a rewrite of httr by the same team, and it changes one thing that matters more than the rest of the API combined: a request is an object you assemble piece by piece and can print before you send it. When a call returns a 401 you can look at the exact URL, headers, and body that were about to go out, instead of guessing. Old httr code keeps working, so there's no rush to rewrite anything that's already in production. New code should use httr2. This guide walks through installing httr2, sending a first GET, passing query parameters and JSON bodies, authenticating, reading a response without fighting the encoding. It also covers backing off politely when a server rate-limits you, and firing requests in parallel when you have hundreds of URLs to get through. Every example runs against a public endpoint, so you can paste it into the console and watch it work. ## Table of contents Twelve sections, roughly in the order you hit these problems when you start sending requests in R. If you already have httr2 installed and a script that works, jump to the part that's breaking. - **What are HTTP requests (and why should you care)?** The request and response cycle, the verbs you'll actually use, and what a status code is telling you. - **Getting started with httr2** Installing the package and why it replaced httr for new work. - **Your first GET request** Build a request, send it, read the body back into R. - **Adding query parameters and headers** Query strings without gluing URLs together by hand, plus setting a user agent. - **POST requests with body data** Sending JSON and form-encoded payloads. - **Authentication strategies** API keys, bearer tokens, basic auth, and keeping credentials out of the script you commit. - **Error handling and retries** Telling a client mistake apart from a server one, and retrying only the failures worth retrying. - **Rate limiting and throttling** Pacing your calls so the API keeps answering them. - **Parallel requests for performance** Running a batch of requests at once instead of one after another. - **Working with paginated APIs** Walking every page of a result set without hand-writing the loop each time. - **Converting curl commands to R** Turning a copy-pasted curl line from the docs or your browser into working R code. - **Common pitfalls and how to avoid them** The mistakes that cost me the most debugging time. ## What are HTTP requests (and why should you care)? Every time you visit a website, your browser sends an HTTP request to a server asking for data. The server processes that request and sends back an HTTP response containing the webpage, image, or whatever you asked for. In R, you can do the same thing programmatically. Instead of clicking around in a browser, you write code that sends requests and processes responses. Common uses: - **Pulling data from APIs** \- Most modern web services offer APIs that return structured data (usually JSON) - **Web scraping** \- When there's no API, you can request HTML pages and extract the data you need - **Building integrations** \- Connect your R code to external services like Slack, GitHub, or Stripe - **Automating workflows** \- Schedule scripts to fetch fresh data without manual intervention The key difference from using a browser? You get to process the responses programmatically, which means you can extract exactly what you need and integrate it directly into your data analysis pipeline. ## Getting started with httr2 For HTTP requests in R, httr2 is the package I reach for. It's a ground-up rewrite of httr by the same author, Hadley Wickham, and it fixes the parts of httr that got awkward once your code grew past a single API call. The alternatives are still around, so it's worth knowing where they fit. The `curl` package is the low-level binding to libcurl that nearly everything else sits on top of; reach for it when you need control over a setting no wrapper exposes. RCurl is legacy at this point. httr works and won't break, but development has moved to httr2, so I'd only stay on httr if I inherited a codebase full of it. If you have used httr, httr2 will feel familiar: same ideas, fewer sharp edges, and a design built around the pipe. Install it like any other package: ```r install.packages("httr2") library(httr2) ``` The core workflow in httr2 is straightforward: 1. Create a request object 2. Modify it (add headers, authentication, etc.) 3. Perform the request 4. Extract data from the response The part that matters is that steps 1 and 2 are separate from step 3\. In httr, calling `GET()` fired the request straight away. In httr2, `request()` hands you an ordinary R object that has not touched the network. You keep piping modifiers into it, query parameters, headers, auth, retry rules, and nothing is sent until you call `req_perform()`. That laziness buys you three practical things. You can define a base request once, with the host and API key, and reuse it across every endpoint instead of repeating yourself. You can print a request and read back exactly what you have built. And you can call `req_dry_run()` to see the raw bytes that would go over the wire, which is the fastest way to debug a broken auth header without burning calls against a rate limit. The naming convention makes the rest easy to guess. Anything that builds or changes a request starts with `req_`. Anything that reads a response starts with `resp_`, such as `resp_body_json()` for a parsed JSON body or `resp_status()` for the status code. Once you know that, most of the API is discoverable by tab completion. ## Your first GET request Let's start simple. Here's how to fetch data from an API: ```r library(httr2) # Create a request req <- request("https://api.github.com/users/hadley") # Perform it and get the response resp <- req_perform(req) # Check the status resp ``` This returns something like: ``` GET https://api.github.com/users/hadley Status: 200 OK Content-Type: application/json Body: In memory (1876 bytes) ``` A 200 status code means success. The response includes headers (like `Content-Type`) and a body containing the actual data. To extract the JSON data from the response: ```r # Parse JSON response user_data <- resp_body_json(resp) # Access specific fields user_data$name user_data$public_repos ``` The `resp_body_json()` function automatically parses the JSON and returns an R list. For other response types, httr2 provides `resp_body_html()`, `resp_body_xml()`, and `resp_body_string()`. ## Adding query parameters and headers Most APIs need additional information in your requests. Query parameters go in the URL (like `?page=1&limit=50`), while headers provide metadata about your request. ### Query parameters Instead of manually building URL strings, use `req_url_query()`: ```r req <- request("https://api.github.com/search/repositories") |> req_url_query( q = "language:R", sort = "stars", per_page = 10 ) resp <- req_perform(req) ``` This automatically handles URL encoding, so you don't have to worry about spaces or special characters breaking your URLs. ### Custom headers Headers are metadata about your request. The most common use case is authentication, but they're also used to specify content types or user agents: ```r req <- request("https://api.example.com/data") |> req_headers( Accept = "application/json", `User-Agent` = "my-r-script/1.0" ) resp <- req_perform(req) ``` Note the backticks around `User-Agent` . That's because R doesn't like hyphens in variable names. The backticks tell R to treat it as a literal string. ### Seeing what gets sent Want to debug your request before sending it? Use `req_dry_run()`: ```r req |> req_dry_run() ``` This prints out exactly what HTTP request will be sent, including all headers and the request body. It helps when something isn't working and you need to see what's actually happening under the hood. ## POST requests with body data GET requests fetch data. POST requests send data to a server. This is how you submit forms, create resources, or send data to an API. ### Sending JSON The most common format for API requests is JSON: ```r req <- request("https://api.example.com/users") |> req_body_json( list( name = "Alice", email = "alice@example.com", age = 30 ) ) resp <- req_perform(req) ``` The `req_body_json()` function automatically: - Converts your R list to JSON - Sets the `Content-Type` header to `application/json` - Changes the request method from GET to POST ### Sending form data Some APIs expect form-encoded data instead of JSON: ```r req <- request("https://api.example.com/login") |> req_body_form( username = "alice", password = "secret123" ) resp <- req_perform(req) ``` This mimics what happens when you submit an HTML form in a browser. ### Uploading files For file uploads, use `req_body_multipart()`: ```r req <- request("https://api.example.com/upload") |> req_body_multipart( file = curl::form_file("data.csv"), description = "Monthly sales data" ) resp <- req_perform(req) ``` ## Authentication strategies Most useful APIs require authentication. httr2 supports all the common methods. ### Bearer tokens The simplest authentication method - just include a token in the headers: ```r req <- request("https://api.github.com/user/repos") |> req_auth_bearer_token("ghp_your_token_here") resp <- req_perform(req) ``` This adds an `Authorization: Bearer ` header to your request. **Pro tip:** Never hardcode tokens in your scripts. Use environment variables instead: ```r token <- Sys.getenv("GITHUB_TOKEN") req <- request("https://api.github.com/user/repos") |> req_auth_bearer_token(token) ``` Set the environment variable in your `.Renviron` file: ``` GITHUB_TOKEN=ghp_your_token_here ``` ### Basic authentication Some APIs use username and password: ```r req <- request("https://api.example.com/data") |> req_auth_basic("username", "password") resp <- req_perform(req) ``` This sends your credentials encoded in the `Authorization` header. It's called "basic" because it's simple, but it's only secure over HTTPS. ### OAuth 2.0 OAuth is more complex but provides better security for accessing user data. httr2 has built-in support for various OAuth flows: ```r client <- oauth_client( id = "your_client_id", secret = "your_client_secret", token_url = "https://api.example.com/oauth/token", name = "my_app" ) req <- request("https://api.example.com/data") |> req_oauth_auth_code( client = client, auth_url = "https://api.example.com/oauth/authorize" ) resp <- req_perform(req) ``` The first time you run this, it'll open a browser window for you to authorize the app. After that, it caches the token so you don't have to authorize again. ## Error handling and retries Not every request succeeds. Servers go down, networks have hiccups, and rate limits kick in. httr2 makes it easy to handle these situations. ### Basic error handling By default, httr2 converts HTTP errors (4xx and 5xx status codes) into R errors: ```r req <- request("https://api.github.com/users/nonexistent") tryCatch( resp <- req_perform(req), error = function(e) { message("Request failed: ", e$message) } ) ``` ### Automatic retries For transient errors (like rate limiting or temporary server issues), you can automatically retry: ```r req <- request("https://api.example.com/data") |> req_retry( max_tries = 3, is_transient = \(resp) resp_status(resp) %in% c(429, 500, 503) ) resp <- req_perform(req) ``` This will retry up to 3 times if the server returns a 429 (rate limit), 500, or 503 error. httr2 automatically uses exponential backoff, waiting longer between each retry. ### Custom error messages Want to provide better error messages to your users? ```r req <- request("https://api.example.com/data") |> req_error( is_error = \(resp) resp_status(resp) >= 400, body = function(resp) { json <- resp_body_json(resp) paste("API error:", json$error$message) } ) ``` Now when an error occurs, your custom message gets displayed instead of the generic HTTP error. ## Rate limiting and throttling APIs often have rate limits: restrictions on how many requests you can make per minute or hour. Blast an API with too many requests and you'll get blocked. httr2's `req_throttle()` helps you stay within limits: ```r req <- request("https://api.github.com/users/hadley") |> req_throttle(rate = 30 / 60) # 30 requests per 60 seconds resp <- req_perform(req) ``` This ensures you never exceed 30 requests per minute. If you make requests too quickly, httr2 automatically waits before sending the next one. For more sophisticated rate limiting: ```r req <- request("https://api.example.com/data") |> req_throttle( capacity = 100, # Maximum 100 requests fill_time_s = 3600 # Refills over 1 hour ) ``` This implements a token bucket algorithm - you start with 100 requests, and the bucket refills at a rate of 1 request every 36 seconds. ## Parallel requests for performance Sequential requests are slow. If you need to fetch data for 100 users, doing it one at a time takes forever. Parallel requests let you make multiple requests simultaneously. ### Simple parallel execution ```r library(httr2) # Create a list of requests users <- c("hadley", "jennybc", "jimhester") reqs <- lapply(users, function(user) { request(paste0("https://api.github.com/users/", user)) |> req_throttle(rate = 30 / 60) # Still respect rate limits! }) # Perform them in parallel resps <- req_perform_parallel(reqs, max_active = 3) # Extract data from each response user_data <- lapply(resps, resp_body_json) ``` The `max_active` parameter controls how many requests run simultaneously. Don't set this too high or you'll overwhelm the server (and get blocked). **Important:** Always use `req_throttle()` with parallel requests. Without it, you'll fire off all requests at once and likely get rate limited. ### Handling errors in parallel requests By default, `req_perform_parallel()` stops on the first error. Use `on_error = "continue"` to keep going: ```r resps <- req_perform_parallel( reqs, on_error = "continue", max_active = 5 ) # Check which requests succeeded successes <- resps_successes(resps) failures <- resps_failures(resps) cat("Succeeded:", length(successes), "\n") cat("Failed:", length(failures), "\n") ``` ## Working with paginated APIs Many APIs return data in pages. Instead of sending all 10,000 results at once, they send 50 at a time and provide a way to request the next page. httr2 has built-in helpers for common pagination patterns: ### Offset-based pagination ```r req <- request("https://api.example.com/items") |> req_url_query(limit = 50) resps <- req_perform_iterative( req, next_req = iterate_with_offset( param_name = "offset", resp_pages = function(resp) { json <- resp_body_json(resp) json$total_pages } ), max_reqs = Inf ) # Combine all results all_items <- unlist(lapply(resps, function(r) { resp_body_json(r)$items }), recursive = FALSE) ``` This automatically adds `offset=0`, `offset=50`, `offset=100`, etc. to subsequent requests until all pages are fetched. ### Cursor-based pagination Some APIs use cursors instead of offsets: ```r req <- request("https://api.example.com/items") resps <- req_perform_iterative( req, next_req = iterate_with_cursor( param_name = "cursor", resp_cursor = function(resp) { json <- resp_body_json(resp) json$next_cursor } ), max_reqs = Inf ) ``` The `iterate_with_cursor()` function extracts the `next_cursor` from each response and uses it in the next request. ## Converting curl commands to R Ever find a working curl command in API documentation and wish you could just use it in R? Good news: httr2 can translate curl commands for you. ```r library(httr2) curl_cmd <- 'curl -X GET --header "Accept: application/json" --header "Authorization: Bearer token123" "https://api.example.com/data?limit=10"' # Convert to httr2 code curl_translate(curl_cmd) ``` This outputs R code you can copy and paste: ```r request("https://api.example.com/data") |> req_url_query(limit = "10") |> req_headers( Accept = "application/json", Authorization = "Bearer token123" ) ``` It even handles complex curl commands with custom headers, authentication, and request bodies. This is a massive time-saver when you're trying to replicate an API call you found in documentation or copied from your browser's developer tools. ## Common pitfalls and how to avoid them ### Forgetting to URL-encode parameters If you build URLs manually, special characters will break them: ```r # Bad url <- paste0("https://api.example.com/search?q=", "R programming") # Space breaks the URL # Good req <- request("https://api.example.com/search") |> req_url_query(q = "R programming") # Automatically encoded ``` ### Not handling rate limits Hammering an API without rate limiting is a fast track to getting blocked: ```r # Bad - no rate limiting for (i in 1:1000) { resp <- request("https://api.example.com/data") |> req_perform() } # Good - throttled req <- request("https://api.example.com/data") |> req_throttle(rate = 10 / 60) # 10 per minute for (i in 1:1000) { resp <- req_perform(req) # httr2 automatically waits when needed } ``` ### Hardcoding secrets in scripts Never put API keys or passwords directly in your code: ```r # Bad - visible in code req <- request("https://api.example.com/data") |> req_auth_bearer_token("super_secret_token_123") # Good - from environment variable token <- Sys.getenv("API_TOKEN") req <- request("https://api.example.com/data") |> req_auth_bearer_token(token) ``` Set the environment variable in your `.Renviron` file, and it stays out of version control. ### Not using req\_dry\_run() when debugging When requests aren't working, looking at the actual HTTP being sent saves hours of frustration: ```r req <- request("https://api.example.com/data") |> req_headers(Authorization = "Bearer token") |> req_url_query(limit = 10) |> req_dry_run() # See exactly what will be sent ``` ### Ignoring HTTP status codes Just because a request doesn't error doesn't mean it worked: ```r resp <- req_perform(req) # Check the status if (resp_status(resp) == 200) { data <- resp_body_json(resp) } else { warning("Request returned status: ", resp_status(resp)) } ``` A 404 means "not found," 401 means "unauthorized," 429 means "rate limited," and 500+ means server errors. Handle them appropriately. ### Making sequential requests when parallel would work If requests are independent, run them in parallel: ```r # Slow - sequential for (id in ids) { resp <- request(paste0("https://api.example.com/items/", id)) |> req_perform() } # Fast - parallel reqs <- lapply(ids, function(id) { request(paste0("https://api.example.com/items/", id)) }) resps <- req_perform_parallel(reqs, max_active = 10) ``` For 100 requests, this can be 10x faster or more. ## Wrapping up Making HTTP requests in R with httr2 is straightforward once you understand the basics. Start with a simple `request()`, add what you need (headers, auth, query params), then `req_perform()` to execute it. The key things to remember: - **Use httr2, not httr** \- it's the modern standard with better features - **Always use req\_throttle()** \- especially with parallel requests - **Keep secrets in environment variables** \- never hardcode them - **Use req\_dry\_run() for debugging** \- see exactly what you're sending - **Handle errors gracefully** \- use req\_retry() for transient failures - **Go parallel when possible** \- it's often 5-10x faster than sequential Whether you're pulling data from APIs, building integrations, or automating workflows, httr2 handles the rest. Pick an API you're interested in and start sending requests.