What is web scraping? A starter guide for developers

Say you need the price of 400 products from a competitor's site, every morning, in a spreadsheet. You open the first page, copy the price, paste it, and by product 30 you're thinking there has to be a better way.

There is, and most developers stumble into it backwards. They google a tutorial, paste 15 lines of Python, get a wall of HTML, and stop. Or worse, they get it working on Monday and it silently returns nothing on Thursday because the site changed one CSS class.

In this guide, I'll break down what web scraping is at the request-and-response level, how it differs from crawling and the other terms people mix it up with, and how to pick the right approach for a given site so your first scraper survives past the demo.

What is web scraping?

Web scraping is the automated extraction of data from websites using a program instead of a person. The program sends the same HTTP requests a browser sends, receives the HTML (or JSON) the server returns, and pulls out the fields you care about. Use it when a site has no public API. The most common setup is a request library plus an HTML parser, which is under 20 lines of code.

That's the whole idea. A browser fetches a page and paints it for your eyes; a scraper fetches the same page and reads it for your program.

The part beginners miss is that the server usually can't tell the difference at first. A request from curl and a request from Chrome are both a few hundred bytes of text asking for /products/42. The server responds with the same HTML either way. What you do with that HTML is where scraping starts.

Scraping exists because the web is the largest dataset there is, and almost none of it comes with a download button. Prices, listings, reviews, job postings, government filings, sports odds; they're published for humans in HTML, and HTML is a structured document. Structure is what makes extraction possible.

Web scraping vs. web crawling vs. data extraction vs. APIs

These four get lumped together in every glossary, and the confusion costs people real time. I've watched someone spend a week building a crawler when they had the 200 URLs they needed in a sitemap the whole time.

Suppose you're collecting every listing on a used-car marketplace. Crawling is the loop that starts at the search page, reads the "next" link, and collects 8,000 listing URLs into a queue. Scraping is what happens to each of those URLs: request it, parse it, take the price and mileage and year. Data extraction is the parser itself, which would work identically on a listing you saved to disk last month. And if the marketplace had a documented API, you'd skip all three and page through JSON instead.

One more distinction, because it shapes how people argue about scraping online. "Scraping" the technique is neutral engineering. "Scraper" the product category (browser extensions, no-code tools, managed APIs) is a market. This article is about the technique, and every example is code you run yourself.

The four stages of a web scraper

Every scraper, whether it's 12 lines or a distributed system, does the same four things. Get the stages straight and the tools stop looking mysterious.

1. Fetch

The scraper sends an HTTP request and gets a response. On the wire it looks like this:

GET /catalogue/page-1.html HTTP/1.1
Host: books.toscrape.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64) ...
Accept: text/html,application/xhtml+xml

The server answers with a status line, headers, and a body. Status 200 and a body full of HTML is what you want. 403, 429, or a 200 whose body is a "please verify you are human" page means the site has decided you're not a browser, and you've moved from scraping into the getting-blocked problem, which is a different article.

In practice, most of what separates a toy scraper from a working one lives in this stage: headers, timeouts, retries, and how many requests per second you send from one IP.

How it works in practice: a fetch with a real user agent and a timeout, which is the minimum I'd ship.

import requests

headers = {
    # Sites often 403 the default "python-requests/2.x" agent on sight
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/128.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

resp = requests.get("https://books.toscrape.com/catalogue/page-1.html",
                    headers=headers, timeout=10)
resp.raise_for_status()   # turns 4xx/5xx into an exception instead of silent garbage
html = resp.text
print(resp.status_code, len(html))

raise_for_status() is the line beginners skip and regret. Without it, a 403 page gets handed to your parser, the parser finds no products, and you get an empty CSV with no error.

2. Parse

HTML is a tree. Parsing turns the text you fetched into that tree so you can walk it. The parser doesn't care what the page is about; it cares that <div class="product"> contains <h3> which contains <a>.

Two things surprise people here. First, browsers repair broken HTML silently, and so do good parsers, so what you see in DevTools is often not byte-for-byte what the server sent. Second, the parser only sees what was in the response body. If a price was filled in by JavaScript after page load, it's not in the tree, and no selector will find it.

How it works in practice: Beautiful Soup with the lxml backend, which is the pairing I default to in Python.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")   # lxml is ~5x faster than html.parser on big pages

# One <article class="product_pod"> per book on this page
cards = soup.select("article.product_pod")
print(len(cards))   # 20 on books.toscrape.com

select() takes a CSS selector, the same syntax you'd use in DevTools. If the count comes back as 0, either the selector is wrong or the content wasn't in the HTML at all (see the three shapes section below).

3. Extract

Extraction is picking fields out of each node and cleaning them into the type you want. Prices arrive as "£51.77", not 51.77. Dates arrive as "3 days ago". Ratings arrive as a CSS class named Three. Every field needs a small conversion, and the conversions are where the bugs hide.

How it works in practice: pull three fields per card and normalise them.

def parse_card(card):
    title = card.select_one("h3 a")["title"]
    raw_price = card.select_one(".price_color").text     # "£51.77"
    price = float(raw_price.lstrip("£"))
    # Rating is encoded as a class: "star-rating Three"
    rating_word = card.select_one(".star-rating")["class"][1]
    rating = ["One", "Two", "Three", "Four", "Five"].index(rating_word) + 1
    return {"title": title, "price": price, "rating": rating}

rows = [parse_card(c) for c in cards]
print(rows[0])   # {'title': 'A Light in the Attic', 'price': 51.77, 'rating': 3}

Note the lstrip("£") and the .index() trick. Neither is clever, and both will break the day the site switches to "$" or renames a class. Extraction code is the part of a scraper you'll edit most, so keep it in small functions you can change without touching fetch or storage.

4. Store

Data you can't query later is data you didn't collect. For a one-off, a CSV is fine. For anything you run on a schedule, use a database with a unique key so re-runs update rows instead of duplicating them.

How it works in practice: SQLite with an upsert, zero dependencies beyond the standard library.

import sqlite3

con = sqlite3.connect("books.db")
con.execute("""CREATE TABLE IF NOT EXISTS books
               (title TEXT PRIMARY KEY, price REAL, rating INTEGER,
                scraped_at TEXT DEFAULT CURRENT_TIMESTAMP)""")
con.executemany(
    "INSERT INTO books(title, price, rating) VALUES (:title, :price, :rating) "
    "ON CONFLICT(title) DO UPDATE SET price=excluded.price, rating=excluded.rating",
    rows,
)
con.commit()

ON CONFLICT ... DO UPDATE is what turns a scraper into a price tracker: run it daily and each title keeps its latest price. For the full range of options (Postgres, Parquet, when to bother with either), see how to store scraped data.

The three shapes a page can take (and why it matters)

This section is the one I wish someone had shown me first. Before you write a single selector, figure out which of three shapes the page is, because each needs a different fetch strategy and the wrong choice wastes hours.

The test takes 30 seconds. Right-click, View Page Source, and Ctrl+F for a value you can see on screen. If it's there, you have static HTML and requests plus a parser will do. If it isn't, open DevTools, go to Network, filter by Fetch/XHR, reload, and look for a response containing your value. Nine times out of ten on modern sites, that's a JSON endpoint you can call directly.

How it works in practice: hitting a JSON endpoint instead of parsing the page that consumes it.

import requests

# Found in DevTools > Network > Fetch/XHR while loading the listings page
url = "https://example-shop.test/api/products"
params = {"page": 1, "per_page": 48, "sort": "price_asc"}

data = requests.get(url, params=params, timeout=10).json()
for item in data["products"]:
    print(item["sku"], item["price"]["amount"])

(The host is a placeholder; swap in whatever your Network tab shows.) No parser, no selectors, no HTML repair. JSON endpoints change less often than markup, and they're usually paginated with clean parameters. Always check for one before reaching for a browser.

Only when both tests fail do you need a real browser. Playwright is my pick; it's maintained by Microsoft, runs Chromium, Firefox, and WebKit, and waits for network activity properly.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/", wait_until="networkidle")
    # This page injects quotes with JS; requests would see none of them
    quotes = page.locator(".quote .text").all_text_contents()
    print(len(quotes), quotes[0][:60])
    browser.close()

Expect roughly 10x the CPU and memory per page compared to plain requests, and a much larger fingerprint for anti-bot systems to inspect. That's the trade: a browser sees everything a user sees, and the site sees everything about the browser. The dynamic web scraping in Python guide covers the full workflow, including intercepting the JSON the browser fetches so you can drop back down to shape two.

How to build your first web scraper

Understanding the stages is one thing; here's the order I'd build them in.

1. Check for an API or a feed before writing any scraper

Search site:target.com api, look for /sitemap.xml, check if there's an RSS feed, and inspect the Network tab. If any of those give you the fields you need, use them. It's less code and it's the access method the site prefers.

2. Run the three-shapes test on one page

View Source, then Network tab, then DevTools. Decide static, JSON, or rendered. Write that decision down; you'll forget it in a month when the scraper breaks.

3. Get one page working end to end

Fetch, parse, extract, store, for a single URL. Print the row. Check every field by eye against the live page. Only then add a loop. Most "my scraper returns nothing" bugs come from adding pagination before the single-page case worked.

4. Start with a delay and a retry, not a framework

Two lines get you further than most people expect: time.sleep(random.uniform(1, 3)) between requests, and a retry on 429 or 5xx with backoff. You don't need Scrapy, a proxy pool, or a job queue for 500 pages a day. Add those when you have a measured reason, such as a block rate you can see in your logs.

5. Log the things that will break

Log the status code, the response length, and the number of items extracted per page. A page that returns 200 with 4 KB instead of 90 KB is a block page. Zero items on a page that used to yield 20 means the markup changed. Both are silent without logging.

The full build, with pagination and error handling, is in the web scraping with Python tutorial. If Python isn't your language, the same four stages apply in JavaScript or Go.

Web scraping in practice: three real workloads

The stages are constant; which stage hurts depends on what you're scraping.

E-commerce price monitoring

Fetch is the hard stage. Large retailers use anti-bot vendors that inspect TLS fingerprints and IP reputation, so a scraper that ran fine for 200 requests starts getting 403 at request 201. Rotating IPs helps, and for the most protective sites you'll need residential rather than datacenter addresses (this is the one place I'll mention that Roundproxies sells both; test on your target before assuming either is enough). Parsing, by contrast, is easy: product pages are templated and rarely change.

Job board aggregation

Extract is the hard stage. Salary appears as "$80k–$95k", "80,000 - 95,000 USD", or "Competitive". Location is "Remote (US)" or "NYC / Hybrid". You'll spend more time writing normalisers than fetchers. Fetch is usually gentle because job boards want to be indexed.

Research and public records

Store and crawl are the hard stages. Government portals have thousands of near-identical pages behind paginated search forms, often with session tokens. The scraping itself is trivial; the crawling logic to enumerate every record without missing or duplicating any is the work. Use a database with a primary key from day one.

Is web scraping legal?

The short answer: scraping publicly accessible data is generally lawful in the US, and courts have mostly rejected the idea that it's "unauthorised access" under the CFAA when no login is bypassed. The long answer is that it depends on what you take and what you do with it, and I'm not a lawyer.

Things that push you toward trouble: scraping behind a login you agreed to terms for, collecting personal data without a lawful basis under GDPR or CCPA, republishing copyrighted content, and hammering a site hard enough to degrade it. Things that keep you on the right side: reading robots.txt (the format is standardised in RFC 9309), rate limiting, only taking what you need, and not pretending the data is yours.

Terms of service prohibiting scraping are a contract question, not a criminal one, but they can still get your account banned and, for a business, a demand letter. Read them for any site you scrape at volume.

Web scraping pitfalls (and how to avoid them)

Test the selector against the raw response, not DevTools

DevTools shows the DOM after JavaScript ran. Your parser sees the response body before it ran. Selectors that work in the console and return nothing in code almost always trace back to this. Print resp.text[:2000] and look.

Send a browser-like user agent from the first request

The default python-requests/2.32 agent is the most-blocked string on the internet. Setting a real Chrome UA fixes a large share of first-day 403s. It won't fool a real anti-bot system, but it stops the cheap filters.

Treat 200 as a status, not a success

Block pages, CAPTCHAs, and "no results" pages all come back 200. Check the response length and the item count, and alert when either drops. A scraper that "works" and has been storing CAPTCHA pages for two weeks is a common story.

Save the raw HTML, then parse it

Disk is cheap and re-fetching isn't. If you store the raw response alongside the parsed rows, you can fix an extraction bug and re-run the parser over last month's pages without touching the site again. This also makes you a better citizen: fewer requests.

Know when web scraping is the wrong tool

If the site offers an API, a bulk export, or a data licence, scraping is slower, more fragile, and possibly a terms violation, all to get the same bytes. And if what you need is behind a login and the terms forbid automated access, no amount of technique changes the legal picture. Scraping is the fallback for data that's public but not packaged, and it stops being the right answer the moment a package exists.

FAQ

Is web scraping the same as using an API?

No. An API is an interface the site built for programs, with documented endpoints and stable JSON. Web scraping reads pages built for humans and reverse-engineers the structure. APIs are better when they exist; scraping is for when they don't or when they're missing the fields you need.

Do I need a proxy to scrape a website?

Not to start. A single IP with polite delays can scrape thousands of pages a day from most sites. Proxies become necessary when the target rate-limits per IP and your volume exceeds that limit, or when the site blocks datacenter ranges outright. When proxies matter for web scraping walks through the decision.

Which language is best for web scraping?

Python has the most mature ecosystem (Requests, Beautiful Soup, Scrapy, Playwright) and the most tutorials, so it's the default. Node.js is a good second choice if you're already rendering with a browser. Go wins on raw throughput for large crawls. The four stages are identical in all of them.

Why does my scraper work in the browser but not in code?

Almost always one of three things: the content is JavaScript-rendered and isn't in the raw HTML, the site blocked your request and returned a block page with a 200, or you're missing a cookie or header the browser sends automatically. Check View Source first, then the status and length of the response, then compare headers in the Network tab.

Can websites detect web scraping?

Yes, with varying effort. Cheap detection looks at user agent and request rate. Serious detection (Cloudflare, Akamai, DataDome, and similar) inspects TLS fingerprints, browser fingerprints, and behavioural signals. Simple sites won't notice a polite scraper; heavily protected ones will notice almost anything.

Wrapping up

The mental model to keep: a scraper is a program that fetches what a browser fetches, parses the tree, extracts fields, and stores rows. Everything else, from proxies to headless browsers to distributed queues, is a fix for one of those four stages breaking at scale.

The first thing to build is a single-page scraper against books.toscrape.com, which exists for exactly this purpose. Get one row printed and correct. Then read the Python web scraping tutorial for pagination, retries, and the rest of the plumbing.