7 ways to scrape Home Depot in 2026

This guide shows how to scrape Home Depot product data (names, prices, availability, review scores) with your own Python code, without a paid scraping API. The obstacle is Akamai Bot Manager, which answers a plain requests call with a 403 before any HTML arrives.

Nearly every guide on page one for this query ends the same way: sign up for the author's scraping API and let it handle Akamai.

That gets you data, but you pay for usage, and when requests start failing you can't see why or fix it yourself.

The seven methods here all run on your own machine with curl_cffi and Playwright. They are ordered from "no anti-bot at all" to "needs a warmed session".

Start with the first method and move down the list only when it stops returning data. The earlier methods are lighter to run and have fewer parts to break.

Akamai gets its own section near the end. That section covers the cookie handoff the later methods depend on: a real browser session obtains the cookies, and the lighter HTTP client reuses them.

How do you scrape Home Depot?

To scrape Home Depot, list product URLs from its public XML sitemaps, then fetch each product with a Chrome TLS fingerprint from a US residential IP.

Read the JSON-LD block for basics, or replay the site's own federation-gateway GraphQL calls for specs, per-store prices, stock, and reviews. When Akamai blocks you, warm cookies in Playwright and reuse them.

Prerequisites

• Python 3.10 or newer (curl_cffi's minimum since v0.14)
• pip install curl_cffi parsel playwright, then playwright install chromium
• A US residential IP with sticky sessions for anything past Method 1
• Chrome DevTools, for capturing the GraphQL payloads in Methods 4 to 7

Every method shares one session and one fetch function. The fetch raises Blocked on Akamai's block page so the scraper stops instead of parsing an error page as a product.

import os
from curl_cffi import requests as cffi

PROXY = os.environ.get("HD_PROXY")  # e.g. http://user:pass@host:port, sticky US residential
HEADERS = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}
session = cffi.Session(impersonate="chrome",  # Chrome TLS + HTTP/2 fingerprint
                       proxies={"https": PROXY, "http": PROXY} if PROXY else None)

class Blocked(Exception):
    """Akamai answered with its block page."""

def fetch(url: str) -> str:
    resp = session.get(url, headers=HEADERS, timeout=30)
    if resp.status_code == 403 or "errors.edgesuite.net" in resp.text[:5000]:
        raise Blocked(f"{resp.status_code} on {url}")
    resp.raise_for_status()
    return resp.text

Akamai's block page links to errors.edgesuite.net and shows a reference number, so the check reads the body as well as the status code.

impersonate="chrome" tracks the newest Chrome profile your curl_cffi version ships. You'll pin a specific version once you pair it with a browser's User-Agent in the Akamai section.

Method 1: Find every product URL in Home Depot's sitemaps

Home Depot's robots.txt points to sitemap/main.xml, and its P/PIPs.xml child indexes 96 product files (PIP-0.xml to PIP-95.xml), each listing /p/ URLs with a lastmod date.

When I pulled those files on 4 November 2026, they loaded without any Akamai challenge. That makes the sitemaps the cheapest way to build a product list.

import random
import re
import time

PIP_INDEX = "https://www.homedepot.com/sitemap/P/PIPs.xml"
LOC = re.compile(r"<loc>\s*([^<\s]+)\s*</loc>")

def product_urls(max_files: int = 2):
    for child in LOC.findall(fetch(PIP_INDEX))[:max_files]:  # PIP-0.xml, PIP-1.xml, ...
        yield from LOC.findall(fetch(child))
        time.sleep(1)

def item_id(url: str) -> str:
    # https://www.homedepot.com/p/<slug>/<itemId>  ->  "<itemId>"
    return url.split("?")[0].rstrip("/").rsplit("/", 1)[-1]

The trailing number in a /p/ URL is the item ID (the "Internet #" on the page). Every GraphQL call later in this guide keys on it, so store it as your primary key.

The same index lists B/PLPs.xml for category pages. Their URLs end in /N-5yc1v..., and that N- value is the navParam Method 5 needs.

Method 2: Read product data from the JSON-LD block

Home Depot product pages ship a schema.org Product object in a <script type="application/ld+json"> tag, rendered on the server, so you can read it without running JavaScript.

That block carries the name, brand, SKU, model number, price, and rating summary, which covers most price-monitoring jobs.

import json
from parsel import Selector

def json_ld_product(html: str) -> dict:
    for raw in Selector(text=html).css('script[type="application/ld+json"]::text').getall():
        try:
            data = json.loads(raw)
        except json.JSONDecodeError:
            continue  # some pages ship more than one block, not all of it valid
        for node in data if isinstance(data, list) else [data]:
            if isinstance(node, dict) and node.get("@type") == "Product":
                return node
    return {}

The parser flattens the parts you'll store. Offers can arrive as an object or a list, so it handles both:

def parse_ld(node: dict) -> dict:
    offers = node.get("offers") or {}
    offers = offers[0] if isinstance(offers, list) and offers else offers
    brand = node.get("brand") or {}
    rating = node.get("aggregateRating") or {}
    return {
        "name": node.get("name"),
        "brand": brand.get("name") if isinstance(brand, dict) else brand,
        "sku": node.get("sku"),
        "model": node.get("model") or node.get("mpn"),
        "gtin": node.get("gtin13") or node.get("gtin12"),
        "price": offers.get("price"),
        "availability": offers.get("availability"),
        "rating": rating.get("ratingValue"),
        "review_count": rating.get("reviewCount"),
    }

url = "https://www.homedepot.com/p/RIDGID-18V-SubCompact-Brushless-Cordless-6-1-2-in-Circular-Saw-Tool-Only-R8656B/315976374"
print(parse_ld(json_ld_product(fetch(url))))

The JSON-LD price belongs to whichever store the session is localized to, and a fresh session gets a default store. If price per location matters, skip ahead to Method 6.

Method 3: Parse the Apollo cache Home Depot embeds in the page

Home Depot pages embed their GraphQL client cache as window.__APOLLO_STATE__, a flat JSON object holding every product the page rendered, carousels included.

Reach for it when JSON-LD is missing a field, or when a category page's HTML is all you have.

import re

APOLLO = re.compile(r"__APOLLO_STATE__\s*=\s*")

def apollo_state(html: str) -> dict:
    m = APOLLO.search(html)
    if not m:
        return {}
    # raw_decode stops at the closing brace and ignores the JavaScript after it
    state, _ = json.JSONDecoder().raw_decode(html, m.end())
    return state

Apollo normalizes its cache, which trips up most first attempts. Nested objects become {"__ref": "..."} pointers, and fields with arguments get keys like pricing({"storeId":"121"}) instead of pricing.

Two helpers absorb both quirks:

def deref(state: dict, value):
    return state.get(value["__ref"], {}) if isinstance(value, dict) and "__ref" in value else value

def field(state: dict, obj: dict, name: str):
    if name in obj:
        return deref(state, obj[name])
    for key, value in obj.items():  # pricing({"storeId":"121"}) and friends
        if key.startswith(name + "("):
            return deref(state, value)
    return None

With those, pulling products out of the cache is a filter over its values:

def apollo_products(state: dict) -> list:
    rows = []
    for obj in state.values():
        if not isinstance(obj, dict) or "itemId" not in obj:
            continue
        ids = field(state, obj, "identifiers") or {}
        price = field(state, obj, "pricing") or {}
        rows.append({
            "item_id": obj["itemId"],
            "title": ids.get("productLabel"),
            "brand": ids.get("brandName"),
            "model": ids.get("modelNumber"),
            "price": price.get("value"),
            "was_price": price.get("original"),
        })
    return rows

Dump one product object with json.dumps(obj, indent=2) before you trust the column names. The field names above match public write-ups of Home Depot's schema, and that schema drifts.

Method 4: Replay the productClientOnlyProduct GraphQL call

Home Depot's product page fetches its full data by POSTing to /federation-gateway/graphql?opname=productClientOnlyProduct, and replaying that request gets you specs, UPC, images, and pricing as clean JSON.

Don't hand-write the query. Capture it from your own browser so the field list matches what the gateway accepts today:

1. Open any product page in Chrome and press F12.
2. In the Network tab, filter by federation-gateway, then reload.
3. Click the request with opname=productClientOnlyProduct, open Payload, and click "view source".
4. Save that JSON (it has operationName, variables, and query) as queries/productClientOnlyProduct.json.

The replay function loads a saved payload and overrides only the variables you name. It refuses names the capture doesn't contain, which catches renamed variables early:

from pathlib import Path

GATEWAY = "https://www.homedepot.com/federation-gateway/graphql"

def graphql(opname: str, overrides: dict, referer: str) -> dict:
    payload = json.loads(Path("queries", f"{opname}.json").read_text())
    for key, value in overrides.items():
        if key not in payload["variables"]:
            raise KeyError(f"{key!r} not in captured variables for {opname}")
        payload["variables"][key] = value
    resp = session.post(f"{GATEWAY}?opname={opname}", json=payload, timeout=30, headers={
        "Origin": "https://www.homedepot.com", "Referer": referer,
        "x-experience-name": "general-merchandise"})  # copy the value your capture shows
    if resp.status_code == 403:
        raise Blocked(f"403 on {opname}")
    resp.raise_for_status()
    body = resp.json()
    if body.get("errors") and not body.get("data"):
        raise ValueError(body["errors"][0].get("message"))
    return body["data"]

Check the request headers in your capture as well as the payload. The browser sends x-experience-name, and the gateway can answer differently without it.

The product call itself is short:

def get_product(item: str, **store) -> dict:
    data = graphql("productClientOnlyProduct", {"itemId": item, **store},
                   referer=f"https://www.homedepot.com/p/{item}")
    return data.get("product") or {}

product = get_product("315976374")
ids = product.get("identifiers") or {}
specs = {s.get("specName"): s.get("specValue")
         for group in product.get("specificationGroup") or []
         for s in group.get("specifications") or []}
print(ids.get("productLabel"), ids.get("upcGtin13"), len(specs), "specs")

The spec groups are the reason to bother with GraphQL. JSON-LD has no dimensions, voltage, or warranty fields, and the GraphQL response nests all of them under specificationGroup.

Method 5: Page through category listings with searchModel

Category and search pages load their product grid through a searchModel GraphQL operation that takes a navParam (the N- value from a /b/ URL), a store ID, and startIndex / pageSize variables for paging.

Capture it the same way as Method 4, from a category page, and save it as queries/searchModel.json. Some captures name the operation psSearchModel instead; use whatever yours says.

NAV = re.compile(r"/N-([0-9A-Za-z]+)")

def browse(category_url: str, store_id: str, pages: int = 5, page_size: int = 24):
    nav = NAV.search(category_url).group(1)  # /b/Tools-Power-Tools-Drills/N-5yc1vZc2bk
    for page in range(pages):
        data = graphql("searchModel", {"navParam": nav, "storeId": store_id,
                                       "startIndex": page * page_size, "pageSize": page_size},
                       referer=category_url)
        model = data.get("searchModel") or {}
        products = model.get("products") or []
        yield from products
        total = (model.get("searchReport") or {}).get("totalProducts") or 0
        if not products or (page + 1) * page_size >= total:
            break
        time.sleep(random.uniform(2, 4))

Each product in the grid carries an itemId, so a category crawl doubles as a discovery step for Method 4. Stop at totalProducts rather than looping until a page comes back empty.

Prefer category pages over keyword search. Home Depot's robots.txt disallows /s/ search URLs, while plain /b/ category pages are open to crawlers.

It also blocks /b/Featured-Products and any URL carrying NCNI-5 or Ntt-, so strip those parameters from your category list.

Method 6: Scrape Home Depot prices and stock for a specific store

Home Depot prices and inventory are per store, and the product GraphQL call returns that store's price and fulfillment block once you pass storeId and zipCode variables.

A store's ID is the number at the end of its store page URL under homedepot.com/l/. You can also pick a store on the site and copy storeId from any gateway request in DevTools.

Capture productClientOnlyProduct from a product page with a store already selected. Then both store variables exist in the saved payload, and the replay function will accept them.

from datetime import datetime, timezone

def find_key(obj, key):
    if isinstance(obj, dict):
        for k, v in obj.items():
            if k == key:
                yield v
            yield from find_key(v, key)
    elif isinstance(obj, list):
        for item in obj:
            yield from find_key(item, key)

find_key searches by name at any depth, which survives Home Depot reshuffling the wrapper objects. The snapshot function loops stores and records one row per fulfillment option:

def store_snapshot(item: str, stores: dict) -> list:
    rows, now = [], datetime.now(timezone.utc).isoformat()
    for store_id, zip_code in stores.items():
        product = get_product(item, storeId=store_id, zipCode=zip_code)
        price = (product.get("pricing") or {}).get("value")
        for options in find_key(product.get("fulfillment") or {}, "fulfillmentOptions"):
            for opt in options or []:
                inv = next(find_key(opt, "inventory"), None) or {}
                rows.append({"item_id": item, "store_id": store_id, "price": price,
                             "option": opt.get("type"), "in_stock": inv.get("isInStock"),
                             "quantity": inv.get("quantity"), "checked_at": now})
        time.sleep(random.uniform(2, 4))
    return rows

Pickup and delivery are separate options, each with its own inventory. A drill can be out of stock for pickup at one store and still ship, so keep the option column.

If your capture returns no fulfillment block, the page loaded it through a separate fulfillment operation. Capture that one too and call it with the same two variables.

Append these rows instead of overwriting them. A price history needs one row per check, and our guide to storing scraped data compares SQLite and JSON Lines for that.

Method 7: Collect product reviews page by page

Home Depot loads reviews through a reviews GraphQL operation that takes the item ID plus a start index and page size, so paging means bumping the start index until a page comes back short.

Capture it by opening a product's reviews and clicking to page 2. Save the payload as queries/reviews.json and check which variable names hold the offset and the page size.

REVIEW_START, REVIEW_SIZE = "startIndex", "pagesize"  # rename to match your capture

def reviews(item: str, max_reviews: int = 300):
    captured = json.loads(Path("queries/reviews.json").read_text())["variables"]
    start, size = captured.get(REVIEW_START, 1), captured.get(REVIEW_SIZE, 10)
    fetched = 0
    while fetched < max_reviews:
        data = graphql("reviews", {"itemId": item, REVIEW_START: start},
                       referer=f"https://www.homedepot.com/p/{item}")
        page = next(find_key(data, "Results"), None) or next(find_key(data, "results"), [])
        yield from page
        fetched += len(page)
        if len(page) < size:
            break
        start += len(page)
        time.sleep(random.uniform(2, 4))

The code yields raw review objects on purpose. I wouldn't hard-code review field names from any guide, this one included; print one review, then map the fields you need (rating, title, text, date, verified flag).

Write them to JSON Lines as you go. Popular tools have thousands of reviews, and a crash at review 2,400 shouldn't cost you the first 2,399.

How to scrape Home Depot past Akamai without a paid API

Home Depot sits behind Akamai Bot Manager, which scores each visitor at the edge on IP reputation, TLS fingerprint, and a JavaScript sensor that sets the _abck cookie.

Fail the score and you get a 403 "Access Denied" page with an Akamai reference number.

Our Akamai bypass guide covers those checks in general. On Home Depot, three things decide whether Methods 2 to 7 work.

Check which vendor you're dealing with

At least one 2026 Home Depot scraping write-up blames HUMAN (PerimeterX). The cookies tell you which vendor answered, so look before you tune anything:

Home Depot's responses set _abck, bm_sz, and an akavpau_prod waiting-room cookie, and its block pages come from errors.edgesuite.net, Akamai's error host. If you ever see _px3 instead, the vendor changed and this section needs a different playbook.

Warm the session in a real browser, then hand off the cookies

The pattern I'd start with is to let a real Chromium pass the sensor check once, then reuse its cookies and User-Agent in curl_cffi for the fast GraphQL calls.

from playwright.sync_api import sync_playwright

def warm_session(proxy: dict | None = None) -> tuple[list, str]:
    # proxy = {"server": "http://host:port", "username": "...", "password": "..."}
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False, proxy=proxy)  # headless is easier to flag
        page = browser.new_page(locale="en-US", timezone_id="America/New_York")
        page.goto("https://www.homedepot.com/", wait_until="domcontentloaded")
        page.mouse.move(420, 310)
        page.mouse.wheel(0, 1400)  # some interaction for the sensor script to report
        page.wait_for_timeout(6000)
        page.goto("https://www.homedepot.com/b/Tools/N-5yc1vZc1xy", wait_until="domcontentloaded")
        page.wait_for_timeout(4000)
        cookies, ua = page.context.cookies(), page.evaluate("navigator.userAgent")
        browser.close()
    return cookies, ua

On a Linux server, run it under xvfb-run so the headed browser has a display. The handoff copies each cookie with its domain, so curl_cffi sends it back to the right host:

def handoff(cookies: list, ua: str) -> None:
    for c in cookies:
        session.cookies.set(c["name"], c["value"], domain=c["domain"], path=c["path"])
    session.headers.update({"User-Agent": ua})  # must match the browser that earned the cookies

cookies, ua = warm_session()
handoff(cookies, ua)

Match the curl_cffi fingerprint to that browser, too. Read the Chrome major version from ua and pin the closest target curl_cffi offers, so the TLS handshake and the User-Agent tell the same story.

Re-warm when Blocked starts firing again. A wrapper that catches it once, calls warm_session(), and retries the request keeps long runs going without a timer.

Keep the IP and the pace boring

Datacenter IPs fail Akamai's reputation check before the sensor matters, so use US residential addresses. Keep one sticky IP for the warm-up and every request that reuses its cookies.

A cookie earned on one IP and replayed from another is a mismatch Akamai can see. Residential vs datacenter proxies explains why the ASN carries that much weight.

Roundproxies sells sticky US residential sessions; any provider that pins an IP for 10 minutes or more works the same way.

Pace each worker at one request every two to four seconds, which is what the random.uniform(2, 4) sleeps above do. Spread volume across more sessions rather than pushing one session faster.

Fallback: capture Home Depot's GraphQL responses in Playwright

If the warm-up passes but your replayed GraphQL calls still fail, stop replaying and record. Let Playwright browse the pages and collect the gateway's responses as the browser receives them.

from contextlib import suppress

def capture_gateway(urls: list) -> list:
    payloads = []
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        page = browser.new_page(locale="en-US")
        def on_response(resp):
            if "/federation-gateway/graphql" in resp.url and resp.ok:
                with suppress(Exception):  # a few gateway replies aren't JSON
                    payloads.append(resp.json())
        page.on("response", on_response)
        for url in urls:
            page.goto(url, wait_until="networkidle")
            page.wait_for_timeout(random.uniform(2000, 4000))
        browser.close()
    return payloads

Feed the payloads to find_key and the parsers above unchanged.

It's slower per product, so route only the URLs that fail the fast path here (Playwright's network docs and our guide to intercepting network traffic cover the event API).

What none of these methods get you

Account-specific prices are out of reach on purpose, Pro Xtra pricing included, because they need a login and this guide doesn't touch accounts.

Items whose price shows only in the cart won't expose it on the product page either.

And there's no national price to scrape: every number is tied to a store, so record the store ID next to every price.

Troubleshooting

"Access Denied" (403) with an Akamai reference number

Why: Akamai rejected the request at the edge, usually because of a datacenter IP, a non-browser TLS fingerprint, or a missing _abck cookie.

Fix: Confirm impersonate="chrome" is set and the IP is residential, then run warm_session() and handoff() through that same IP.

"Oops!! Something went wrong. Please refresh page."

Why: A Home Depot-branded error page came back with a 200, so fetch() didn't raise and the parsers returned empty dicts.

Fix: Treat an empty json_ld_product() result as a block, not a missing product. Re-warm the session and retry once before marking the URL as failed.

GraphQL returns 200 with "errors" and "data": null

Why: The captured query references a field the gateway no longer serves, or you passed a variable the operation doesn't declare.

Fix: Re-capture the payload from DevTools on a live page. The KeyError in graphql() catches renamed variables before the request goes out.

Prices don't match what you see in the browser

Why: Your browser has a store selected and your scraper uses the session's default store.

Fix: Pass storeId and zipCode on every call (Method 6) and save the store ID with each price.

FAQ

Is it legal to scrape Home Depot?

Scraping public product pages is lawful in many jurisdictions, but Home Depot's Terms of Use still govern how you use its site.

Stick to public catalog data, stay out of accounts and checkout, and keep request rates low. Get legal advice before any commercial use; this isn't legal advice.

Does Home Depot have a public API for product data?

No public product API is open for anyone to sign up for and query. The website itself runs on an internal federation-gateway GraphQL endpoint, which Methods 4 to 7 replay. It's undocumented and changes without notice.

Can you scrape Home Depot without a headless browser?

Yes, for most requests. Sitemaps, JSON-LD, the Apollo cache, and the GraphQL calls are all plain HTTP once your session carries valid Akamai cookies.

You need a real browser for the cookie warm-up, and as a fallback for URLs that keep failing.

Do you need proxies to scrape Home Depot?

Yes, past a handful of pages. Akamai scores IP reputation first, so datacenter and VPN addresses tend to get a 403 before anything else is checked.

Use sticky US residential IPs and keep each warmed session on one address.

How do I find a Home Depot store ID?

A store ID is the number at the end of the store's page URL under homedepot.com/l/. You can also pick a store on the site and copy the storeId value from any federation-gateway request in DevTools.

Pass it with the store's ZIP code to get that store's prices and stock.

What does Home Depot's robots.txt allow?

Home Depot's robots.txt allows product pages under /p/ and category pages under /b/, and it disallows search pages under /s/, cart, checkout, and account paths.

It publishes its sitemap index at /sitemap/main.xml. robots.txt is a crawling convention, not a license, so read the Terms of Use too.

Wrapping up

Start with Method 1 and a few hundred URLs from one PIP file. It's the only step that works without an Akamai session, and it tells you which item IDs to feed everything else.

Then capture productClientOnlyProduct in DevTools, run warm_session() once, and replay Method 4 for ten items before you scale anything.

The parts that rot are the captured payloads and the cookie names. When runs start failing, re-capture first and re-check Home Depot's robots.txt and the curl_cffi releases second.