7 ways to scrape Foot Locker

You've built a scraper that works fine on static sites. Then you point it at footlocker.com and get a wall of HTML that says "Access Denied" — or worse, a 200 response with zero product data.

Foot Locker runs Akamai Bot Manager and renders everything client-side with React. A naive requests.get() won't even see a product listing. But the site still serves data to browsers, which means your scraper can get it too — if you know where to look.

Here are seven proven methods to scrape Foot Locker for product data, prices, and inventory — ordered from simplest to most advanced.

How to Scrape Foot Locker: Quick Answer

Scraping Foot Locker requires bypassing Akamai Bot Manager and handling a JavaScript-rendered React frontend. The fastest approach is intercepting Foot Locker's internal product API, which returns clean JSON without parsing HTML. For heavier scraping, combine Playwright with stealth plugins and residential proxy rotation to avoid IP bans and TLS fingerprint detection.

Why Foot Locker Is Hard to Scrape

Foot Locker isn't a WordPress blog. It's a React single-page application protected by one of the most widely deployed anti-bot systems on the internet.

Here's what you're up against when you send a request:

Akamai Bot Manager sits in front of every page. It checks your IP reputation, analyzes your TLS handshake (JA3 fingerprint), runs client-side JavaScript challenges, and sets tracking cookies (_abck and bm_sz) that determine whether you're human or a bot.

Akamai also compares your HTTP/2 frame settings against known browser profiles. Even if your headers look right, a mismatched cipher suite gives you away.

Client-side rendering means the initial HTML response contains almost no product data. Foot Locker's frontend is built with React. The browser loads JavaScript bundles that fetch product information from internal APIs and render it into the DOM.

If you curl a product page, you'll see a skeleton <div id="root"></div> and a pile of script tags — zero product names, zero prices, zero sizes. Standard HTTP libraries like requests or urllib see this empty shell.

Rate limiting and IP blocking kick in fast. Even if you bypass the initial challenge, hammering the same endpoint from a single datacenter IP will get you blocked within minutes.

Akamai uses behavioral analysis too. If your request pattern doesn't look like a human browsing — e.g., you never visit category pages, only product URLs — that's a red flag.

Knowing which layer is blocking you determines which method to use. A 403 usually means Akamai caught your fingerprint. A 200 with challenge HTML means JS execution failed. An empty DOM means client-side rendering wasn't handled.

Methods Overview

Start with Method 1. It's the fastest path to clean data and doesn't require a browser at all.

1. Intercept the Internal Product API

This is the method most people miss, and it's by far the most efficient.

Foot Locker's frontend doesn't embed product data in HTML. It fetches it from internal API endpoints that return structured JSON. If you can replicate those requests, you skip both HTML parsing and browser automation entirely.

Best when: You need product details, pricing, sizes, or inventory for known SKUs or categories.

Finding the endpoints

Open footlocker.com in Chrome, navigate to any product page, and open DevTools (F12). Click the Network tab and filter by "Fetch/XHR."

You'll see requests hitting endpoints like:

https://www.footlocker.com/api/products/pdp/[SKU]
https://www.footlocker.com/api/search?query=[term]

The responses come back as JSON with product names, prices, available sizes, images, and stock status.

Replicating the request

The trick is sending the right headers. Foot Locker's API checks for a valid session and expects browser-like request signatures.

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/131.0.0.0 Safari/537.36",
    "Accept": "application/json",
    "Accept-Language": "en-US,en;q=0.9",
    "Referer": "https://www.footlocker.com/",
    "x-fl-request-id": "your-unique-request-id",  # generate a UUID
    "Origin": "https://www.footlocker.com",
}

# Fetch product data by SKU
url = "https://www.footlocker.com/api/products/pdp/V0846133"
response = requests.get(url, headers=headers, timeout=10)

if response.status_code == 200:
    data = response.json()
    print(f"Name: {data.get('name')}")
    print(f"Price: {data.get('price')}")
    print(f"Sizes: {data.get('availableSizes')}")
else:
    print(f"Blocked: {response.status_code}")

This won't always work out of the box. Akamai may still challenge the request if your IP or TLS fingerprint looks suspicious. If you get a 403, move to Method 2 or 3.

Tradeoff: API endpoints can change without notice. Foot Locker doesn't document these publicly, so you'll need to re-check DevTools if your scraper breaks.

2. Set Proper Headers and Session Cookies

Sometimes a simple HTTP request works — if you look enough like a real browser.

Akamai's first line of defense is checking whether your request headers match what a real browser sends. Most scraping failures happen because the default python-requests User-Agent screams "I'm a bot."

Best when: You need to grab a handful of pages and don't want to spin up a browser.

import requests
import uuid

session = requests.Session()

# Step 1: Visit the homepage to collect Akamai cookies
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/131.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;"
              "q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1",
})

# Warm the session — Akamai sets _abck and bm_sz here
session.get("https://www.footlocker.com/", timeout=15)

# Step 2: Now request the actual product page
product_url = "https://www.footlocker.com/product/nike-air-force-1-07-mens/CW2288111.html"
response = session.get(product_url, timeout=15)
print(f"Status: {response.status_code}")
print(f"Cookies: {[c.name for c in session.cookies]}")

Check the cookies after the first request. If you see _abck and bm_sz, Akamai's tracking is active. The values in those cookies determine whether your next request passes or gets challenged.

Tradeoff: This method breaks the moment Akamai's JavaScript challenge kicks in. Without executing JS, you can't solve the challenge, and the cookie values stay invalid. For anything beyond basic access, you need a real browser.

3. Playwright with Stealth Plugins

When HTTP requests alone won't cut it, a headless browser with anti-detection patches is the next step.

Playwright launches a real Chromium instance that executes JavaScript, solves Akamai's challenges automatically, and gives you the fully rendered DOM — exactly what a human sees.

Best when: You need JS-rendered content, or the API interception method returns 403s.

Setup

pip install playwright playwright-stealth
playwright install chromium

Scraping a product page

import asyncio
from playwright.async_api import async_playwright
from playwright_stealth import stealth_async

async def scrape_footlocker(url):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            viewport={"width": 1920, "height": 1080},
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                       "AppleWebKit/537.36 (KHTML, like Gecko) "
                       "Chrome/131.0.0.0 Safari/537.36",
        )
        page = await context.new_page()
        await stealth_async(page)  # patches detection signals

        await page.goto(url, wait_until="networkidle")

        # Wait for product data to render
        await page.wait_for_selector('[data-testid="product-name"]',
                                      timeout=10000)

        # Extract product info from the rendered DOM
        name = await page.text_content('[data-testid="product-name"]')
        price = await page.text_content('[data-testid="product-price"]')

        print(f"Product: {name}")
        print(f"Price: {price}")

        await browser.close()

asyncio.run(scrape_footlocker(
    "https://www.footlocker.com/product/nike-air-force-1-07-mens/CW2288111.html"
))

The stealth_async call patches common browser fingerprinting leaks — things like navigator.webdriver being set to true, missing plugins, and inconsistent screen dimensions. Without it, Akamai detects Playwright instantly.

Tradeoff: Headless browsers are slow and resource-heavy. Each page load takes 3-10 seconds and consumes real memory. Fine for hundreds of products, painful for tens of thousands.

Pro tip: intercept network requests

Instead of parsing the DOM, you can intercept the API calls Playwright makes internally. This gives you clean JSON from Method 1 while using the browser's session to bypass Akamai.

async def intercept_api(route, request):
    if "/api/products/" in request.url:
        response = await route.fetch()
        body = await response.json()
        print(f"Intercepted: {body.get('name')} - ${body.get('price')}")
    await route.continue_()

await page.route("**/api/products/**", intercept_api)

This is the best of both worlds — browser-level authentication with API-level data quality.

4. Scrapy with scrapy-impersonate

For large-scale crawling, Scrapy is still the best framework. The problem is Scrapy's default HTTP handler uses urllib3, which has a TLS fingerprint that Akamai recognizes immediately.

scrapy-impersonate replaces that handler with curl_cffi, which mimics the TLS handshake of real browsers (Chrome, Firefox, Safari). This alone bypasses Akamai's network-level fingerprinting on many Foot Locker pages.

Best when: You're crawling thousands of product pages and need Scrapy's pipeline, concurrency, and export features.

Setup

pip install scrapy scrapy-impersonate

Spider

# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

# Impersonate Chrome 131
IMPERSONATE = "chrome131"

# Be polite
DOWNLOAD_DELAY = 2
CONCURRENT_REQUESTS = 4

# spiders/footlocker_spider.py
import scrapy
import json

class FootlockerSpider(scrapy.Spider):
    name = "footlocker"
    start_urls = [
        "https://www.footlocker.com/category/mens/shoes.html",
    ]

    def parse(self, response):
        # Look for product links in the rendered page
        product_links = response.css(
            'a[href*="/product/"]::attr(href)'
        ).getall()

        for link in product_links:
            yield response.follow(link, self.parse_product)

        # Follow pagination
        next_page = response.css(
            'a[aria-label="Next"]::attr(href)'
        ).get()
        if next_page:
            yield response.follow(next_page, self.parse)

    def parse_product(self, response):
        # Extract JSON-LD structured data if available
        json_ld = response.css(
            'script[type="application/ld+json"]::text'
        ).get()

        if json_ld:
            data = json.loads(json_ld)
            yield {
                "name": data.get("name"),
                "price": data.get("offers", {}).get("price"),
                "sku": data.get("sku"),
                "url": response.url,
            }

The key insight from The Web Scraping Club's testing is that many Akamai-protected sites — including fashion and retail — only need proper TLS fingerprinting and updated headers to bypass. No browser required.

Tradeoff: This works for category and product pages but may fail on pages with aggressive JS challenges. If you hit walls, combine with proxy rotation (Method 5).

5. Residential Proxy Rotation

Every method above improves when you add proxy rotation. Akamai tracks IPs aggressively, and even a perfect browser simulation gets flagged if 500 requests come from the same datacenter IP.

Residential proxies route your traffic through real consumer ISP connections. To Akamai, your scraper looks like a person in Cleveland browsing sneakers on their home WiFi.

Best when: You're scraping at volume (1,000+ pages/day) and getting IP-banned despite correct headers and browser emulation.

Integration with requests

import requests

proxies = {
    "http": "http://user:pass@gate.proxy-provider.com:7777",
    "https": "http://user:pass@gate.proxy-provider.com:7777",
}

response = requests.get(
    "https://www.footlocker.com/api/products/pdp/CW2288111",
    headers=headers,  # use headers from Method 1
    proxies=proxies,
    timeout=15,
)

Integration with Playwright

context = await browser.new_context(
    proxy={
        "server": "http://gate.proxy-provider.com:7777",
        "username": "user",
        "password": "pass",
    },
    viewport={"width": 1920, "height": 1080},
)

If you're running this at scale and need residential IPs, Roundproxies offers rotating residential pools that work well for ecommerce targets like Foot Locker.

Rotation strategy

Don't rotate on every request. That looks more suspicious than a real user browsing.

Assign one IP per "session" of 10-20 requests, then rotate. This mimics how a real person browses — they don't change IPs between clicking a product and checking the price.

Tradeoff: Residential proxies cost money. Expect $1-15 per GB depending on provider. Factor this into your per-page cost calculation before committing to a large crawl.

6. Google Cache and Wayback Machine

This is the lazy method, and sometimes lazy is smart.

Google and Archive.org's crawlers are whitelisted by most sites, including Akamai-protected ones. They've already scraped the pages you want and cached the results.

Best when: You need historical data, one-off lookups, or a quick check before building a full scraper.

Google Cache

import requests

# Google's cache URL format
cached_url = (
    "https://webcache.googleusercontent.com/search"
    "?q=cache:footlocker.com/product/nike-air-force-1-07-mens/CW2288111.html"
)

response = requests.get(cached_url, timeout=15)
# Parse the cached HTML with BeautifulSoup

Wayback Machine API

import requests

# Check if Wayback Machine has a snapshot
cdx_url = (
    "https://web.archive.org/cdx/search/cdx"
    "?url=footlocker.com/product/*"
    "&output=json&limit=10&fl=timestamp,original"
)

response = requests.get(cdx_url, timeout=15)
snapshots = response.json()

for snapshot in snapshots[1:]:  # skip header row
    timestamp, url = snapshot
    archive_url = f"https://web.archive.org/web/{timestamp}/{url}"
    print(archive_url)

Tradeoff: Cached data is stale. Google re-crawls pages every few days to weeks, and Wayback Machine snapshots can be months old. Prices, stock levels, and availability will be outdated. Use this for market research, not real-time monitoring.

7. Full Production Pipeline: Browser + Proxy Chain + Queue

When you need to scrape Foot Locker's entire catalog reliably — thousands of products, daily updates, automatic retries — you need a production-grade setup.

This combines the best of the previous methods into a pipeline.

Best when: You're building a price tracker, inventory monitor, or data product that runs unattended.

Architecture

The pipeline has four components, each handling a different failure mode:

Job queue — Redis or RabbitMQ holds URLs to scrape. Failed jobs get retried with exponential backoff. Start with a 5-second delay, double it on each retry, cap at 5 minutes. After 3 failures, flag the URL for manual review — it might be a dead link or a page behind login.

Browser pool — Multiple Playwright instances, each with its own proxy and browser profile. Rotate profiles every 50-100 pages. Each profile should have a unique combination of viewport size, timezone, and language settings. This prevents Akamai from correlating multiple sessions to the same operator.

Data extraction — Intercept API responses (Method 3 pro tip) instead of parsing DOM. Falls back to DOM parsing if interception fails. Always validate the extracted data — check for expected fields, reasonable price ranges, and valid SKU formats before storing.

Storage — Write results to PostgreSQL or export as JSON/CSV. Include a scraped_at timestamp on every record so you can track freshness and detect stale data.

Simplified orchestrator

import asyncio
import random
from playwright.async_api import async_playwright
from playwright_stealth import stealth_async

PROXY_LIST = [
    {"server": "http://proxy1:7777", "username": "u1", "password": "p1"},
    {"server": "http://proxy2:7777", "username": "u2", "password": "p2"},
    # add more proxies for rotation
]

async def scrape_product(url, proxy):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(proxy=proxy)
        page = await context.new_page()
        await stealth_async(page)

        products = []

        # Intercept API calls for clean data
        async def capture(route, request):
            if "/api/products/" in request.url:
                resp = await route.fetch()
                body = await resp.json()
                products.append(body)
            await route.continue_()

        await page.route("**/api/products/**", capture)
        await page.goto(url, wait_until="networkidle", timeout=30000)

        await browser.close()
        return products

async def main():
    urls = [
        "https://www.footlocker.com/category/mens/shoes.html",
        "https://www.footlocker.com/category/womens/shoes.html",
    ]

    for url in urls:
        proxy = random.choice(PROXY_LIST)
        data = await scrape_product(url, proxy)
        print(f"Scraped {len(data)} products from {url}")
        await asyncio.sleep(random.uniform(2, 5))  # human-like delay

asyncio.run(main())

In production, you'd add retry logic, error logging, and a proper task queue. But this skeleton shows the core pattern: stealth browser, proxy rotation, API interception.

Tradeoff: Complexity. This takes real engineering time to build and maintain. Foot Locker periodically updates their site structure, Akamai configs change, and browser versions need updating. Budget for ongoing maintenance.

Which Method Should You Use?

Start simple. Try Method 1 first — it works more often than you'd expect. Only escalate when you hit a wall.

Troubleshooting Common Errors

"403 Forbidden" or "Access Denied"

Akamai blocked your request. Check your User-Agent header first — this is the most common cause. If headers look correct, your IP or TLS fingerprint is flagged. Switch to a residential proxy or use scrapy-impersonate / curl_cffi to fix TLS fingerprinting.

"200 OK" but empty or challenge HTML

Akamai returned a JavaScript challenge page with a 200 status code. Your HTTP client isn't executing JS. Switch to Playwright (Method 3) or check if the _abck cookie value starts with a specific pattern indicating a failed challenge.

Products render in browser but scraper sees nothing

The page uses client-side rendering. The data isn't in the initial HTML — it's fetched via JavaScript after page load. Use Playwright with wait_until="networkidle" or intercept API calls directly.

Rate limited after a few requests

You're hitting too many requests from one IP. Add delays between requests (2-5 seconds minimum) and rotate proxies. Akamai also tracks request patterns — vary your crawl order instead of scraping sequentially. If you're crawling category pages alphabetically, shuffle the URL list. Predictable patterns are easy for bot detection to spot.

"ERR_HTTP2_PROTOCOL_ERROR" or connection resets

Akamai sometimes mimics server errors to confuse bots. If you see connection-level errors that don't match real downtime, your TLS fingerprint is likely the problem. Switch to curl_cffi or scrapy-impersonate to send browser-matching TLS handshakes. This is especially common with Python's default urllib3 or httpx libraries, which have distinctive fingerprints that Akamai recognizes.

Scraper works once, fails on second run

Akamai may have flagged your IP after the first session. If you're testing without proxies, your home IP might be temporarily blacklisted. Wait 10-15 minutes, clear your cookies, and try again with a different User-Agent string. For development and testing, use a residential proxy from the start to avoid burning your own IP.

A Note on Responsible Scraping

Foot Locker's Terms of Service restrict automated access. Before building a scraper, consider what data you actually need and whether there's a less invasive way to get it.

Some ground rules worth following: respect robots.txt directives, add delays between requests so you don't strain their servers, cache aggressively so you're not re-scraping unchanged pages, and only collect publicly visible product data.

Web scraping publicly available data is generally legal, but the specifics depend on your jurisdiction and use case. If you're doing this commercially, talk to a lawyer first.

Wrapping Up

Foot Locker's combination of Akamai Bot Manager and React rendering makes it one of the trickier ecommerce sites to scrape. But none of those defenses are impenetrable.

The internal API (Method 1) is your best first move — it's fast, returns clean JSON, and avoids HTML parsing entirely. When that fails, Playwright with stealth patches (Method 3) handles the JS challenges. At scale, Scrapy with scrapy-impersonate and residential proxies (Methods 4 + 5) gives you the throughput without the IP bans.

Pick the simplest method that works for your use case, and only add complexity when you need it.