Web Scraping How to scrape a website Akamai

7 ways to scrape Idealo prices and price history

Contributor Jamie Flubbon Contributor Bernt

You can scrape Idealo prices and price history without paying for a scraping API, but a plain requests call won't get you there.

idealo.de sits behind Akamai. According to Scrapfly's October 2026 guide, Akamai answers a stock Python HTTP client with a 403 before any HTML loads. The check runs at the connection stage, so the fix belongs in how your client connects. That is the job curl_cffi does in this guide: it lets a Python script present itself the way a browser does during the handshake.

Once past that handshake, the site gives away a lot. A single product page fetch returns three useful layers of data:

  • A JSON-LD summary of the product, which you can parse without touching the visual layout
  • The first 20 shop offers, so one request covers the current price spread across retailers
  • The product ID that Idealo's JSON price-chart endpoint takes, which is where the price history comes from

That means current prices and historical prices both start from the same page, and you rarely need a full browser to collect them.

Most top-ranking results for this topic sell something: an Apify actor, a hosted scraping API, or a tutorial whose code only runs through its vendor's SDK. The seven methods here use open-source libraries, and all of the requests leave from code you run on your own machine or server. You decide the request rate, the storage format, and when to rerun the job.

Libraries used: curl_cffi, BeautifulSoup, and Playwright for one step
Data covered: JSON-LD product summary, the first 20 shop offers, price-chart JSON for price history
Paid API required: No

How do you scrape Idealo?

To scrape Idealo prices, request a product page at /preisvergleich/OffersOfProduct/{id}.html with a Chrome TLS fingerprint from a German IP. Then read the JSON-LD Product block and the productOffers-listItem rows. That one request returns the price range, GTIN, rating, and the first 20 shop offers. Load the remaining offers in blocks of 20, and call the price-chart endpoint for up to a year of daily prices.

Why the request setup matters

Idealo.de is a German comparison site, and the offers it shows depend on where the request comes from. A German IP gets the German shop list and euro prices, which match what a shopper in Germany sees. If you scrape from a data center abroad, you can get blocked, or you can get data that doesn't match the live site.

The Chrome TLS fingerprint matters because a plain HTTP client identifies itself during the TLS handshake, before Idealo reads any headers. Copying Chrome's handshake removes the most common giveaway for a scripted request. For German exits, a residential pool with country-level targeting covers the IP side. The Roundproxies residential proxy pool spans 195+ countries, including Germany, and supports city-level targeting.

Pick the method that fits your question

Each of the seven methods below answers a different question. Pick the ones that match what you need to know:

  • Current price for a product you already know: method 1. Add method 2 if you need shops beyond the first 20 offers.
  • How a price moved over time: method 3. It takes 2 requests per product and returns the daily low price for up to a year.
  • Finding products in a category: method 4. Each category page lists roughly 36 products.
  • Linking your own catalog to Idealo: method 5. It maps each EAN to an Idealo product ID in 2 to 6 requests. You only do this once per SKU, then store the ID.
  • Comparing prices across borders: method 6. It takes one request per market for AT, FR, IT, ES, and UK.
  • Daily monitoring at scale: method 7. Diff the category cards at about 1 request per 36 products, and only re-fetch product pages whose card price changed.

The table shows what each method returns, what it costs in requests, and which change on Idealo's side is most likely to break it. The last column tells you what to watch in your scraper's error logs.

Method What you get Requests Breaks when
1. Product page Price range, GTIN, rating, first 20 offers 1 per product Offer-row classes change
2. Offer blocks Every shop offer 1 per 20 offers The load-more request changes
3. Price chart Daily low price, up to 1 year 2 per product The chart path moves
4. Category pages ~36 products per page 1 per page Card classes change
5. EAN matching Your SKU mapped to an Idealo ID 2 to 6, once per SKU Search stops matching barcodes
6. Country sites AT, FR, IT, ES, UK prices 1 per market A market's URL path changes
7. Card diffing Which products moved today ~1 per 36 products Card prices go stale

How the methods combine in practice

A typical price-monitoring setup chains the methods together. Run method 5 once to map your SKUs to Idealo IDs. Run method 7 daily to find which products moved. Then call methods 1 and 2 only for the products that changed. Use method 3 when you need a longer trend than your own scrape history covers.

Card prices on category pages can lag behind the product page. Treat a card diff as a signal to re-check a product, and confirm the price on its product page before you reprice or send an alert.

Can idealo be trusted?

Yes, idealo can be trusted as a price reference, as long as you check a few things before acting on a number. Each product page carries a JSON-LD summary with the lowest price, the highest price, and the offer count across every listed shop. The figures you scrape come from the same data a shopper sees.

Four gaps are worth knowing:

  • Card prices on category pages can lag behind the product page, so confirm a price there before you rely on it.
  • The offer list holds new-item offers only. Used and B-stock offers sit elsewhere on the page, so the real floor price can be lower than the minimum you collect.
  • Shops and prices depend on where the request comes from. A German IP gets the German shop list in euros.
  • Shipping appears on its own line, so compare totals rather than headline prices.

Each offer also carries a shop ID that matches the shop's profile URL, so you can look up the retailer behind any price.

Is idealo free?

Yes, idealo is free to use. Product pages, shop offer lists, and price charts are public pages that load without a fee. Scraping them can also cost nothing in software: every method in this guide runs on open-source libraries (curl_cffi, BeautifulSoup, and Playwright) and needs no paid API.

The main cost is on your side. You need a German IP for idealo.de, plus one per country if you compare markets, and that usually means paying for a proxy plan.

Hosted tools price their idealo scrapers differently. Octoparse lists its Idealo template with no per-line charge. Bright Data offers 5K free records a month, and Oxylabs' free trial covers up to 2K results. Beyond those allowances, you pay per result or per month.

Prerequisites

  • Python 3.10 or newer, which curl_cffi has required since v0.14
  • pip install curl_cffi beautifulsoup4 lxml
  • For Method 2: pip install playwright, then playwright install chromium
  • A German IP for idealo.de, and one per country for Method 6

Every method shares one session, one fetch helper, and one price parser.

Stock requests sends Python's TLS handshake. curl_cffi sends Chrome's, which takes the cheapest signal off Akamai's table.

import json, random, re, time
from urllib.parse import urljoin, urlparse, parse_qs, quote_plus
from bs4 import BeautifulSoup
from curl_cffi import requests as cffi

BASE = "https://www.idealo.de"
HEADERS = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "de-DE,de;q=0.9,en;q=0.5",
}
session = cffi.Session(impersonate="chrome")  # Chrome TLS + HTTP/2 fingerprint

class Blocked(Exception):
    pass

The fetch helper turns Akamai's answers into one exception, so every method can retry the same way:

def fetch(url: str, headers: dict | None = None, **kw):
    resp = session.get(url, headers={**HEADERS, **(headers or {})}, timeout=30, **kw)
    if resp.status_code in (403, 429):
        raise Blocked(f"{resp.status_code} at {url}")  # Akamai's edge refuses outright
    if "Access Denied" in resp.text[:3000]:
        raise Blocked(f"block page at {url}")  # Akamai's standard denial page
    resp.raise_for_status()
    return resp

def soup_of(url: str) -> BeautifulSoup:
    return BeautifulSoup(fetch(url).text, "lxml")

Pass params= straight through **kw. Methods 3 and 5 rely on that.

Idealo prints German numbers, so 1.082,00 € means one thousand and eighty-two euros. The UK site flips the separators:

NUM = re.compile(r"\d[\d.,\s]*")

def parse_price(text: str | None, decimal: str = ",") -> float | None:
    if not text:
        return None
    m = NUM.search(text.replace("\xa0", " "))
    if not m:
        return None
    thousands = "." if decimal == "," else ","
    raw = m.group(0).strip().replace(" ", "").replace(thousands, "")
    try:
        return float(raw.replace(decimal, "."))
    except ValueError:
        return None

The same function reads offer counts, since 580 Angebote parses to 580.0. Pass decimal="." for idealo.co.uk and for prices inside URLs.

Method 1: Pull price, specs, and the first 20 offers from a product page

A product page at /preisvergleich/OffersOfProduct/{id}.html holds a JSON-LD Product block with the price range, plus one li.productOffers-listItem row for each of the first 20 offers.

You don't need the slug after the ID. Crawlbase's idealo recipe and the s-r actor's sample output both use the bare form, OffersOfProduct/203251234.html.

The JSON-LD sits in script[data-testid="Product"] according to Scrapfly. I match on the @type instead, which survives a renamed attribute:

def product_jsonld(soup: BeautifulSoup) -> dict:
    for tag in soup.select('script[type="application/ld+json"]'):
        try:
            data = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue
        nodes = data if isinstance(data, list) else data.get("@graph", [data])
        for node in nodes:
            if isinstance(node, dict) and node.get("@type") == "Product":
                return node
    return {}

Its offers key is a schema.org AggregateOffer: lowPrice, highPrice, offerCount, and priceCurrency across every shop, not only the 20 on screen.

Each offer row keeps the shop name and price in a JSON attribute, data-mtrx-click.

Its click-out link is an /relocator/relocate URL whose query string holds the offer key, shop ID, and price. The parser reads both:

def parse_offer(li) -> dict:
    try:
        mtrx = json.loads(li.get("data-mtrx-click") or "{}")
    except json.JSONDecodeError:
        mtrx = {}
    link = li.select_one("a.productOffers-listItemOfferCtaLeadout, a.productOffers-listItemTitle")
    lead = leadout_params(link.get("href", "")) if link else {}
    price = (mtrx.get("products") or [{}])[0].get("price")
    delivery = li.select_one(".productOffers-listItemOfferDeliveryStatusDatesRange")
    shipping = li.find(string=re.compile("Versand"))  # best effort: the shipping line's text
    return {
        "shop": mtrx.get("shop_name"), "shop_id": lead.get("sid"),
        "price": price if price is not None else parse_price(lead.get("price"), "."),
        "shipping_text": shipping.strip() if shipping else None,
        "delivery": delivery.get_text(" ", strip=True) if delivery else None,
        "offer_key": lead.get("offerKey"),
    }

You parse the relocator link; you never request it. The edge section below explains why.

shop_id is worth keeping. In Scrapfly's sample, shop ID 302353 also appears in that shop's profile URL, /preisvergleich/Shop/302353.html.

Three helpers finish the method. leadout_params reads the relocator query string, and product_id_of reads the ID from any country's product URL, which Method 6 needs:

def leadout_params(href: str) -> dict:
    q = parse_qs(urlparse(href).query)
    return {k: q[k][0] for k in ("offerKey", "sid", "price", "pos") if k in q}

def parse_offers(soup: BeautifulSoup) -> list[dict]:
    rows = [parse_offer(li) for li in soup.select("li.productOffers-listItem")]
    return [r for r in rows if r["shop"] or r["offer_key"]]

def product_id_of(url: str) -> str | None:
    m = re.search(r"/(\d{5,})(?=[_./])", urlparse(url).path)  # 207644441_..., /207644441/, 203251234.html
    return m.group(1) if m else None

The scraper itself joins the JSON-LD summary to the offer rows:

def scrape_product(url: str) -> dict:
    soup = soup_of(url)
    ld = product_jsonld(soup)
    agg = ld.get("offers") or {}
    brand = ld.get("brand")
    return {
        "product_id": product_id_of(url), "name": ld.get("name"),
        "brand": brand.get("name") if isinstance(brand, dict) else brand,
        "gtin": ld.get("gtin13") or ld.get("gtin"),
        "low_price": agg.get("lowPrice"), "high_price": agg.get("highPrice"),
        "offer_count": agg.get("offerCount"), "currency": agg.get("priceCurrency"),
        "rating": (ld.get("aggregateRating") or {}).get("ratingValue"),
        "offers": parse_offers(soup),
    }

row = scrape_product(f"{BASE}/preisvergleich/OffersOfProduct/207644441.html")
print(row["name"], row["low_price"], row["offer_count"], len(row["offers"]))

Expect offer_count to dwarf len(row["offers"]) on popular products. Scrapfly's iPhone 17 Pro sample reported 395 offers in its aggregate.

gtin may come back None. Two of the actors I read return a GTIN from product pages, while Scrapfly says the page doesn't print one.

Check your own sample before you build Method 5 on it.

Method 2: Load every shop offer past the first 20

The product page renders about 20 offers, and its load-more button fetches the rest in blocks of 20, which you capture once in a browser and replay over HTTP.

The block size comes from two actor docs: ScrapeUnblocker's says each further block of 20 is one more request, and cliqtomedia's bills per extra 20.

Neither documents the request itself, and I couldn't load idealo from my sandbox to read it. So step one is to let the page show you.

Playwright clicks button.productOffers-listLoadMore until it disappears and logs every XHR and fetch call on the way. The Playwright network docs cover the page.on("request") hook:

from playwright.sync_api import sync_playwright

def offers_in_browser(url: str, max_clicks: int = 40) -> tuple[list[dict], list[str]]:
    calls = []
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)  # headless Chromium is easier to fingerprint
        page = browser.new_page(locale="de-DE")
        page.on("request", lambda r: calls.append(r.url) if r.resource_type in ("xhr", "fetch") else None)
        page.goto(url, wait_until="domcontentloaded")
        for _ in range(max_clicks):
            more = page.locator("button.productOffers-listLoadMore")
            if more.count() == 0 or not more.first.is_visible():
                break
            more.first.click()
            page.wait_for_timeout(800)  # give the next block time to render
        html = page.content()
        browser.close()
    return parse_offers(BeautifulSoup(html, "lxml")), calls

The function is a working scraper on its own. The rendered DOM holds every offer once the button is gone, and parse_offers reads it unchanged.

Filter the log for calls that carry the product ID:

url = f"{BASE}/preisvergleich/OffersOfProduct/207644441.html"
offers, calls = offers_in_browser(url)
print(len(offers), "offers in the DOM")
for call in calls:
    if "207644441" in call and "price-chart" not in call:
        print(call)  # one line per click; find the number that grows by 20

Turn that URL into a template with {id} and {start} placeholders. Then page through it with the curl_cffi session, sending the Referer and any headers the browser sent:

def offers_by_template(template: str, product_id: str, total: int,
                       referer: str, extra: dict | None = None, step: int = 20) -> list[dict]:
    rows, seen = [], set()
    for start in range(step, total, step):
        resp = fetch(template.format(id=product_id, start=start),
                     headers={"Referer": referer, **(extra or {})})
        block = [r for r in parse_offers(BeautifulSoup(resp.text, "lxml"))
                 if (r["offer_key"] or (r["shop"], r["price"])) not in seen]
        if not block:
            break  # empty or repeated block: past the end
        seen.update(r["offer_key"] or (r["shop"], r["price"]) for r in block)
        rows.extend(block)
        time.sleep(random.uniform(2, 4))
    return rows

Pass total from Method 1's offer_count. If the endpoint answers with JSON that wraps HTML, pull out that field and feed it to parse_offers.

The ScrapeUnblocker docs also note the list holds new-item offers only.

Used and B-stock offers sit elsewhere on the page, so a product's real floor price can be lower than your minimum.

For more on capturing and replaying calls like this one, see our Playwright network interception guide.

Method 3: Get up to a year of daily prices from the price-chart endpoint

The chart on each product page draws from /price-chart/sites/{siteId}/products/{productId}/history?period={period}, which returns daily lowest prices in cents plus average, low, and high statistics.

Scrapfly documented the endpoint in October 2026. Site ID 1 is idealo.de, and period takes 1M, 3M, 6M, or 1Y.

The call wants the product page's cookies and a Referer pointing back at it, so the function loads the page first in the same session:

def to_eur(cents) -> float | None:
    try:
        return round(float(cents) / 100, 2)
    except (TypeError, ValueError):
        return None

def price_history(product_id: str, period: str = "3M", site_id: int = 1) -> dict:
    page = f"{BASE}/preisvergleich/OffersOfProduct/{product_id}.html"
    fetch(page)  # sets the cookies the chart widget's call carries
    resp = fetch(f"{BASE}/price-chart/sites/{site_id}/products/{product_id}/history",
                 headers={"Accept": "application/json", "Referer": page},
                 params={"period": period})  # 1M, 3M, 6M or 1Y
    data = resp.json()
    stats = data.get("statistics") or {}
    return {
        "product_id": product_id, "period": period, "start": data.get("startDate"),
        "avg": to_eur(stats.get("avgPrice")), "low": to_eur(stats.get("lowestPrice")),
        "high": to_eur(stats.get("highestPrice")),
        "points": [(pt.get("x"), to_eur(pt.get("y"))) for pt in data.get("data") or []],
    }

x is a date string and y is cents. Scrapfly's three-month pull for an iPhone 17 came back with 85 daily points.

The chart draws one line per product: the cheapest offer each day, without the shop that held it or what the others charged.

If you need per-shop history, snapshot Method 1 or 2 on a schedule. Method 7 keeps that cheap.

I only have a site ID for idealo.de. On the other country sites, open the chart with DevTools open and read the number from the request.

Method 4: Scrape Idealo categories past the 4,000-product cap

A category page at /preisvergleich/ProductCategory/{categoryId}.html lists product cards you can page through, and each card carries the product ID, name, lowest price, and offer count.

Category 19116 is mobile phones on idealo.de, per the piotrv1001 actor's docs. The same docs put idealo's pagination ceiling at about 4,000 products: 112 pages of 36.

Search and category pages share one card layout. The class names below come from Scrapfly's parser, matched on substrings because idealo appends hashes:

def parse_card(card, decimal: str = ",") -> dict:
    link = card.select_one('a[href*="OffersOfProduct"]')
    title = card.select_one('[class*="sr-productSummary__title"]')
    price = card.select_one('[class*="sr-detailedPriceInfo__price"]')
    count = card.select_one('[class*="sr-detailedPriceInfo__offerCount"]')
    href = urljoin(BASE, link["href"]) if link else None
    n = parse_price(count.get_text(" ", strip=True)) if count else None
    return {
        "product_id": product_id_of(href) if href else None,
        "name": title.get_text(" ", strip=True) if title else None,
        "price": parse_price(price.get_text(" ", strip=True), decimal) if price else None,
        "offer_count": int(n) if n is not None else None,
        "url": href,  # None means a single-shop offer with no comparison page
    }

Later pages add an offset to the path, as in ProductCategory/3513I16-705.html from the ecomscrape actor's sample.

The offset step isn't obvious, so I read the next URL from the pagination links instead of computing it:

OFFSET = re.compile(r"I16-(\d+)\.html")

def next_page(soup: BeautifulSoup, current: int) -> str | None:
    found = {}
    for a in soup.select('[class*="sr-pagination"] a[href]'):
        m = OFFSET.search(a["href"])
        if m:
            found[int(m.group(1))] = urljoin(BASE, a["href"])
    later = [o for o in found if o > current]
    return found[min(later)] if later else None

The crawler stops on three signals: no next link, the page limit, or a page without new IDs.

That last check matters. The axlymxp actor's docs warn that idealo quietly serves page 1 again when you ask for a page past the end.

A loop that only checks for cards never stops:

def crawl_listing(start_url: str, seen: set, max_pages: int = 120) -> list[dict]:
    rows, url, offset = [], start_url, 0
    for _ in range(max_pages):
        soup = soup_of(url)
        cards = [parse_card(c) for c in soup.select('[class*="sr-resultList__item"]')]
        fresh = [c for c in cards if c["product_id"] and c["product_id"] not in seen]
        if not fresh:
            break  # page 1 re-served, or only single-shop offers left
        seen.update(c["product_id"] for c in fresh)
        rows.extend(fresh)
        url = next_page(soup, offset)
        if not url:
            break
        offset = int(OFFSET.search(url).group(1))
        time.sleep(random.uniform(3, 6))
    return rows

To get past 4,000, apply a brand or price filter in your browser, copy each filtered URL, and crawl them all with one shared seen set:

FILTERED_URLS = [
    # one category URL per brand or price filter, copied from the address bar
    "https://www.idealo.de/preisvergleich/ProductCategory/19116.html",
]
seen: set = set()
phones = []
for filtered in FILTERED_URLS:
    phones += crawl_listing(filtered, seen)
print(len(phones), "products")

The shared set drops duplicates across filters. If one filter still fills all 112 pages, split it again.

Method 5: Match your EANs to Idealo product IDs

To scrape idealo prices for your own catalogue, you first need each SKU's idealo product ID. The product page, its offer list and its price history are all keyed to that ID. The dependable way to get it is to search idealo for the EAN, fall back to the product title, confirm the GTIN on each candidate's product page, and cache the matched ID against your SKU. That way each item is searched only once.

Why barcode search alone is unreliable

Idealo's search does not resolve barcodes consistently. Scrapfly reported that 4006381333931 found its STABILO fineliner, while 5449000000996 returned only unrelated shop offers. Treat any barcode hit as a candidate to verify, not as a match.

Search is also the most sensitive part of the site. The scrapyx actor's docs say idealo's robots.txt disallows MainSearchProductCategory.html. I couldn't fetch robots.txt to confirm this, so check it yourself before you run searches at volume. The cache below keeps search traffic low, because each SKU costs one lookup for its lifetime and every later price scrape goes straight to the product page.

Step 1: Search and drop single-shop offers

The search helper reuses Method 4's card parser. It keeps only results that link to a comparison page:

def search_products(query: str) -> list[dict]:
    soup = soup_of(f"{BASE}/preisvergleich/MainSearchProductCategory.html?q={quote_plus(query)}")
    cards = [parse_card(c) for c in soup.select('[class*="sr-resultList__item"]')]
    return [c for c in cards if c["url"]]  # drop single-shop offers

This filter matters more than it looks. In Scrapfly's sample, a search for eclipse glasses returned 155 results, and only 6 were products with a comparison page. The rest were listings from individual shops, with no product page to verify a GTIN against and no offer list to scrape. If you leave them in, the resolver spends its candidate slots on dead ends.

Step 2: Verify the GTIN and cache the match

The resolver checks the SQLite cache first. On a miss it searches by EAN, then by title, and opens the top three candidates for each query. It accepts a candidate only when the GTIN on its product page equals your EAN. A random 2–4 second pause between candidate fetches keeps the request rate modest.

import sqlite3

ids = sqlite3.connect("idealo_ids.db")
ids.execute("CREATE TABLE IF NOT EXISTS ids (ean TEXT PRIMARY KEY, product_id TEXT)")

def resolve(ean: str, title: str) -> str | None:
    hit = ids.execute("SELECT product_id FROM ids WHERE ean = ?", (ean,)).fetchone()
    if hit:
        return hit[0]
    for query in (ean, title):  # barcode first, title as the fallback
        for cand in search_products(query)[:3]:
            if scrape_product(cand["url"])["gtin"] == ean:
                ids.execute("INSERT OR REPLACE INTO ids VALUES (?, ?)", (ean, cand["product_id"]))
                ids.commit()
                return cand["product_id"]
            time.sleep(random.uniform(2, 4))
    return None  # log it and match by hand

Each match is committed as soon as it is found, so a crash halfway through a large catalogue keeps the work already done. The cap of three candidates bounds the cost of a miss to two searches and at most six product-page fetches per SKU. Raise it if your titles are generic and the right product often sits lower in the results.

The GTIN check is the step that makes the title fallback safe. A title search will usually return something that looks right. Without the check, you would silently record prices for a neighbouring product.

When resolve() returns None

A None usually means one of three things:

  • A variant mismatch. Variants trip people up most. A phone in four colours can be four products, each with its own GTIN, so a title match often lands on the wrong one. Adding the colour or storage size to the title you pass in narrows the results.
  • A missing GTIN in the JSON-LD. The product exists on idealo, but its page doesn't expose a GTIN, so the check can never pass. Confirm the product by eye and record the ID yourself.
  • A product idealo doesn't list. There is no comparison page to find, and no price data to scrape for that SKU.

Log every miss with its EAN and title and work through the list by hand. Once you insert a manual match into the ids table, resolve() returns it from the cache like any automatic match, and that SKU never hits search again.

Method 6: Scrape Idealo's other five country sites

The same product ID works on all six Idealo sites, and the idealo.de product page links to each one, so one German page gives you every country URL.

The axlymxp actor's docs describe the ID as stable across all six sites. The paths differ by country, which is why hard-coding them is a trap:

Site Product path Currency
idealo.de, idealo.at /preisvergleich/OffersOfProduct/{id}.html EUR
idealo.fr /prix/... EUR
idealo.it /confronta-prezzi/... EUR
idealo.es /precios/{id}.html EUR
idealo.co.uk /compare/{id}/{slug}.html GBP

Scrapfly's guide says each idealo.de product page lists the product's prices on the other five sites.

The links in that block carry the product ID, so the extractor keeps only links that contain it:

INTL = re.compile(r"https://www\.idealo\.(at|fr|it|es|co\.uk)/[^\"'\s<>]+")

def country_links(de_url: str) -> dict[str, str]:
    pid = product_id_of(de_url)
    links = {}
    for m in INTL.finditer(fetch(de_url).text):
        if pid and pid in m.group(0):  # skips footer links to the other homepages
            links.setdefault(m.group(1), m.group(0))
    return links

Each market gets its own session from an IP in that country, plus a matching Accept-Language. Listings resolve to the visitor's location, per Scrapfly:

LANG = {"at": "de-AT,de;q=0.9", "fr": "fr-FR,fr;q=0.9", "it": "it-IT,it;q=0.9",
        "es": "es-ES,es;q=0.9", "co.uk": "en-GB,en;q=0.9"}
PROXIES = {}  # e.g. {"fr": "http://user:[email protected]:8000"}, one per market

de_url = f"{BASE}/preisvergleich/OffersOfProduct/207644441.html"
prices = {"de": scrape_product(de_url)["low_price"]}
for market, url in country_links(de_url).items():
    proxy = PROXIES.get(market)
    session = cffi.Session(impersonate="chrome", proxies={"https": proxy} if proxy else None)
    HEADERS["Accept-Language"] = LANG[market]
    prices[market] = scrape_product(url)["low_price"]
    time.sleep(random.uniform(3, 6))
print(prices)  # the co.uk value is in GBP

Run this at module level, as shown, so fetch picks up the new session. Inside a function you'd need a global session line.

JSON-LD and data-mtrx-click prices are plain numbers on every site. Only text you parse yourself, like UK card prices, needs decimal=".".

Method 7: Monitor thousands of products by diffing category cards

Crawl the category cards every day, compare each product's lowest price and offer count with yesterday's, and fetch full product pages only for the products whose numbers moved.

The arithmetic is the reason. A 4,000-product category is about 112 card pages, against 4,000 product pages plus their offer blocks for a full refresh.

The snapshot table stores one row per product, and the diff returns anything new or changed:

import sqlite3

snap = sqlite3.connect("idealo_snapshots.db")
snap.execute("""CREATE TABLE IF NOT EXISTS cards
    (product_id TEXT PRIMARY KEY, price REAL, offer_count INTEGER, seen_at TEXT)""")

def changed_since_last_run(cards: list[dict]) -> list[dict]:
    moved = []
    for c in cards:
        old = snap.execute("SELECT price, offer_count FROM cards WHERE product_id = ?",
                           (c["product_id"],)).fetchone()
        if old != (c["price"], c["offer_count"]):
            moved.append(c)  # new product, new low price, or a shop joined or left
        snap.execute("INSERT OR REPLACE INTO cards VALUES (?, ?, ?, datetime('now'))",
                     (c["product_id"], c["price"], c["offer_count"]))
    snap.commit()
    return moved

The daily job ties Methods 4 and 1 together:

cards = crawl_listing(f"{BASE}/preisvergleich/ProductCategory/19116.html", set())
moved = changed_since_last_run(cards)
print(f"{len(moved)} of {len(cards)} products moved")
with open("idealo_offers.jsonl", "a", encoding="utf-8") as fh:
    for c in moved:
        row = {**scrape_product(c["url"]), "checked_at": time.strftime("%Y-%m-%d")}
        fh.write(json.dumps(row, ensure_ascii=False) + "\n")
        time.sleep(random.uniform(3, 6))

JSON Lines keeps a crash from losing the rows already written. Our guide to storing scraped data compares it with CSV and SQLite for runs like this.

Use a card change as the trigger for a refetch, and the product page as the record.

The piotrv1001 docs note that a card's best price is sometimes lower than the product's own offer list, which they put down to cached or conditional prices.

The diff also misses a shop changing price above the minimum. If that matters, add a weekly full pass on your top products.

Idealo's blocks are loud, which makes them easy to catch: Akamai refuses at the edge with a 403 instead of serving a fake page with status 200.

Scrapfly saw server: AkamaiGHost on those refusals. In ReventaData's test on 16 October 2026, direct and datacenter requests got HTTP 403, while residential connections returned real pages.

Symptom What you see What fetch does Fix
Edge refusal 403, server: AkamaiGHost Raises Blocked Check impersonation, then switch to a German residential IP
Denial page "Access Denied" HTML Raises Blocked Same, plus a longer pause
Rate limit 429 Raises Blocked Back off, then halve your rate
Wrong market Valid page, odd prices or shops Nothing Match exit IP and Accept-Language

A refused session tends to stay refused, so the retry drops cookies and waits longer each time:

def fetch_with_backoff(url: str, tries: int = 4, **kw):
    for attempt in range(tries):
        try:
            return fetch(url, **kw)
        except Blocked:
            session.cookies.clear()  # Akamai's cookies mark the session; start clean
            time.sleep(15 * 2 ** attempt + random.uniform(0, 10))  # ~15s, 30s, 60s, 120s
    raise Blocked(f"gave up on {url}")

Swap it in for fetch once you run more than a few dozen pages. If every request from an IP fails, retire that IP for the day.

Idealo doesn't publish a rate limit. I'd start at one page every 3 to 6 seconds per IP, with one session per IP, and slow down at the first 403.

Past a few hundred pages a day, spread sessions across residential IPs in Germany. curl_cffi takes proxies={"https": "http://user:pass@host:port"} on the session.

For what Akamai's sensor collects when you reach the browser stage, see our Akamai write-up.

Every "go to shop" button points at /relocator/relocate, and following it sends a click-out to the merchant.

Idealo is funded by those click-outs; Wikipedia describes its revenue as advertiser-paid traffic. A scraper that follows them bills real shops for visits nobody made and skews their stats.

You lose nothing by skipping them. The offer key, shop ID, price, and position are already in the URL's query string, which leadout_params reads without a request.

What these methods can't get you

The final checkout price is out of reach. Vouchers, payment surcharges, and member prices live on the shop's site, behind that relocator.

Shipping is partial. Sources disagree on the default sort: PricePirate says idealo ranks by item price alone, and the memo23 docs say cheapest total first.

Sort on your own computed total.

Stock levels are absent. The closest field is a delivery window such as "bis Do. 20.08.", which says nothing about units.

Troubleshooting

"Blocked: 403 at https://www.idealo.de/preisvergleich/..."

Why: Akamai refused the TLS handshake, the IP, or both. Datacenter IPs get this on the first request.

Fix: Confirm impersonate="chrome" is set, move to a German residential IP, and use fetch_with_backoff.

"json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)"

Why: price_history got HTML back, usually a denial page, or the call went out without the product page's cookies and Referer.

Fix: Print resp.text[:300]. Load the product page in the same session first, as the function does, and keep the Referer.

scrape_product returns a name and price but offers is empty

Why: The offer-row classes changed, or the page served a shell that fills the list with JavaScript.

Fix: Save fetch(url).text and count productOffers-listItem in it. If it's zero but the browser shows offers, use offers_in_browser from Method 2.

The category crawl keeps returning the same products

Why: You asked for a page past the end, and idealo served page 1 again.

Fix: Keep the shared seen set from Method 4. A page with no new IDs ends the crawl.

FAQ

Reading public Idealo pages is lawful in many places, but idealo's terms and the EU database right can limit copying large parts of its catalog.

Keep to public pages, skip personal data such as reviewer names, and get legal advice before commercial reuse. This isn't legal advice.

Does Idealo have an API for price data?

Not for reading it. Idealo's PWS API is for merchants pushing their own offers into the comparison, so anyone who needs other shops' prices reads the public pages instead.

Do you need proxies to scrape Idealo?

For more than a handful of pages, yes. Akamai refuses datacenter IPs, and idealo.de resolves listings by location.

Use German residential IPs with one session per IP. Roundproxies sells residential pools, and any provider with German sticky sessions works.

Why does my Idealo scraper only get 20 offers?

The product page renders about 20 offers and loads the rest in blocks of 20 when you click the button under the list.

Capture that request once in a browser (Method 2), then replay it with an increasing offset.

Do Idealo prices include shipping?

Usually not. The headline price is the item price, and each offer row shows shipping separately, so compute your own total before you rank shops.

Can you scrape Idealo without a headless browser?

Yes, for nearly everything. Product pages, category pages, and the price-chart JSON all answer curl_cffi from a German residential IP.

The one browser step is capturing the load-more request in Method 2, and you only do that once.

Wrapping up

Start with Method 1 on one product and print the raw JSON-LD before you trust a field.

Add Method 4 to find product IDs and Method 2 the first time a product's offer_count passes 20.

Once you track more than a few hundred products, Method 7 keeps the daily run small.

The parts that rot are the offer-row classes, the load-more request, and the price-chart path.

When a run comes back empty, check those first, and watch the curl_cffi repository for new Chrome targets.