> ## Content Index
> Fetch the complete content index at: https://roundproxies.com/blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# 5 ways to scrape Threads.com in 2026
- URL: https://roundproxies.com/blog/scrape-threads/
- Published: 2026-11-01T21:47:13.000Z
- Updated: 2026-11-01T22:42:53.000Z
- Description: Scrape Threads posts from embedded JSON, page deeper via GraphQL, and search with Meta's official API.
- Author: Emilie Beher
- Tags: Web Scraping, #dated-6ae7aff556ce24000126912b

You can scrape Threads.com posts, replies, and profiles with an ordinary HTTP client. Each public page arrives with its data already embedded as JSON, before any JavaScript runs.

Most of page one pushes you toward a headless browser or a paid scraper. The five methods below run on [curl\_cffi](https://roundproxies.com/blog/curl%5Fcffi/) and plain Python, with Playwright kept as a fallback.

Threads moved from threads.net to threads.com in April 2025\. Old links still redirect, so the code accepts either domain.

## How do you scrape Threads.com posts?

To scrape Threads.com posts, request a public post or profile URL with a Chrome TLS fingerprint and browser headers, then parse the `<script data-sjs>` JSON blobs that contain `thread_items`. For older posts, replay the page's own GraphQL request using its `lsd` token. For keyword search, use Meta's official Threads API.

| Method                      | What you get                       | Needs                            | Breaks when                    |
| --------------------------- | ---------------------------------- | -------------------------------- | ------------------------------ |
| 1\. Post page JSON          | Post, replies, likes, reply counts | Nothing                          | Meta renames JSON keys         |
| 2\. Profile page JSON       | Bio, followers, recent posts       | Nothing                          | Meta renames JSON keys         |
| 3\. GraphQL paging          | Older posts past the first batch   | A doc\_id from DevTools          | The doc\_id rotates            |
| 4\. Official keyword search | Public posts matching a query      | Meta app and approved permission | You exceed 2,200 queries a day |
| 5\. Media download          | Images and videos                  | Nothing                          | The signed URL expires         |

## Prerequisites

- Python 3.10 or newer (curl\_cffi's current minimum)
- `pip install curl_cffi parsel requests`
- For Method 4: a Meta developer app with the Threads use case and a user access token
- For the fallback: `pip install playwright`, then `playwright install chromium`

All five methods share one session. curl\_cffi reproduces Chrome's TLS and HTTP/2 handshake, which stock `requests` can't ([more on what a TLS fingerprint gives away](https://roundproxies.com/blog/what-is-tls-fingerprint/)).

```python
from curl_cffi import requests as cffi

# Without Sec-Fetch-* headers, Threads returns a 200 with an empty app shell
PAGE_HEADERS = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
}

session = cffi.Session(impersonate="chrome")  # Chrome TLS + HTTP/2 fingerprint

def fetch(url: str) -> str:
    resp = session.get(url, headers=PAGE_HEADERS, timeout=20)
    resp.raise_for_status()
    return resp.text

```

The header set is the part people skip. One open-source Threads scraper documents the empty-shell response in its changelog, and it's the first thing I'd check when a scraper returns nothing.

## Method 1: Extract a post and its replies from the page JSON

A Threads post page stores the post and its loaded replies in `<script type="application/json" data-sjs>` tags, under a key named `thread_items`.

You can confirm it in a minute. Open a post, press F12, and search the Elements panel for the post's first few words; the match sits inside one of those script tags.

Two helpers do the heavy lifting. One pulls the right script blobs, and the other finds a key at any depth.

```python
import json
from parsel import Selector

def hidden_datasets(html: str, must_contain: str) -> list:
    """Return every data-sjs JSON blob whose raw text mentions `must_contain`."""
    blobs = Selector(text=html).css('script[type="application/json"][data-sjs]::text').getall()
    # cheap substring test first; most blobs are config you don't need
    return [json.loads(raw) for raw in blobs if must_contain in raw]

def find_key(obj, key: str):
    """Yield every value stored under `key`, however deep it sits."""
    if isinstance(obj, dict):
        for k, v in obj.items():
            if k == key:
                yield v
            yield from find_key(v, key)
    elif isinstance(obj, list):
        for item in obj:
            yield from find_key(item, key)

```

Searching by key instead of by fixed path survives Meta reshuffling the wrapper objects around the data.

Each item in `thread_items` wraps a `post` object. The field names below are the ones community parsers use, so print one raw item and compare before you trust a column.

```python
def parse_post(item: dict) -> dict:
    post = item.get("post", item)
    user = post.get("user") or {}
    info = post.get("text_post_app_info") or {}
    return {
        "code": post.get("code"),  # the shortcode in /post/<code>
        "username": user.get("username"),
        "text": (post.get("caption") or {}).get("text"),
        "published": post.get("taken_at"),  # Unix seconds
        "likes": post.get("like_count"),
        "replies": info.get("direct_reply_count"),
        "reposts": info.get("repost_count"),
        "quotes": info.get("quote_count"),
        "url": f"https://www.threads.com/@{user.get('username')}/post/{post.get('code')}",
    }

```

Reply counts live under `text_post_app_info`. If your payload keeps repost or quote counts somewhere else, those two lines are the ones to change.

The scraper ties the three functions together:

```python
def scrape_post(url: str) -> dict:
    html = fetch(url)
    posts = {}
    for blob in hidden_datasets(html, "thread_items"):
        for thread_items in find_key(blob, "thread_items"):
            for item in thread_items:
                parsed = parse_post(item)
                posts.setdefault(parsed["code"], parsed)  # de-dupe, keep page order
    if not posts:
        raise ValueError("No thread_items in page")
    ordered = list(posts.values())
    return {"post": ordered[0], "replies": ordered[1:]}

print(scrape_post("https://www.threads.com/@meta/post/DDzbnVKx57R"))

```

The first entry is the post you asked for. The rest are whatever replies the page rendered, which is usually a partial set; Method 3 fetches the remainder.

## Method 2: Scrape a Threads profile and its recent posts

A profile page at `threads.com/@username` embeds the user object, with bio, `follower_count`, and verification flags, next to the first batch of that user's posts.

```python
def parse_profile(user: dict) -> dict:
    return {
        "username": user.get("username"),
        "full_name": user.get("full_name"),
        "bio": user.get("biography"),
        "followers": user.get("follower_count"),
        "verified": user.get("is_verified"),
        "private": user.get("text_post_app_is_private"),
        "links": [link.get("url") for link in user.get("bio_links") or []],
    }

```

The scraper reuses both helpers from Method 1:

```python
def scrape_profile(username: str) -> dict:
    html = fetch(f"https://www.threads.com/@{username}")
    profile, posts = {}, []
    for blob in hidden_datasets(html, "ScheduledServerJS"):
        for user in find_key(blob, "user"):
            # quotes and reposts embed other users too, so match the handle
            if isinstance(user, dict) and user.get("username") == username and "follower_count" in user:
                profile = profile or parse_profile(user)
        for thread_items in find_key(blob, "thread_items"):
            posts.extend(parse_post(item) for item in thread_items)
    return {"profile": profile, "posts": posts}

```

Skip the username check and you'll occasionally store a quoted account's follower count under the wrong profile. Logged out, the page carries the latest handful of posts, not the archive.

## Method 3: Page past the first batch with Threads' GraphQL endpoint

Threads loads older posts by POSTing to `/api/graphql` with a `doc_id` that names the query, a `variables` JSON object, and an `lsd` token printed into the page.

Don't copy a `doc_id` from a blog post, this one included. Meta rotates them, and every scraper holding a stale ID fails on the same day.

Grab a current one from your own browser. Open a profile, filter the Network tab by `graphql`, and scroll down.

Pick the request whose response contains `thread_items`, then copy `doc_id` and `variables` from its payload.

The `lsd` token comes from the profile HTML you already fetched:

```python
import re

def page_lsd(html: str) -> str:
    # Meta pages print the token as ["LSD",[],{"token":"..."}
    m = re.search(r'"LSD",\[\],\{"token":"([^"]+)"', html)
    if not m:
        raise ValueError("LSD token not found; the page was probably an empty shell")
    return m.group(1)

```

Then replay the request with that token and the same-origin headers a browser sends:

```python
def graphql(doc_id: str, variables: dict, lsd: str, referer: str) -> dict:
    resp = session.post(
        "https://www.threads.com/api/graphql",
        data={"lsd": lsd, "doc_id": doc_id, "variables": json.dumps(variables)},
        headers={
            "X-FB-LSD": lsd,
            "X-IG-App-ID": "238260118697367",  # web app ID from public reverse engineering; confirm in DevTools
            "Origin": "https://www.threads.com",
            "Referer": referer,
            "Sec-Fetch-Site": "same-origin",
            "Sec-Fetch-Mode": "cors",
            "Sec-Fetch-Dest": "empty",
        },
        timeout=20,
    )
    resp.raise_for_status()
    return resp.json()

```

Use `/api/graphql`. The older `/graphql/query` path now answers logged-out clients with a 403.

Paging is a loop over the cursor:

```python
import random
import time

def paginate(doc_id: str, variables: dict, lsd: str, referer: str, max_pages: int = 5):
    for _ in range(max_pages):
        data = graphql(doc_id, variables, lsd, referer)
        for thread_items in find_key(data, "thread_items"):
            yield from (parse_post(item) for item in thread_items)
        info = next(find_key(data, "page_info"), None) or {}
        if not info.get("has_next_page"):
            break
        # Meta's GraphQL cursors usually go in "after"; match your captured variables
        variables = {**variables, "after": info.get("end_cursor")}
        time.sleep(random.uniform(2, 5))

```

If the endpoint returns HTML or an `errors` object, refetch the page for a fresh `lsd` before you blame the `doc_id`.

A stale token is the cheaper problem to fix, so rule it out first.

## Method 4: Search public Threads posts with the official API

Meta's `keyword_search` endpoint returns public posts matching a keyword or topic tag, sorted by `TOP` or `RECENT`, at up to 100 posts per call.

It's the only sanctioned search on Threads, and scraping can't match its reach. Logged out, threads.com search shows roughly one page of about 20 results, according to one scraper author's testing.

Setup happens in the Meta developer dashboard. Create an app with the Threads use case, add the `threads_basic` and `threads_keyword_search` permissions, and generate a token in Graph API Explorer.

Until App Review approves `threads_keyword_search`, results only include your own posts ([Meta's keyword search docs](https://developers.facebook.com/documentation/threads/keyword-search) spell out the rules).

```python
import os
import requests

def keyword_search(query: str, search_type: str = "RECENT", since: int | None = None) -> list:
    params = {
        "q": query,
        "search_type": search_type,  # TOP (default) or RECENT
        "fields": "id,text,media_type,permalink,timestamp,username,has_replies",
        "limit": 100,  # documented maximum
        "access_token": os.environ["THREADS_ACCESS_TOKEN"],
    }
    if since:
        params["since"] = since  # Unix time, no earlier than 1688540400
    resp = requests.get("https://graph.threads.com/v1.0/keyword_search", params=params, timeout=20)
    resp.raise_for_status()
    return resp.json().get("data", [])

```

The quota is per user, not per app: 2,200 queries per rolling 24 hours, and repeating a keyword spends another query.

Empty results don't count against it. Meta also returns an empty array for keywords it deems sensitive.

The `1688540400` floor is 5 July 2023, the week Threads launched.

## Method 5: Download images and videos before the links expire

Media URLs sit in the same post objects you scraped in Methods 1 to 3: `image_versions2.candidates` for photos, `video_versions` for clips, and `carousel_media` for multi-image posts.

Pass this function the raw `post` dict, not the output of `parse_post`, which drops the media fields.

```python
def media_urls(post: dict) -> list:
    urls = [v["url"] for v in post.get("video_versions") or []][:1]  # first video rendition
    candidates = (post.get("image_versions2") or {}).get("candidates") or []
    if candidates:
        urls.append(max(candidates, key=lambda c: c.get("width", 0))["url"])  # widest image
    for child in post.get("carousel_media") or []:
        urls.extend(media_urls(child))
    return list(dict.fromkeys(urls))  # de-dupe, keep order

```

Downloading goes through the same session:

```python
from datetime import datetime, timezone
from pathlib import Path
from urllib.parse import parse_qs, urlparse

def expires_at(url: str):
    oe = parse_qs(urlparse(url).query).get("oe", [None])[0]  # hex Unix timestamp
    return datetime.fromtimestamp(int(oe, 16), tz=timezone.utc) if oe else None

def download(url: str, folder: str = "threads_media") -> Path:
    Path(folder).mkdir(exist_ok=True)
    path = Path(folder, Path(urlparse(url).path).name)
    resp = session.get(url, timeout=60)
    resp.raise_for_status()
    path.write_bytes(resp.content)
    return path

```

Download in the same run that scraped the post. The URLs are signed, and once the `oe` timestamp passes, the CDN refuses them; log `expires_at()` if you queue downloads for later.

## Scrape Threads without a browser: what the server checks

Threads embeds the JSON or withholds it based on what your request looks like.

JavaScript only turns that JSON into visible posts, so a client that never runs JS can still read everything.

| What Threads looks at  | Bare requests call        | What works                             |
| ---------------------- | ------------------------- | -------------------------------------- |
| Fetch metadata headers | Missing                   | Sec-Fetch-\* set for a page navigation |
| GraphQL path           | /graphql/query gets a 403 | /api/graphql with lsd and doc\_id      |
| GraphQL token          | Hard-coded or absent      | lsd read from the same page            |
| Requests per IP        | Unbounded                 | Delays plus IP rotation                |

I'd use curl\_cffi even though the documented failures above are about headers. It takes the TLS handshake off the list of suspects while you debug the rest.

At volume, one IP will start drawing 429s. Spread sessions across [residential IPs](https://claude.ai/blog/how-residential-proxies-work/) and keep each IP for a few related requests.

Add jitter between calls, too. Roundproxies sells residential pools, and any provider with sticky sessions does the job.

### Fallback: capture GraphQL with Playwright when the JSON is missing

When a URL keeps coming back empty, let a real browser load it and record the GraphQL responses it receives.

```python
from contextlib import suppress
from playwright.sync_api import sync_playwright

def capture_graphql(url: str, scrolls: int = 5) -> list:
    payloads = []
    def on_response(resp):
        if "/api/graphql" in resp.url and resp.ok:
            with suppress(Exception):  # some responses aren't JSON
                payloads.append(resp.json())

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.on("response", on_response)  # record every GraphQL reply
        page.goto(url, wait_until="domcontentloaded")
        for _ in range(scrolls):
            page.mouse.wheel(0, 4000)  # each scroll triggers the next batch of posts
            page.wait_for_timeout(1500)
        browser.close()
    return payloads

```

Feed `payloads` into `find_key(..., "thread_items")` and `parse_post` unchanged. A browser per worker is expensive, so route only the failing URLs here ([more on intercepting network traffic in Playwright](https://roundproxies.com/blog/intercept-network-playwright/)).

## Troubleshooting

### "ValueError: No thread\_items in page"

**Why:** Threads served the empty app shell, or redirected you to a login page.

**Fix:** Print `resp.url` inside `fetch()` to catch login redirects, confirm the `Sec-Fetch-*` headers went out, and keep `impersonate="chrome"`. Private and age-restricted accounts don't embed posts for logged-out visitors at all.

### 403 "Page Not Found" from /graphql/query

**Why:** Meta closed that path to logged-out clients.

**Fix:** POST to `/api/graphql` with `lsd` and `doc_id`, as in Method 3.

### "JSONDecodeError" from the GraphQL call

**Why:** The endpoint sent back HTML, which usually means a stale `lsd` or a rotated `doc_id`.

**Fix:** Refetch the page, read a fresh `lsd`, and retry once. If it still fails, capture a new `doc_id` from DevTools.

### "429 Too Many Requests"

**Why:** Too many requests from one IP in a short window.

**Fix:** Back off exponentially, widen the delay range, and rotate IPs between sessions ([our 429 guide covers the backoff math](https://roundproxies.com/blog/http-error-429/)).

## FAQ

### Can you scrape Threads without logging in?

Yes. Public posts, replies, and profiles embed their data for logged-out visitors, which is what Methods 1 to 3 read.

Search is the weak spot: logged out, you get about one page of results, and follower lists need an account.

### Is it legal to scrape Threads?

Scraping public data is lawful in many jurisdictions, but Threads runs under Instagram's Terms of Use, which bar automated collection without Meta's permission.

Keep to public data, go slow, and skip personal data you don't need. Talk to a lawyer before any commercial use; this isn't legal advice.

### Does Threads have an official API for public posts?

Yes, with limits. The Threads API covers your own posts and insights, plus public keyword and topic-tag search once Meta approves threads\_keyword\_search. Public profile lookups need threads\_profile\_discovery, which carries extra access restrictions.

### Can I use the Threads oEmbed endpoint to collect post data?

No. Since 15 June 2026 the oEmbed endpoint works without a token, at up to 1,000 requests an hour.

But [Meta's embed docs](https://developers.facebook.com/documentation/threads/tools-and-resources/embed-a-threads-post) forbid using its output for anything except displaying the post. Use it to embed posts on your site, not to fill a database.

### Can you collect Threads posts through ActivityPub?

Not usefully. Only accounts that opted into fediverse sharing federate at all.

Community testing found their outbox returns a post count rather than posts, and fetches need signed requests from a publicly reachable server.

### Do you need proxies to scrape Threads?

Not for small, slow jobs from your own connection. You need them for sustained runs, because rate limits apply per IP and a single address is the first thing to collect 429s.

## Wrapping up

Start with Method 1 on a single post and print the raw `thread_items` before you write a parser; the payload tells you more than any guide can.

Add Method 3 when you need history and Method 4 when you need search. Keep the Playwright fallback for URLs that refuse to cooperate.

The parts that rot are the `doc_id` and the field names. Re-check both whenever a run comes back empty, and see the [curl\_cffi repository](https://github.com/lexiforest/curl%5Fcffi) for new impersonation targets as Chrome versions move.