You can scrape Threads.com posts, replies, and profiles with an ordinary HTTP client. Each public page arrives with its data already embedded as JSON, before any JavaScript runs.
Most of page one pushes you toward a headless browser or a paid scraper. The five methods below run on curl_cffi and plain Python, with Playwright kept as a fallback.
Threads moved from threads.net to threads.com in April 2025. Old links still redirect, so the code accepts either domain.
How do you scrape Threads.com posts?
To scrape Threads.com posts, request a public post or profile URL with a Chrome TLS fingerprint and browser headers, then parse the <script data-sjs> JSON blobs that contain thread_items. For older posts, replay the page's own GraphQL request using its lsd token. For keyword search, use Meta's official Threads API.
| Method | What you get | Needs | Breaks when |
|---|---|---|---|
| 1. Post page JSON | Post, replies, likes, reply counts | Nothing | Meta renames JSON keys |
| 2. Profile page JSON | Bio, followers, recent posts | Nothing | Meta renames JSON keys |
| 3. GraphQL paging | Older posts past the first batch | A doc_id from DevTools |
The doc_id rotates |
| 4. Official keyword search | Public posts matching a query | Meta app and approved permission | You exceed 2,200 queries a day |
| 5. Media download | Images and videos | Nothing | The signed URL expires |
Prerequisites
- Python 3.10 or newer (curl_cffi's current minimum)
pip install curl_cffi parsel requests- For Method 4: a Meta developer app with the Threads use case and a user access token
- For the fallback:
pip install playwright, thenplaywright install chromium
All five methods share one session. curl_cffi reproduces Chrome's TLS and HTTP/2 handshake, which stock requests can't (more on what a TLS fingerprint gives away).
from curl_cffi import requests as cffi
# Without Sec-Fetch-* headers, Threads returns a 200 with an empty app shell
PAGE_HEADERS = {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "none",
"Sec-Fetch-User": "?1",
"Upgrade-Insecure-Requests": "1",
}
session = cffi.Session(impersonate="chrome") # Chrome TLS + HTTP/2 fingerprint
def fetch(url: str) -> str:
resp = session.get(url, headers=PAGE_HEADERS, timeout=20)
resp.raise_for_status()
return resp.text
The header set is the part people skip. One open-source Threads scraper documents the empty-shell response in its changelog, and it's the first thing I'd check when a scraper returns nothing.
Method 1: Extract a post and its replies from the page JSON
A Threads post page stores the post and its loaded replies in <script type="application/json" data-sjs> tags, under a key named thread_items.
You can confirm it in a minute. Open a post, press F12, and search the Elements panel for the post's first few words; the match sits inside one of those script tags.
Two helpers do the heavy lifting. One pulls the right script blobs, and the other finds a key at any depth.
import json
from parsel import Selector
def hidden_datasets(html: str, must_contain: str) -> list:
"""Return every data-sjs JSON blob whose raw text mentions `must_contain`."""
blobs = Selector(text=html).css('script[type="application/json"][data-sjs]::text').getall()
# cheap substring test first; most blobs are config you don't need
return [json.loads(raw) for raw in blobs if must_contain in raw]
def find_key(obj, key: str):
"""Yield every value stored under `key`, however deep it sits."""
if isinstance(obj, dict):
for k, v in obj.items():
if k == key:
yield v
yield from find_key(v, key)
elif isinstance(obj, list):
for item in obj:
yield from find_key(item, key)
Searching by key instead of by fixed path survives Meta reshuffling the wrapper objects around the data.
Each item in thread_items wraps a post object. The field names below are the ones community parsers use, so print one raw item and compare before you trust a column.
def parse_post(item: dict) -> dict:
post = item.get("post", item)
user = post.get("user") or {}
info = post.get("text_post_app_info") or {}
return {
"code": post.get("code"), # the shortcode in /post/<code>
"username": user.get("username"),
"text": (post.get("caption") or {}).get("text"),
"published": post.get("taken_at"), # Unix seconds
"likes": post.get("like_count"),
"replies": info.get("direct_reply_count"),
"reposts": info.get("repost_count"),
"quotes": info.get("quote_count"),
"url": f"https://www.threads.com/@{user.get('username')}/post/{post.get('code')}",
}
Reply counts live under text_post_app_info. If your payload keeps repost or quote counts somewhere else, those two lines are the ones to change.
The scraper ties the three functions together:
def scrape_post(url: str) -> dict:
html = fetch(url)
posts = {}
for blob in hidden_datasets(html, "thread_items"):
for thread_items in find_key(blob, "thread_items"):
for item in thread_items:
parsed = parse_post(item)
posts.setdefault(parsed["code"], parsed) # de-dupe, keep page order
if not posts:
raise ValueError("No thread_items in page")
ordered = list(posts.values())
return {"post": ordered[0], "replies": ordered[1:]}
print(scrape_post("https://www.threads.com/@meta/post/DDzbnVKx57R"))
The first entry is the post you asked for. The rest are whatever replies the page rendered, which is usually a partial set; Method 3 fetches the remainder.
Method 2: Scrape a Threads profile and its recent posts
A profile page at threads.com/@username embeds the user object, with bio, follower_count, and verification flags, next to the first batch of that user's posts.
def parse_profile(user: dict) -> dict:
return {
"username": user.get("username"),
"full_name": user.get("full_name"),
"bio": user.get("biography"),
"followers": user.get("follower_count"),
"verified": user.get("is_verified"),
"private": user.get("text_post_app_is_private"),
"links": [link.get("url") for link in user.get("bio_links") or []],
}
The scraper reuses both helpers from Method 1:
def scrape_profile(username: str) -> dict:
html = fetch(f"https://www.threads.com/@{username}")
profile, posts = {}, []
for blob in hidden_datasets(html, "ScheduledServerJS"):
for user in find_key(blob, "user"):
# quotes and reposts embed other users too, so match the handle
if isinstance(user, dict) and user.get("username") == username and "follower_count" in user:
profile = profile or parse_profile(user)
for thread_items in find_key(blob, "thread_items"):
posts.extend(parse_post(item) for item in thread_items)
return {"profile": profile, "posts": posts}
Skip the username check and you'll occasionally store a quoted account's follower count under the wrong profile. Logged out, the page carries the latest handful of posts, not the archive.
Method 3: Page past the first batch with Threads' GraphQL endpoint
Threads loads older posts by POSTing to /api/graphql with a doc_id that names the query, a variables JSON object, and an lsd token printed into the page.
Don't copy a doc_id from a blog post, this one included. Meta rotates them, and every scraper holding a stale ID fails on the same day.
Grab a current one from your own browser. Open a profile, filter the Network tab by graphql, and scroll down.
Pick the request whose response contains thread_items, then copy doc_id and variables from its payload.
The lsd token comes from the profile HTML you already fetched:
import re
def page_lsd(html: str) -> str:
# Meta pages print the token as ["LSD",[],{"token":"..."}
m = re.search(r'"LSD",\[\],\{"token":"([^"]+)"', html)
if not m:
raise ValueError("LSD token not found; the page was probably an empty shell")
return m.group(1)
Then replay the request with that token and the same-origin headers a browser sends:
def graphql(doc_id: str, variables: dict, lsd: str, referer: str) -> dict:
resp = session.post(
"https://www.threads.com/api/graphql",
data={"lsd": lsd, "doc_id": doc_id, "variables": json.dumps(variables)},
headers={
"X-FB-LSD": lsd,
"X-IG-App-ID": "238260118697367", # web app ID from public reverse engineering; confirm in DevTools
"Origin": "https://www.threads.com",
"Referer": referer,
"Sec-Fetch-Site": "same-origin",
"Sec-Fetch-Mode": "cors",
"Sec-Fetch-Dest": "empty",
},
timeout=20,
)
resp.raise_for_status()
return resp.json()
Use /api/graphql. The older /graphql/query path now answers logged-out clients with a 403.
Paging is a loop over the cursor:
import random
import time
def paginate(doc_id: str, variables: dict, lsd: str, referer: str, max_pages: int = 5):
for _ in range(max_pages):
data = graphql(doc_id, variables, lsd, referer)
for thread_items in find_key(data, "thread_items"):
yield from (parse_post(item) for item in thread_items)
info = next(find_key(data, "page_info"), None) or {}
if not info.get("has_next_page"):
break
# Meta's GraphQL cursors usually go in "after"; match your captured variables
variables = {**variables, "after": info.get("end_cursor")}
time.sleep(random.uniform(2, 5))
If the endpoint returns HTML or an errors object, refetch the page for a fresh lsd before you blame the doc_id.
A stale token is the cheaper problem to fix, so rule it out first.
Method 4: Search public Threads posts with the official API
Meta's keyword_search endpoint returns public posts matching a keyword or topic tag, sorted by TOP or RECENT, at up to 100 posts per call.
It's the only sanctioned search on Threads, and scraping can't match its reach. Logged out, threads.com search shows roughly one page of about 20 results, according to one scraper author's testing.
Setup happens in the Meta developer dashboard. Create an app with the Threads use case, add the threads_basic and threads_keyword_search permissions, and generate a token in Graph API Explorer.
Until App Review approves threads_keyword_search, results only include your own posts (Meta's keyword search docs spell out the rules).
import os
import requests
def keyword_search(query: str, search_type: str = "RECENT", since: int | None = None) -> list:
params = {
"q": query,
"search_type": search_type, # TOP (default) or RECENT
"fields": "id,text,media_type,permalink,timestamp,username,has_replies",
"limit": 100, # documented maximum
"access_token": os.environ["THREADS_ACCESS_TOKEN"],
}
if since:
params["since"] = since # Unix time, no earlier than 1688540400
resp = requests.get("https://graph.threads.com/v1.0/keyword_search", params=params, timeout=20)
resp.raise_for_status()
return resp.json().get("data", [])
The quota is per user, not per app: 2,200 queries per rolling 24 hours, and repeating a keyword spends another query.
Empty results don't count against it. Meta also returns an empty array for keywords it deems sensitive.
The 1688540400 floor is 5 July 2023, the week Threads launched.
Method 5: Download images and videos before the links expire
Media URLs sit in the same post objects you scraped in Methods 1 to 3: image_versions2.candidates for photos, video_versions for clips, and carousel_media for multi-image posts.
Pass this function the raw post dict, not the output of parse_post, which drops the media fields.
def media_urls(post: dict) -> list:
urls = [v["url"] for v in post.get("video_versions") or []][:1] # first video rendition
candidates = (post.get("image_versions2") or {}).get("candidates") or []
if candidates:
urls.append(max(candidates, key=lambda c: c.get("width", 0))["url"]) # widest image
for child in post.get("carousel_media") or []:
urls.extend(media_urls(child))
return list(dict.fromkeys(urls)) # de-dupe, keep order
Downloading goes through the same session:
from datetime import datetime, timezone
from pathlib import Path
from urllib.parse import parse_qs, urlparse
def expires_at(url: str):
oe = parse_qs(urlparse(url).query).get("oe", [None])[0] # hex Unix timestamp
return datetime.fromtimestamp(int(oe, 16), tz=timezone.utc) if oe else None
def download(url: str, folder: str = "threads_media") -> Path:
Path(folder).mkdir(exist_ok=True)
path = Path(folder, Path(urlparse(url).path).name)
resp = session.get(url, timeout=60)
resp.raise_for_status()
path.write_bytes(resp.content)
return path
Download in the same run that scraped the post. The URLs are signed, and once the oe timestamp passes, the CDN refuses them; log expires_at() if you queue downloads for later.
Scrape Threads without a browser: what the server checks
Threads embeds the JSON or withholds it based on what your request looks like.
JavaScript only turns that JSON into visible posts, so a client that never runs JS can still read everything.
| What Threads looks at | Bare requests call |
What works |
|---|---|---|
| Fetch metadata headers | Missing | Sec-Fetch-* set for a page navigation |
| GraphQL path | /graphql/query gets a 403 |
/api/graphql with lsd and doc_id |
| GraphQL token | Hard-coded or absent | lsd read from the same page |
| Requests per IP | Unbounded | Delays plus IP rotation |
I'd use curl_cffi even though the documented failures above are about headers. It takes the TLS handshake off the list of suspects while you debug the rest.
At volume, one IP will start drawing 429s. Spread sessions across residential IPs and keep each IP for a few related requests.
Add jitter between calls, too. Roundproxies sells residential pools, and any provider with sticky sessions does the job.
Fallback: capture GraphQL with Playwright when the JSON is missing
When a URL keeps coming back empty, let a real browser load it and record the GraphQL responses it receives.
from contextlib import suppress
from playwright.sync_api import sync_playwright
def capture_graphql(url: str, scrolls: int = 5) -> list:
payloads = []
def on_response(resp):
if "/api/graphql" in resp.url and resp.ok:
with suppress(Exception): # some responses aren't JSON
payloads.append(resp.json())
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.on("response", on_response) # record every GraphQL reply
page.goto(url, wait_until="domcontentloaded")
for _ in range(scrolls):
page.mouse.wheel(0, 4000) # each scroll triggers the next batch of posts
page.wait_for_timeout(1500)
browser.close()
return payloads
Feed payloads into find_key(..., "thread_items") and parse_post unchanged. A browser per worker is expensive, so route only the failing URLs here (more on intercepting network traffic in Playwright).
Troubleshooting
"ValueError: No thread_items in page"
Why: Threads served the empty app shell, or redirected you to a login page.
Fix: Print resp.url inside fetch() to catch login redirects, confirm the Sec-Fetch-* headers went out, and keep impersonate="chrome". Private and age-restricted accounts don't embed posts for logged-out visitors at all.
403 "Page Not Found" from /graphql/query
Why: Meta closed that path to logged-out clients.
Fix: POST to /api/graphql with lsd and doc_id, as in Method 3.
"JSONDecodeError" from the GraphQL call
Why: The endpoint sent back HTML, which usually means a stale lsd or a rotated doc_id.
Fix: Refetch the page, read a fresh lsd, and retry once. If it still fails, capture a new doc_id from DevTools.
"429 Too Many Requests"
Why: Too many requests from one IP in a short window.
Fix: Back off exponentially, widen the delay range, and rotate IPs between sessions (our 429 guide covers the backoff math).
FAQ
Can you scrape Threads without logging in?
Yes. Public posts, replies, and profiles embed their data for logged-out visitors, which is what Methods 1 to 3 read.
Search is the weak spot: logged out, you get about one page of results, and follower lists need an account.
Is it legal to scrape Threads?
Scraping public data is lawful in many jurisdictions, but Threads runs under Instagram's Terms of Use, which bar automated collection without Meta's permission.
Keep to public data, go slow, and skip personal data you don't need. Talk to a lawyer before any commercial use; this isn't legal advice.
Does Threads have an official API for public posts?
Yes, with limits. The Threads API covers your own posts and insights, plus public keyword and topic-tag search once Meta approves threads_keyword_search. Public profile lookups need threads_profile_discovery, which carries extra access restrictions.
Can I use the Threads oEmbed endpoint to collect post data?
No. Since 15 June 2026 the oEmbed endpoint works without a token, at up to 1,000 requests an hour.
But Meta's embed docs forbid using its output for anything except displaying the post. Use it to embed posts on your site, not to fill a database.
Can you collect Threads posts through ActivityPub?
Not usefully. Only accounts that opted into fediverse sharing federate at all.
Community testing found their outbox returns a post count rather than posts, and fetches need signed requests from a publicly reachable server.
Do you need proxies to scrape Threads?
Not for small, slow jobs from your own connection. You need them for sustained runs, because rate limits apply per IP and a single address is the first thing to collect 429s.
Wrapping up
Start with Method 1 on a single post and print the raw thread_items before you write a parser; the payload tells you more than any guide can.
Add Method 3 when you need history and Method 4 when you need search. Keep the Playwright fallback for URLs that refuse to cooperate.
The parts that rot are the doc_id and the field names. Re-check both whenever a run comes back empty, and see the curl_cffi repository for new impersonation targets as Chrome versions move.
