You can scrape Bluesky posts, profiles, threads, and follower lists through the open AT Protocol API that the Bluesky app itself calls. Most of it needs no account and no key.
Search is the exception. On and off since early 2025, app.bsky.feed.searchPosts on the public host has answered logged-out callers with a 403 HTML page.
Most tutorials on page one still make that call logged out, which is why their search code fails.
Skip the bsky.app HTML as well. It's a JavaScript app, so the parsing in our Python web scraping guide gets you an empty shell; the data comes from the API.
How do you scrape Bluesky posts?
To scrape Bluesky, call the AT Protocol XRPC API at public.api.bsky.app for profiles, author feeds, threads, and followers, with no login. Keyword search needs an app password, sent through your own PDS. For an account's full history, export its repo as a CAR file; for live posts, subscribe to Jetstream over WebSocket.
| Method | What you get | Auth | Best for |
|---|---|---|---|
| 1. Public API reads | Profiles, posts, followers | None | One account at a time |
| 2. Thread endpoints | Replies, likes, reposts | None | Engagement on one post |
| 3. Logged-in search | Posts matching a keyword | App password | Topic and brand monitoring |
| 4. PDS listRecords | Every post record an account has | None | Full history, no counts |
| 5. CAR repo export | The whole account in one file | None | Archives and bulk jobs |
| 6. Jetstream | Every new post on the network | None | Real-time collection |
Prerequisites
- Python 3.10 or newer
pip install requests websockets atproto(theatprotoSDK only appears in Method 5)- For Method 3: a Bluesky account and an app password from Settings > Privacy and security > App passwords
Every method below builds on one helper, a thin wrapper over a requests session. Our Python requests guide covers sessions if you haven't used them.
import time
import requests
PUBLIC = "https://public.api.bsky.app/xrpc"
http = requests.Session()
# A contact address in the User-Agent lets an operator reach you before they block you
http.headers["User-Agent"] = "bsky-research/1.0 ([email protected])"
def xrpc(method: str, base: str = PUBLIC, **params) -> dict:
resp = http.get(f"{base}/{method}", params=params, timeout=20)
resp.raise_for_status()
return resp.json()
Every list endpoint on Bluesky pages the same way: it returns a cursor, you send it back, and you stop when it's gone. One generator covers feeds, followers, likes, and raw records.
def paginate(method: str, key: str, max_items: int = 1000, base: str = PUBLIC, **params):
cursor, seen = None, 0
while seen < max_items:
# requests drops params set to None, so the first call goes out without a cursor
page = xrpc(method, base=base, limit=100, cursor=cursor, **params)
for item in page.get(key, []):
yield item
seen += 1
if seen >= max_items:
return
cursor = page.get("cursor")
if not cursor:
return
time.sleep(0.5) # polite pacing; the AppView doesn't publish its limits
The key argument names the list in the response: feed, followers, likes, records. A limit of 100 is the maximum on each endpoint used here.
Method 1: Scrape Bluesky profiles, posts, and followers from the public API
public.api.bsky.app serves profiles, author feeds, and follower lists to anyone, behind a cache, and Bluesky's API hosts guide asks public-web readers to use that host.
A profile is one call, and it accepts a handle or a DID:
def scrape_profile(actor: str) -> dict:
p = xrpc("app.bsky.actor.getProfile", actor=actor)
return {
"did": p["did"], # permanent ID; handles can change, DIDs don't
"handle": p["handle"],
"name": p.get("displayName"),
"bio": p.get("description"),
"followers": p.get("followersCount"),
"following": p.get("followsCount"),
"posts": p.get("postsCount"),
"joined": p.get("createdAt"),
}
Store the DID as your primary key. A user who moves to a custom domain handle keeps the same DID, and your joins survive the rename.
Posts come back as "post views": the stored record plus counts the AppView computes. This parser flattens one, and every later method reuses it.
def parse_post(post: dict) -> dict:
record = post.get("record", {})
rkey = post["uri"].rsplit("/", 1)[-1] # at://<did>/app.bsky.feed.post/<rkey>
handle = post["author"]["handle"]
return {
"uri": post["uri"],
"url": f"https://bsky.app/profile/{handle}/post/{rkey}",
"author": handle,
"text": record.get("text", ""),
"created": record.get("createdAt"),
"langs": record.get("langs"),
"likes": post.get("likeCount", 0),
"reposts": post.get("repostCount", 0),
"replies": post.get("replyCount", 0),
"quotes": post.get("quoteCount", 0),
}
createdAt is whatever the author's client wrote, and it can be wrong. indexedAt on the post view is when Bluesky saw it, which is the safer sort key.
The author feed pages back through an account's timeline, newest first:
for item in paginate("app.bsky.feed.getAuthorFeed", "feed", max_items=500,
actor="bsky.app", filter="posts_no_replies"):
if "reason" in item: # a repost by this account, not a post they wrote
continue
print(parse_post(item["post"]))
followers = [f["did"] for f in paginate(
"app.bsky.graph.getFollowers", "followers", max_items=5000, actor="bsky.app")]
Other filter values are posts_with_replies (the default), posts_with_media, posts_and_author_threads, and posts_with_video. Swap getFollowers for getFollows to get who an account follows.
Big accounts have millions of followers, so set max_items before you start.
Write rows out as JSON Lines while you page; our guide to storing scraped data compares that with CSV and SQLite.
Method 2: Scrape replies, likes, and reposts for a single post
getPostThread returns a post with its whole reply tree, up to 1,000 levels deep, but it wants an AT URI instead of the bsky.app link you copied from the browser.
The conversion resolves the handle to a DID first:
def to_at_uri(url: str) -> str:
# https://bsky.app/profile/<handle-or-did>/post/<rkey>
parts = url.rstrip("/").split("/")
actor, rkey = parts[-3], parts[-1]
if not actor.startswith("did:"):
actor = xrpc("com.atproto.identity.resolveHandle", handle=actor)["did"]
return f"at://{actor}/app.bsky.feed.post/{rkey}"
Replies arrive nested, so a recursive walk flattens them and records how deep each one sits:
def walk_replies(node: dict, depth: int = 0):
for child in node.get("replies", []):
if "post" not in child: # notFoundPost or blockedPost: nothing to parse
continue
yield {**parse_post(child["post"]), "depth": depth}
yield from walk_replies(child, depth + 1)
def scrape_thread(url: str, depth: int = 50) -> dict:
uri = to_at_uri(url)
thread = xrpc("app.bsky.feed.getPostThread", uri=uri, depth=depth)["thread"]
return {"post": parse_post(thread["post"]), "replies": list(walk_replies(thread))}
The default depth is 6, which cuts off long arguments mid-thread. Deleted and blocked replies appear as stubs without a post key; skip them and the rest of the tree still parses.
Engagement lists use the same pager:
uri = to_at_uri("https://bsky.app/profile/bsky.app/post/REPLACE_WITH_RKEY")
likers = [l["actor"]["handle"] for l in paginate("app.bsky.feed.getLikes", "likes", uri=uri)]
reposters = [a["handle"] for a in paginate(
"app.bsky.feed.getRepostedBy", "repostedBy", uri=uri)]
quotes = [parse_post(p) for p in paginate("app.bsky.feed.getQuotes", "posts", uri=uri)]
Paste any public post link in place of the placeholder. Each list pages 100 accounts at a time, so a post with 40,000 likes costs 400 requests.
Method 3: Search Bluesky posts by keyword with an app password
Keyword search works once you log in: create a session with an app password, then send searchPosts to your own PDS with an atproto-proxy header that names Bluesky's AppView.
This is the call most tutorials still make, and what it returns:
curl -s -o /dev/null -w "%{http_code}\n" \
"https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts?q=python&limit=1"
# 403 (the body is an HTML error page, not JSON)
curl -s -o /dev/null -w "%{http_code}\n" \
"https://public.api.bsky.app/xrpc/app.bsky.actor.searchActors?q=python&limit=1"
# 200 (account search still works logged out)
Developers probing in September and October 2026 got that 403 on every query to the public host. api.bsky.app answered some keyless searches and refused others, which is no foundation for a scraper.
Log in once and keep the session. createSession allows 30 calls per account per 5 minutes and 300 a day, so a login per request locks you out fast.
import os
def login(handle: str, app_password: str, auth_host: str = "https://bsky.social") -> dict:
resp = http.post(f"{auth_host}/xrpc/com.atproto.server.createSession",
json={"identifier": handle, "password": app_password}, timeout=20)
resp.raise_for_status()
session = resp.json()
# The DID document in the response names the PDS that hosts this account
services = session.get("didDoc", {}).get("service", [])
pds = next((s["serviceEndpoint"] for s in services
if s["id"].endswith("#atproto_pds")), auth_host)
return {**session, "pds": f"{pds}/xrpc", "auth_host": auth_host}
session = login(os.environ["BSKY_HANDLE"], os.environ["BSKY_APP_PASSWORD"])
If your account lives on a self-hosted PDS, pass that server as auth_host. Never use your main password here; an app password can be revoked on its own.
The search loop pages the same way as before, with auth headers on each request:
def search_posts(session: dict, query: str, max_items: int = 500, **filters):
headers = {"Authorization": f"Bearer {session['accessJwt']}",
"atproto-proxy": "did:web:api.bsky.app#bsky_appview"}
cursor, seen = None, 0
while seen < max_items:
resp = http.get(f"{session['pds']}/app.bsky.feed.searchPosts", headers=headers,
params={"q": query, "limit": 100, "cursor": cursor, **filters},
timeout=20)
resp.raise_for_status()
page = resp.json()
posts = page.get("posts", [])
yield from (parse_post(p) for p in posts[: max_items - seen])
seen += len(posts)
cursor = page.get("cursor")
if not cursor or not posts:
return
time.sleep(1)
Bluesky's PDS currently routes app.bsky.* calls to the AppView without the proxy header. Sending it costs nothing and keeps the request working if that default changes.
Filters go in as keyword arguments:
results = search_posts(session, '"web scraping" python', max_items=300,
sort="latest", lang="en",
since="2026-10-01T00:00:00Z") # since is inclusive, until exclusive
for post in results:
print(post["created"], post["author"], post["text"][:80])
Other filters are until, author, mentions, domain, url, and tag, and the query accepts Lucene syntax.
The lexicon warns that the cursor may stop before the full result set. Split long date ranges into day-sized windows instead of paging one giant query.
Access tokens expire within minutes to hours. When a call returns ExpiredToken, trade the refresh token for a new pair:
def refresh(session: dict) -> dict:
resp = http.post(f"{session['auth_host']}/xrpc/com.atproto.server.refreshSession",
headers={"Authorization": f"Bearer {session['refreshJwt']}"}, timeout=20)
resp.raise_for_status()
return {**session, **resp.json()} # new accessJwt and refreshJwt
The refresh token rotates, so overwrite the stored session every time. Reusing an old refresh token fails.
Method 4: Pull an account's raw post records from its PDS
Every Bluesky account lives on a Personal Data Server, and com.atproto.repo.listRecords on that server returns the account's post records as JSON without touching the AppView.
You need the PDS address first, and the account's DID document holds it:
def resolve_pds(did: str) -> str:
if did.startswith("did:plc:"):
doc = http.get(f"https://plc.directory/{did}", timeout=20).json()
else: # did:web:example.com
host = did.split(":", 2)[2]
doc = http.get(f"https://{host}/.well-known/did.json", timeout=20).json()
return next(s["serviceEndpoint"] for s in doc["service"]
if s["id"].endswith("#atproto_pds"))
With the PDS known, the generic pager reads straight from it:
def raw_posts(handle: str, max_items: int = 50_000):
did = xrpc("com.atproto.identity.resolveHandle", handle=handle)["did"]
pds = f"{resolve_pds(did)}/xrpc"
for rec in paginate("com.atproto.repo.listRecords", "records", max_items,
base=pds, repo=did, collection="app.bsky.feed.post"):
value = rec["value"]
yield {"uri": rec["uri"], "text": value.get("text"),
"created": value.get("createdAt"), "reply": "reply" in value}
Raw records carry no like or repost counts, because the AppView computes those. In exchange you get everything the account still has, including replies an author feed filter hides.
Change collection to app.bsky.feed.like, app.bsky.graph.follow, or app.bsky.actor.profile for other record types. Add reverse="true" to page oldest first.
The limits here belong to the PDS. Bluesky's own PDS instances allow 3,000 requests per 5 minutes per IP, per the rate limits page.
Method 5: Download an account's whole repo as a CAR file
com.atproto.sync.getRepo returns an account's entire repository, with posts, likes, follows, and profile, as one binary CAR file that the atproto SDK can decode.
from atproto import CAR # pip install atproto
def export_repo(handle: str) -> list:
did = xrpc("com.atproto.identity.resolveHandle", handle=handle)["did"]
resp = http.get(f"{resolve_pds(did)}/xrpc/com.atproto.sync.getRepo",
params={"did": did}, timeout=180)
resp.raise_for_status()
with open(f"{did.replace(':', '_')}.car", "wb") as fh:
fh.write(resp.content) # keep the raw export; re-parse it whenever you like
car = CAR.from_bytes(resp.content)
return [b for b in car.blocks.values()
if isinstance(b, dict) and b.get("$type") == "app.bsky.feed.post"]
One request replaces hundreds of listRecords pages, which makes this the cheapest way to archive a few thousand accounts.
The trade-off is record keys. Blocks are keyed by content hash, so building at:// URIs means walking the repo's Merkle tree; use Method 4 when you need URIs.
Pass since=<rev> to download only what changed after an earlier export. Images aren't in the CAR at all: com.atproto.sync.listBlobs lists them and getBlob fetches each one.
Method 6: Stream every new Bluesky post live with Jetstream
Jetstream turns the network firehose into filtered JSON over one WebSocket: ask for app.bsky.feed.post commits and each new post arrives seconds after it's written, with no login.
The subscription URL carries the filter, and kinds=commit drops identity and account events you don't need for posts:
import asyncio
import json
from pathlib import Path
import websockets
JETSTREAM = ("wss://jetstream.us-east.bsky.network/xrpc/network.bsky.jetstream.subscribeEvents"
"?collections=app.bsky.feed.post&kinds=commit")
KEYWORDS = ("web scraping", "atproto")
CURSOR_FILE = Path("jetstream.cursor") # last processed seq, for resuming
The collection filter runs on the server. Keyword matching is your job:
async def tail(cursor: int | None = None):
url = f"{JETSTREAM}&cursor={cursor}" if cursor else JETSTREAM
async with websockets.connect(url, subprotocols=["xrpc.v1.json"]) as ws:
async for n, frame in aenumerate(ws):
evt = json.loads(frame).get("payload", {})
if n % 500 == 0 and "seq" in evt:
CURSOR_FILE.write_text(str(evt["seq"])) # checkpoint, not every event
if evt.get("operation") == "create" and "record" in evt:
text = evt["record"].get("text", "")
if any(k in text.lower() for k in KEYWORDS):
print(evt["seq"], evt["did"], text[:100])
aenumerate and the reconnect loop are short:
async def aenumerate(aiter):
n = 0
async for item in aiter:
yield n, item
n += 1
async def run_forever():
while True:
saved = int(CURSOR_FILE.read_text()) if CURSOR_FILE.exists() else None
try:
await tail(saved)
except (websockets.ConnectionClosed, OSError) as exc:
print("reconnecting after:", exc)
await asyncio.sleep(5)
asyncio.run(run_forever())
The cursor is inclusive and delivery is at-least-once, so expect repeats after a reconnect. Key your storage on at://{did}/{collection}/{rkey} and duplicates overwrite themselves.
Bluesky-hosted instances keep a lookback window of about 36 hours. Sleep longer than that and the old cursor gets rejected, so the stream suits collectors that stay up.
Handle delete events too. Bluesky's developer guidelines require services to remove content a user asked to delete, and the stream is how you learn about it.
The Jetstream docs list the v2 hosts (jetstream.us-east and jetstream.us-west) and cap one subscription at 100 collections and 10,000 DIDs.
History before the lookback needs Network Replay, which runs on a metered Bluesky API key.
Where to send each request when you scrape Bluesky
Bluesky's data sits behind four kinds of host, and the 403s, empty results, and surprise limits people report come from sending a call to the wrong one.
| Host | Auth | Answers | searchPosts |
|---|---|---|---|
public.api.bsky.app |
None | Profiles, feeds, threads, followers, likes (cached) | 403 HTML page |
api.bsky.app |
None | Same calls, uncached | Intermittent 403 |
Your PDS + atproto-proxy |
App password session | All app.bsky.* reads, search included |
Works |
| The account's own PDS | None | com.atproto.repo.* and com.atproto.sync.*: raw records, CAR exports, blobs |
Not served |
| Jetstream v2 | None for live, API key for replay | Filtered live stream of records | Not served |
I'd default to the public host for reads, a logged-in session for search, and Jetstream for anything you'd otherwise poll more than once a minute.
Polling search every 30 seconds for a hashtag is the usual mistake. A collection-filtered Jetstream connection sees the same posts sooner, and the live tail isn't metered.
What no method here can get you
Direct messages run through a separate chat service and only expose your own conversations. Deleted posts are gone from every host above once the delete propagates.
Historical engagement counts don't exist either. likeCount is the count right now, so if you need a curve, snapshot the same posts on a schedule. Bluesky has no view counts to scrape.
Troubleshooting
403 Forbidden with an HTML body from searchPosts
Why: The public AppView refuses unauthenticated search. Fix: Use Method 3 and send search through your PDS with a session. The HTML body also breaks resp.json(), so check resp.status_code before parsing.
400 "ExpiredToken"
Why: The access token in your session timed out. Fix: Call refresh() from Method 3, store the returned pair, and retry the request once.
429 "RateLimitExceeded"
Why: You crossed the per-IP PDS limit or the per-account login limit. Fix: Sleep until the ratelimit-reset header's time and reuse one session (our 429 guide covers backoff).
"server rejected WebSocket connection: HTTP 400"
Why: Usually a saved Jetstream cursor older than the lookback window, or a filter past the 100-collection cap. Fix: Delete jetstream.cursor, reconnect from live, and backfill the gap with Method 4.
FAQ
Can you scrape Bluesky without an account?
Yes, for nearly everything. Profiles, feeds, threads, likes, followers, raw PDS records, CAR exports, and the Jetstream live tail all work logged out.
Keyword search is the one feature that needs an app password session.
Is it legal to scrape Bluesky?
Bluesky posts are public by design, and the AT Protocol exposes them on purpose, but the Terms of Service and developer guidelines still apply.
Those guidelines require honoring user deletion requests. Get legal advice before a commercial use; this isn't legal advice.
Does Bluesky have an official API?
Yes. The AT Protocol XRPC API is the official interface, and the Bluesky app uses the same endpoints this guide calls. It's free, documented at docs.bsky.app, and needs no developer approval.
What are Bluesky's API rate limits?
Bluesky's PDS instances allow 3,000 requests per 5 minutes per IP, and createSession allows 30 logins per 5 minutes and 300 per day per account.
The AppView calls its limits generous without publishing numbers. Jetstream's live tail is unmetered.
Do you need proxies to scrape Bluesky?
Usually not. The API is open, the limits are per IP and generous, and a single paced worker stays well under them.
Roundproxies sells datacenter and residential IPs, but for Bluesky I'd spend that money last, after Jetstream and a slower pager.
Wrapping up
Start with Method 1 against one account and print a raw post view before you trust any parser.
Add Method 3 when you need search, and switch to Jetstream once you're polling the same query every few minutes.
The parts that go stale are host behaviour and Jetstream's limits. When search starts returning 403 through your session too, check the API hosts guide first.
