> ## Content Index > Fetch the complete content index at: https://roundproxies.com/blog/llms.txt > Use this file to discover other available public pages before exploring further. # The 6 easiest ways to scrape Bluesky in 2026 - URL: https://roundproxies.com/blog/scrape-bluesky/ - Published: 2026-11-04T18:35:13.000Z - Updated: 2026-11-04T19:53:03.000Z - Description: Scrape Bluesky in Python: which host answers each call, the search 403 fix, and a resumable stream. - Author: Marius Bernard - Tags: Web Scraping, #dated-6ae7c5e756ce24000126918e To scrape Bluesky, skip the website and query the AT Protocol API, the same open API the Bluesky app calls to load its own feeds. It returns posts, profiles, threads, and follower lists as structured JSON, and most of those endpoints need no account and no API key. Search is the one place that trips people up. On and off since early 2025, `app.bsky.feed.searchPosts` on the public host has answered logged-out callers with a 403 HTML page instead of JSON. Most tutorials on page one still make that call logged out, so their search code fails, often with a JSON decode error from parsing an HTML error page. If your scraper needs keyword search, plan for an authenticated session from the start. Reading a known profile, thread, or follower list usually works without one. Skip the bsky.app HTML as well. The site is a JavaScript app, so fetching a page with a plain HTTP client and running the parsing from [our Python web scraping guide](https://roundproxies.com/blog/web-scraping-python/) gets you an empty shell. The posts arrive afterward through API requests in the browser. Calling those endpoints yourself gets you the same data with less code, and you don't need a headless browser. **Quick recommendation.** Start with the public API for profiles, threads, and followers, and add a login only when you need search. ## How do you scrape Bluesky posts? The simplest way to scrape Bluesky is to skip the website's HTML and call the AT Protocol XRPC API directly. The Bluesky app is a client for that API, so the JSON it renders is also available at `public.api.bsky.app`. That covers profiles, author feeds, threads, and followers, with no login. Keyword search is the exception. It needs an app password, and the requests go through your own PDS (personal data server). For an account's full history, export its repo as a CAR file. For live posts, subscribe to Jetstream over WebSocket. The table lists the six methods, starting with the least setup. Five of them need no Bluesky account. | Method | What you get | Auth | Best for | | -------------------- | -------------------------------- | ------------ | -------------------------- | | 1\. Public API reads | Profiles, posts, followers | None | One account at a time | | 2\. Thread endpoints | Replies, likes, reposts | None | Engagement on one post | | 3\. Logged-in search | Posts matching a keyword | App password | Topic and brand monitoring | | 4\. PDS listRecords | Every post record an account has | None | Full history, no counts | | 5\. CAR repo export | The whole account in one file | None | Archives and bulk jobs | | 6\. Jetstream | Every new post on the network | None | Real-time collection | ### Which method to start with - **One account's profile and recent posts.** Use method 1\. The responses include engagement counts. Page through the author feed until you have gone back as far as you need. - **Who replied to, liked, or reposted a post.** Use method 2\. It is built around a single post and the activity attached to it. - **Posts about a topic, brand, or keyword.** Use method 3\. It is the only method that searches across accounts, which is why it needs a login. - **An account's complete post history.** Use method 4 to page through the records, or method 5 to download everything in one request. Both skip like and repost counts. - **Many accounts archived in bulk.** Use method 5\. Each CAR file holds a full repo, so you make one request per account. You will need a library that can parse the CAR format. - **Posts as they are published.** Use method 6\. Jetstream pushes new posts from across the network as JSON, and you keep the ones that match your filter. It delivers posts going forward, so pair it with method 4 or 5 if you also need older posts. ### What to know before you start Public API responses are paginated. Each response includes a cursor value. Send it back with your next request to get the following page, and stop when no cursor comes back. For method 3, create an app password in Bluesky's settings and use it in place of your main password. You can revoke an app password at any time without changing your account login. Methods 4 and 5 lack engagement counts because of where they read from. A PDS stores what the account wrote: post text, timestamps, embeds, and reply references. Like and repost totals are calculated across the network by the AppView behind `public.api.bsky.app`. If you need full history and counts together, pull the records from the PDS, then look up counts for the posts you care about through the public API. The API is rate limited. If you get an HTTP 429 response, slow down and retry after a pause. Do not resend the request right away. ## Prerequisites - Python 3.10 or newer - `pip install requests websockets atproto` (the `atproto` SDK only appears in Method 5) - For Method 3: a Bluesky account and an app password from Settings > Privacy and security > App passwords Every method below builds on one helper, a thin wrapper over a requests session. [Our Python requests guide](https://roundproxies.com/blog/requests-python/) covers sessions if you haven't used them. ```python import time import requests PUBLIC = "https://public.api.bsky.app/xrpc" http = requests.Session() # A contact address in the User-Agent lets an operator reach you before they block you http.headers["User-Agent"] = "bsky-research/1.0 (you@example.com)" def xrpc(method: str, base: str = PUBLIC, **params) -> dict: resp = http.get(f"{base}/{method}", params=params, timeout=20) resp.raise_for_status() return resp.json() ``` Every list endpoint on Bluesky pages the same way: it returns a `cursor`, you send it back, and you stop when it's gone. One generator covers feeds, followers, likes, and raw records. ```python def paginate(method: str, key: str, max_items: int = 1000, base: str = PUBLIC, **params): cursor, seen = None, 0 while seen < max_items: # requests drops params set to None, so the first call goes out without a cursor page = xrpc(method, base=base, limit=100, cursor=cursor, **params) for item in page.get(key, []): yield item seen += 1 if seen >= max_items: return cursor = page.get("cursor") if not cursor: return time.sleep(0.5) # polite pacing; the AppView doesn't publish its limits ``` The `key` argument names the list in the response: `feed`, `followers`, `likes`, `records`. A limit of 100 is the maximum on each endpoint used here. ## Method 1: Scrape Bluesky profiles, posts, and followers from the public API `public.api.bsky.app` serves profiles, author feeds, and follower lists to anyone, behind a cache, and Bluesky's [API hosts guide](https://docs.bsky.app/docs/advanced-guides/api-directory) asks public-web readers to use that host. A profile is one call, and it accepts a handle or a DID: ```python def scrape_profile(actor: str) -> dict: p = xrpc("app.bsky.actor.getProfile", actor=actor) return { "did": p["did"], # permanent ID; handles can change, DIDs don't "handle": p["handle"], "name": p.get("displayName"), "bio": p.get("description"), "followers": p.get("followersCount"), "following": p.get("followsCount"), "posts": p.get("postsCount"), "joined": p.get("createdAt"), } ``` Store the DID as your primary key. A user who moves to a custom domain handle keeps the same DID, and your joins survive the rename. Posts come back as "post views": the stored record plus counts the AppView computes. This parser flattens one, and every later method reuses it. ```python def parse_post(post: dict) -> dict: record = post.get("record", {}) rkey = post["uri"].rsplit("/", 1)[-1] # at:///app.bsky.feed.post/ handle = post["author"]["handle"] return { "uri": post["uri"], "url": f"https://bsky.app/profile/{handle}/post/{rkey}", "author": handle, "text": record.get("text", ""), "created": record.get("createdAt"), "langs": record.get("langs"), "likes": post.get("likeCount", 0), "reposts": post.get("repostCount", 0), "replies": post.get("replyCount", 0), "quotes": post.get("quoteCount", 0), } ``` `createdAt` is whatever the author's client wrote, and it can be wrong. `indexedAt` on the post view is when Bluesky saw it, which is the safer sort key. The author feed pages back through an account's timeline, newest first: ```python for item in paginate("app.bsky.feed.getAuthorFeed", "feed", max_items=500, actor="bsky.app", filter="posts_no_replies"): if "reason" in item: # a repost by this account, not a post they wrote continue print(parse_post(item["post"])) followers = [f["did"] for f in paginate( "app.bsky.graph.getFollowers", "followers", max_items=5000, actor="bsky.app")] ``` Other `filter` values are `posts_with_replies` (the default), `posts_with_media`, `posts_and_author_threads`, and `posts_with_video`. Swap `getFollowers` for `getFollows` to get who an account follows. Big accounts have millions of followers, so set `max_items` before you start. Write rows out as JSON Lines while you page; [our guide to storing scraped data](https://roundproxies.com/blog/store-scraped-data/) compares that with CSV and SQLite. ## Method 2: Scrape replies, likes, and reposts for a single post `getPostThread` returns a post with its whole reply tree, up to 1,000 levels deep, but it wants an AT URI instead of the bsky.app link you copied from the browser. The conversion resolves the handle to a DID first: ```python def to_at_uri(url: str) -> str: # https://bsky.app/profile//post/ parts = url.rstrip("/").split("/") actor, rkey = parts[-3], parts[-1] if not actor.startswith("did:"): actor = xrpc("com.atproto.identity.resolveHandle", handle=actor)["did"] return f"at://{actor}/app.bsky.feed.post/{rkey}" ``` Replies arrive nested, so a recursive walk flattens them and records how deep each one sits: ```python def walk_replies(node: dict, depth: int = 0): for child in node.get("replies", []): if "post" not in child: # notFoundPost or blockedPost: nothing to parse continue yield {**parse_post(child["post"]), "depth": depth} yield from walk_replies(child, depth + 1) def scrape_thread(url: str, depth: int = 50) -> dict: uri = to_at_uri(url) thread = xrpc("app.bsky.feed.getPostThread", uri=uri, depth=depth)["thread"] return {"post": parse_post(thread["post"]), "replies": list(walk_replies(thread))} ``` The default depth is 6, which cuts off long arguments mid-thread. Deleted and blocked replies appear as stubs without a `post` key; skip them and the rest of the tree still parses. Engagement lists use the same pager: ```python uri = to_at_uri("https://bsky.app/profile/bsky.app/post/REPLACE_WITH_RKEY") likers = [l["actor"]["handle"] for l in paginate("app.bsky.feed.getLikes", "likes", uri=uri)] reposters = [a["handle"] for a in paginate( "app.bsky.feed.getRepostedBy", "repostedBy", uri=uri)] quotes = [parse_post(p) for p in paginate("app.bsky.feed.getQuotes", "posts", uri=uri)] ``` Paste any public post link in place of the placeholder. Each list pages 100 accounts at a time, so a post with 40,000 likes costs 400 requests. ## Method 3: Search Bluesky posts by keyword with an app password Keyword search works once you log in: create a session with an app password, then send `searchPosts` to your own PDS with an `atproto-proxy` header that names Bluesky's AppView. This is the call most tutorials still make, and what it returns: ```bash curl -s -o /dev/null -w "%{http_code}\n" \ "https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts?q=python&limit=1" # 403 (the body is an HTML error page, not JSON) curl -s -o /dev/null -w "%{http_code}\n" \ "https://public.api.bsky.app/xrpc/app.bsky.actor.searchActors?q=python&limit=1" # 200 (account search still works logged out) ``` Developers probing in September and October 2026 got that 403 on every query to the public host. `api.bsky.app` answered some keyless searches and refused others, which is no foundation for a scraper. Log in once and keep the session. `createSession` allows 30 calls per account per 5 minutes and 300 a day, so a login per request locks you out fast. ```python import os def login(handle: str, app_password: str, auth_host: str = "https://bsky.social") -> dict: resp = http.post(f"{auth_host}/xrpc/com.atproto.server.createSession", json={"identifier": handle, "password": app_password}, timeout=20) resp.raise_for_status() session = resp.json() # The DID document in the response names the PDS that hosts this account services = session.get("didDoc", {}).get("service", []) pds = next((s["serviceEndpoint"] for s in services if s["id"].endswith("#atproto_pds")), auth_host) return {**session, "pds": f"{pds}/xrpc", "auth_host": auth_host} session = login(os.environ["BSKY_HANDLE"], os.environ["BSKY_APP_PASSWORD"]) ``` If your account lives on a self-hosted PDS, pass that server as `auth_host`. Never use your main password here; an app password can be revoked on its own. The search loop pages the same way as before, with auth headers on each request: ```python def search_posts(session: dict, query: str, max_items: int = 500, **filters): headers = {"Authorization": f"Bearer {session['accessJwt']}", "atproto-proxy": "did:web:api.bsky.app#bsky_appview"} cursor, seen = None, 0 while seen < max_items: resp = http.get(f"{session['pds']}/app.bsky.feed.searchPosts", headers=headers, params={"q": query, "limit": 100, "cursor": cursor, **filters}, timeout=20) resp.raise_for_status() page = resp.json() posts = page.get("posts", []) yield from (parse_post(p) for p in posts[: max_items - seen]) seen += len(posts) cursor = page.get("cursor") if not cursor or not posts: return time.sleep(1) ``` Bluesky's PDS currently routes `app.bsky.*` calls to the AppView without the proxy header. Sending it costs nothing and keeps the request working if that default changes. Filters go in as keyword arguments: ```python results = search_posts(session, '"web scraping" python', max_items=300, sort="latest", lang="en", since="2026-10-01T00:00:00Z") # since is inclusive, until exclusive for post in results: print(post["created"], post["author"], post["text"][:80]) ``` Other filters are `until`, `author`, `mentions`, `domain`, `url`, and `tag`, and the query accepts Lucene syntax. The lexicon warns that the cursor may stop before the full result set. Split long date ranges into day-sized windows instead of paging one giant query. Access tokens expire within minutes to hours. When a call returns `ExpiredToken`, trade the refresh token for a new pair: ```python def refresh(session: dict) -> dict: resp = http.post(f"{session['auth_host']}/xrpc/com.atproto.server.refreshSession", headers={"Authorization": f"Bearer {session['refreshJwt']}"}, timeout=20) resp.raise_for_status() return {**session, **resp.json()} # new accessJwt and refreshJwt ``` The refresh token rotates, so overwrite the stored session every time. Reusing an old refresh token fails. ## Method 4: Pull an account's raw post records from its PDS Every Bluesky account lives on a Personal Data Server, and `com.atproto.repo.listRecords` on that server returns the account's post records as JSON without touching the AppView. You need the PDS address first, and the account's DID document holds it: ```python def resolve_pds(did: str) -> str: if did.startswith("did:plc:"): doc = http.get(f"https://plc.directory/{did}", timeout=20).json() else: # did:web:example.com host = did.split(":", 2)[2] doc = http.get(f"https://{host}/.well-known/did.json", timeout=20).json() return next(s["serviceEndpoint"] for s in doc["service"] if s["id"].endswith("#atproto_pds")) ``` With the PDS known, the generic pager reads straight from it: ```python def raw_posts(handle: str, max_items: int = 50_000): did = xrpc("com.atproto.identity.resolveHandle", handle=handle)["did"] pds = f"{resolve_pds(did)}/xrpc" for rec in paginate("com.atproto.repo.listRecords", "records", max_items, base=pds, repo=did, collection="app.bsky.feed.post"): value = rec["value"] yield {"uri": rec["uri"], "text": value.get("text"), "created": value.get("createdAt"), "reply": "reply" in value} ``` Raw records carry no like or repost counts, because the AppView computes those. In exchange you get everything the account still has, including replies an author feed filter hides. Change `collection` to `app.bsky.feed.like`, `app.bsky.graph.follow`, or `app.bsky.actor.profile` for other record types. Add `reverse="true"` to page oldest first. The limits here belong to the PDS. Bluesky's own PDS instances allow 3,000 requests per 5 minutes per IP, per the [rate limits page](https://docs.bsky.app/docs/advanced-guides/rate-limits). ## Method 5: Download an account's whole repo as a CAR file `com.atproto.sync.getRepo` returns an account's entire repository, with posts, likes, follows, and profile, as one binary CAR file that the `atproto` SDK can decode. ```python from atproto import CAR # pip install atproto def export_repo(handle: str) -> list: did = xrpc("com.atproto.identity.resolveHandle", handle=handle)["did"] resp = http.get(f"{resolve_pds(did)}/xrpc/com.atproto.sync.getRepo", params={"did": did}, timeout=180) resp.raise_for_status() with open(f"{did.replace(':', '_')}.car", "wb") as fh: fh.write(resp.content) # keep the raw export; re-parse it whenever you like car = CAR.from_bytes(resp.content) return [b for b in car.blocks.values() if isinstance(b, dict) and b.get("$type") == "app.bsky.feed.post"] ``` One request replaces hundreds of `listRecords` pages, which makes this the cheapest way to archive a few thousand accounts. The trade-off is record keys. Blocks are keyed by content hash, so building `at://` URIs means walking the repo's Merkle tree; use Method 4 when you need URIs. Pass `since=` to download only what changed after an earlier export. Images aren't in the CAR at all: `com.atproto.sync.listBlobs` lists them and `getBlob` fetches each one. ## Method 6: Stream every new Bluesky post live with Jetstream Jetstream turns the network firehose into filtered JSON over one WebSocket: ask for `app.bsky.feed.post` commits and each new post arrives seconds after it's written, with no login. The subscription URL carries the filter, and `kinds=commit` drops identity and account events you don't need for posts: ```python import asyncio import json from pathlib import Path import websockets JETSTREAM = ("wss://jetstream.us-east.bsky.network/xrpc/network.bsky.jetstream.subscribeEvents" "?collections=app.bsky.feed.post&kinds=commit") KEYWORDS = ("web scraping", "atproto") CURSOR_FILE = Path("jetstream.cursor") # last processed seq, for resuming ``` The collection filter runs on the server. Keyword matching is your job: ```python async def tail(cursor: int | None = None): url = f"{JETSTREAM}&cursor={cursor}" if cursor else JETSTREAM async with websockets.connect(url, subprotocols=["xrpc.v1.json"]) as ws: async for n, frame in aenumerate(ws): evt = json.loads(frame).get("payload", {}) if n % 500 == 0 and "seq" in evt: CURSOR_FILE.write_text(str(evt["seq"])) # checkpoint, not every event if evt.get("operation") == "create" and "record" in evt: text = evt["record"].get("text", "") if any(k in text.lower() for k in KEYWORDS): print(evt["seq"], evt["did"], text[:100]) ``` `aenumerate` and the reconnect loop are short: ```python async def aenumerate(aiter): n = 0 async for item in aiter: yield n, item n += 1 async def run_forever(): while True: saved = int(CURSOR_FILE.read_text()) if CURSOR_FILE.exists() else None try: await tail(saved) except (websockets.ConnectionClosed, OSError) as exc: print("reconnecting after:", exc) await asyncio.sleep(5) asyncio.run(run_forever()) ``` The cursor is inclusive and delivery is at-least-once, so expect repeats after a reconnect. Key your storage on `at://{did}/{collection}/{rkey}` and duplicates overwrite themselves. Bluesky-hosted instances keep a lookback window of about 36 hours. Sleep longer than that and the old cursor gets rejected, so the stream suits collectors that stay up. Handle `delete` events too. Bluesky's developer guidelines require services to remove content a user asked to delete, and the stream is how you learn about it. The [Jetstream docs](https://bsky.network/docs/jetstream) list the v2 hosts (`jetstream.us-east` and `jetstream.us-west`) and cap one subscription at 100 collections and 10,000 DIDs. History before the lookback needs Network Replay, which runs on a metered Bluesky API key. ## Where to send each request when you scrape Bluesky Bluesky's data sits behind four kinds of host, and the 403s, empty results, and surprise limits people report come from sending a call to the wrong one. | Host | Auth | Answers | searchPosts | | ------------------------ | --------------------------------- | ---------------------------------------------------------------------------- | ---------------- | | public.api.bsky.app | None | Profiles, feeds, threads, followers, likes (cached) | 403 HTML page | | api.bsky.app | None | Same calls, uncached | Intermittent 403 | | Your PDS + atproto-proxy | App password session | All app.bsky.\* reads, search included | Works | | The account's own PDS | None | com.atproto.repo.\* and com.atproto.sync.\*: raw records, CAR exports, blobs | Not served | | Jetstream v2 | None for live, API key for replay | Filtered live stream of records | Not served | I'd default to the public host for reads, a logged-in session for search, and Jetstream for anything you'd otherwise poll more than once a minute. Polling search every 30 seconds for a hashtag is the usual mistake. A collection-filtered Jetstream connection sees the same posts sooner, and the live tail isn't metered. ### What no method here can get you Direct messages run through a separate chat service and only expose your own conversations. Deleted posts are gone from every host above once the delete propagates. Historical engagement counts don't exist either. `likeCount` is the count right now, so if you need a curve, snapshot the same posts on a schedule. Bluesky has no view counts to scrape. ## Troubleshooting ### 403 Forbidden with an HTML body from searchPosts **Why:** The public AppView refuses unauthenticated search. **Fix:** Use Method 3 and send search through your PDS with a session. The HTML body also breaks `resp.json()`, so check `resp.status_code` before parsing. ### 400 "ExpiredToken" **Why:** The access token in your session timed out. **Fix:** Call `refresh()` from Method 3, store the returned pair, and retry the request once. ### 429 "RateLimitExceeded" **Why:** You crossed the per-IP PDS limit or the per-account login limit. **Fix:** Sleep until the `ratelimit-reset` header's time and reuse one session ([our 429 guide](https://roundproxies.com/blog/http-error-429/) covers backoff). ### "server rejected WebSocket connection: HTTP 400" **Why:** Usually a saved Jetstream cursor older than the lookback window, or a filter past the 100-collection cap. **Fix:** Delete `jetstream.cursor`, reconnect from live, and backfill the gap with Method 4. ## FAQ ### Can you scrape Bluesky without an account? Yes, for nearly everything. Profiles, feeds, threads, likes, followers, raw PDS records, CAR exports, and the Jetstream live tail all work logged out. Keyword search is the one feature that needs an app password session. ### Is it legal to scrape Bluesky? Bluesky posts are public by design, and the AT Protocol exposes them on purpose, but the Terms of Service and developer guidelines still apply. Those guidelines require honoring user deletion requests. Get legal advice before a commercial use; this isn't legal advice. ### Does Bluesky have an official API? Yes. The AT Protocol XRPC API is the official interface, and the Bluesky app uses the same endpoints this guide calls. It's free, documented at docs.bsky.app, and needs no developer approval. ### What are Bluesky's API rate limits? Bluesky's PDS instances allow 3,000 requests per 5 minutes per IP, and createSession allows 30 logins per 5 minutes and 300 per day per account. The AppView calls its limits generous without publishing numbers. Jetstream's live tail is unmetered. ### Do you need proxies to scrape Bluesky? Usually not. The API is open, the limits are per IP and generous, and a single paced worker stays well under them. Roundproxies sells datacenter and residential IPs, but for Bluesky I'd spend that money last, after Jetstream and a slower pager. ## Wrapping up Start with Method 1 against one account and print a raw post view before you trust any parser. Add Method 3 when you need search, and switch to Jetstream once you're polling the same query every few minutes. The parts that go stale are host behaviour and Jetstream's limits. When search starts returning 403 through your session too, check the [API hosts guide](https://docs.bsky.app/docs/advanced-guides/api-directory) first.