How to Scrape a Website That Requires a Login (Step by Step, With Code)

The short answer: use a real browser, sign in once by hand, save that signed-in state, and start every later run from it — from the same IP address the login was made on. Then watch for the day the site signs you out, because it will.

Everything below is that answer in working code. It uses Scrappy's cloud browser API, but the steps are the same whatever you run them on; the parts that matter are the fixed IP and noticing the sign-out.

Why the usual approaches break

There are three things people try first, and each fails in its own way.

• Fetch the page with an HTTP client and a pasted cookie. Works for a day. The page is usually rendered by JavaScript, so the HTML you fetch has no data in it — and the cookie stops working as soon as the site notices it is arriving from a different IP and a different client than the one that signed in.
• Script the login form every run. Logins are the most defended page on any site. Repeated scripted sign-ins trigger two-factor prompts, CAPTCHAs and account locks, and each run of your scraper looks like a new device signing in.
• Run headless Chrome on a server. Better, but a datacentre IP and a headless browser are both easy to recognise, and a login replayed from a cloud IP range is precisely what a risk engine is built to question.

What a site expects to see is a returning visitor: the same browser, from the same place, already signed in. So that is what to give it.

The approach

1. Open a real (headful) browser on a residential IP.
2. Sign in once, as a person — in a live view of that browser, so two-factor codes and security questions are answered by you.
3. Save the signed-in state: cookies and localStorage, plus the IP and user agent it was created with.
4. Start every later run from that saved state, through the same IP.
5. When a run lands on the sign-in page instead of the app, stop and ask a person to sign in again — do not overwrite the saved login with a signed-out one.

Setup

You need an API key with the browser:read and browser:write scopes (Settings → API keys). A new account has 100 free credits; a session costs 5 to start and 1 per minute.

import os, time, requests

API = "https://scrappy.gg/api/browser"
H = {"Authorization": f"Bearer {os.environ['SCRAPPY_API_KEY']}"}

def call(method, path, **kw):
    r = requests.request(method, f"{API}{path}", headers=H, timeout=90, **kw)
    r.raise_for_status()
    body = r.json()
    return body.get("data", body)  # most routes wrap their answer in "data"

Step 1: create a profile to hold the login

A profile is an empty container until you capture a login into it. Name it after the account and the site.

profile = call("POST", "/profiles", json={"name": "acme-portal (ops@yourco)"})
profile_id = profile["id"]

Step 2: sign in once, by hand

Start a session, mint a viewer link and open it. The viewer is the live browser: click and type in it exactly as you would on your own machine, including the two-factor code.

created = call("POST", "/sessions", json={})
session_id = created["session"]["id"]          # not created["id"]

call("POST", f"/sessions/{session_id}/navigate",
     json={"url": "https://portal.example.com/login", "waitUntil": "domcontentloaded"})

viewer = call("POST", f"/sessions/{session_id}/viewer-token")
print("Sign in here:", viewer["viewerUrl"])     # valid for ~5 minutes; mint a new one if it lapses
input("Press Enter once you can see the signed-in page...")

The link works on a phone too, which matters more than it sounds: the person who owns the account is often not the person running the script, and a text message is the fastest way to reach them.

Step 3: capture the login

saved = call("POST", f"/profiles/{profile_id}/capture", json={"sessionId": session_id})
print(saved)   # {"captured": true, "cookieCount": 23, "originCount": 2}
call("DELETE", f"/sessions/{session_id}")

The capture records the session's cookies and per-origin localStorage, encrypted, and also the exit IP and user agent the session used. You do not choose the IP; it is whatever the login was made from, and that is the point. A capture that comes back empty is refused rather than saved, so a hiccup cannot wipe a working login.

Step 4: scrape, already signed in

Every later run passes the profileId. The session starts with the cookies in place before the first page loads, through the same IP as the login.

def scrape_orders():
    created = call("POST", "/sessions", json={"profileId": profile_id})
    sid = created["session"]["id"]
    if not created.get("pinned"):
        print("warning: this run is not on the login's own IP; expect a challenge")
    try:
        call("POST", f"/sessions/{sid}/navigate",
             json={"url": "https://portal.example.com/orders", "waitUntil": "domcontentloaded"})

        # Wait for the data, not just the page: the table renders after the bundle runs.
        call("POST", f"/sessions/{sid}/wait-for-selector",
             json={"selector": "table.orders tbody tr", "timeout": 15000})

        rows = call("POST", f"/sessions/{sid}/evaluate", json={"expression": """
            JSON.stringify([...document.querySelectorAll('table.orders tbody tr')].map(tr =>
              [...tr.cells].map(td => td.innerText.trim())))
        """})["result"]
        return rows
    finally:
        call("DELETE", f"/sessions/{sid}")

Two details in there save hours:

• waitUntil: "domcontentloaded", then a wait for the element you need. A navigation returns as soon as the new document exists, before any script has run. Reading the page at that moment gives you an empty table and no error.
• Release in finally. A session you forget keeps running until it is reaped for idleness — and keeps billing until then.

When the site rotates its session cookie during the run (many do), the new one is written back into the profile when the session is released, so the next run starts from the freshest state rather than the one you captured a week ago.

Step 5: notice when the site signs you out

Every login ends eventually — a password change, a security sweep, a fixed 30-day expiry. The failure to design for is the quiet one: a scraper that keeps running on the sign-in page and records "no orders" for a week.

Scrappy watches for it. When a session started from a profile lands on a sign-in page instead of the app, it refuses to save the signed-out state over the good one and sends a BROWSER_LOGIN_LOST webhook:

curl -X POST https://scrappy.gg/api/user/webhooks \
  -H "Authorization: Bearer $SCRAPPY_API_KEY" -H "Content-Type: application/json" \
  -d '{"name": "Logins", "url": "https://yourco.com/hooks/scrappy",
       "secret": "a-long-random-string", "events": ["BROWSER_LOGIN_LOST"]}'

The payload names the profileId, the site's origin and the path it was bounced to. The usual handler sends the account owner a fresh viewer link (step 2 again) and pauses the job until they have signed in. Capturing into the same profile keeps its IP.

When it still gets challenged

• A run fails with 503 UPSTREAM_UNAVAILABLE on navigate. The route to the site failed, not the login. Retry the same session; if it persists, create the next one with "rotateEgress": true for that run only. The login travels with it, and the pin is restored by the next capture.
• pinned: false on create. The login's IP is cooling down after a failure. Wait a few minutes rather than replaying the login from somewhere new.
• A CAPTCHA appears. POST /sessions/{id}/captcha tells you whether one is actually rendered (most sign-in pages load the vendor's script whether or not a challenge shows). A press-and-hold wall needs a person: send the viewer link.
• Two-factor every single time. The site is treating each run as a new device. Check that pinned is true, and that you are not clearing cookies or rotating the exit by default.

Is this allowed?

Automating your own account, or one you are authorised to use, is a question of the site's terms of service — read them. Keep saved logins as carefully as passwords: a session cookie grants access without passing two-factor, which is why Scrappy encrypts them at rest, never returns them from any endpoint, and deletes them when you delete the profile.

Summary

• Hold the login: POST /api/browser/profiles
• Sign in by hand: POST /sessions, /navigate, /viewer-token
• Save it: POST /profiles/{id}/capture
• Scrape signed in: POST /sessions with profileId, then /navigate, /wait-for-selector, /evaluate
• Hear when it ends: BROWSER_LOGIN_LOST webhook

The full reference is in the saved logins guide and the cloud browser guide. If you are driving the browser from an AI agent rather than a script, the MCP server exposes the same calls as tools.