All posts

November 3, 20267 min readbrowsertutorialscraping

How to Keep a Playwright or Puppeteer Session Logged In Across Runs

Save storage state, restore it before the first page load, and return from the same IP. The local recipe in code, four ways it still fails, and the hosted way.

By Scrappy Team

The short answer: save the browser's signed-in state (cookies and localStorage, not cookies alone), restore it before the first page of the next run loads, and make that run from the same IP and timezone the login was made from. Then watch for the day the site signs you out anyway, and never save a signed-out state over a good one. Most "my saved session keeps getting logged out" problems come from the third step, not the first.

Here is the local recipe for Playwright and Puppeteer, then the four ways it still fails, each of which we have hit running saved logins for our customers, then how our hosted browser does it.

The local recipe

Playwright: storage state

Sign in once, by hand, in a headed browser, and save the state:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    context = browser.new_context()
    page = context.new_page()
    page.goto("https://portal.example.com/login")
    input("Sign in in the browser window, then press Enter here...")
    context.storage_state(path="portal-state.json")   # cookies + localStorage
    browser.close()

Every later run starts from it:

with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context(storage_state="portal-state.json")
    page = context.new_page()
    page.goto("https://portal.example.com/orders")
    # ... do the work ...
    context.storage_state(path="portal-state.json")   # keep the rotated cookies
    browser.close()

The save at the end matters. Many sites rotate their session cookie as you use them, so the state from a week ago may already be stale while today's is fine.

Playwright or Puppeteer: a persistent profile

The other approach keeps the whole browser profile on disk: launch_persistent_context(user_data_dir=...) in Playwright, or userDataDir in Puppeteer. That covers IndexedDB and service workers too, which storage state does not. The cost: one profile can be used by one browser at a time, it grows, and it is awkward to move between machines.

Why it still gets logged out

1. You came back from somewhere else

This is the big one. A session cookie is issued to a browser at an address. Replay it from a different IP, especially a datacenter one, or with a timezone that does not match the IP, and the site's risk engine sees a stolen session. You get a two-factor prompt, a "was this you?" email, or a quiet sign-out.

The fix is boring and strict: a fixed exit per login. On your own machine that is your home IP, which is why "it works on my laptop and not on the server" is so common. On a server it means a static residential proxy per login, used every time, with the browser's timezone set to match it. A rotating proxy undoes all of it.

2. You restored cookies but not localStorage

Plenty of apps keep part of the sign-in in localStorage: a token, a tenant id, a "remember this device" flag. Restore only cookies and you get a page that half-loads, then sends you to sign in. Storage state saves both, but code that copies cookies by hand often does not. Storage state also leaves out sessionStorage, and IndexedDB unless your Playwright version offers the option to include it and you turn it on.

Saved state can also be large. Four of the saved logins we hold carry 148 to 192 KB of it. For six days, our own restore failed on every one of those four: a large write reported "slow down", and our code took that as "failed". Logins under about 108 KB restored fine, which is what gave it away. If a login restores for some sites but not others, check its size.

3. You restored it too late

Inject the state before the first navigation. A page that loads signed out often redirects to the sign-in form and replaces the very cookies you were about to restore. With storage state this is automatic, since it is applied when the context is created. If you set cookies on an open page, set them first, then navigate.

4. You saved a signed-out state over a good one

The quiet failure. A run lands on the sign-in page, finishes "successfully" with no data, and saves its state at the end, overwriting the working login with a signed-out one. Now every later run is signed out, and nothing errored.

Check before you save: is the page you are on actually signed in? Only then write the state back. We learned this one from production too. In one week, our guard refused 15 automatic saves for a single customer login that the site had signed out. Each time, it kept the last good copy instead of writing over it.

The hosted way

Running saved logins for other people at scale is the four points above, done every time. That is what Scrappy's saved logins do, through a REST API:

import os, requests

API = "https://scrappy.gg/api/browser"
H = {"Authorization": f"Bearer {os.environ['SCRAPPY_API_KEY']}"}

def call(method, path, **kw):
    r = requests.request(method, f"{API}{path}", headers=H, timeout=90, **kw)
    r.raise_for_status()
    body = r.json()
    return body.get("data", body)

# Once: sign in by hand in the live viewer, then capture.
profile = call("POST", "/profiles", json={"name": "portal (ops@yourco)"})
sid = call("POST", "/sessions", json={})["session"]["id"]
call("POST", f"/sessions/{sid}/navigate", json={"url": "https://portal.example.com/login"})
print("Sign in here:", call("POST", f"/sessions/{sid}/viewer-token")["viewerUrl"])
input("Press Enter once signed in...")
call("POST", f"/profiles/{profile['id']}/capture", json={"sessionId": sid})
call("DELETE", f"/sessions/{sid}")

# Every run after: start already signed in.
run = call("POST", "/sessions", json={"profileId": profile["id"]})
print("same IP as the login:", run.get("pinned"))

How each failure is handled:

  • Same place: the capture records the session's exit IP and user agent. Every later session on that login goes out through the same static residential address, with the browser's timezone set to that address's. pinned on create tells you whether this run is on it.
  • Cookies and localStorage: both are captured, encrypted at rest, and never returned by any endpoint.
  • Before the first page: the state is injected when the session is built, before your first navigation.
  • Never overwrite a good login: a capture that comes back empty is refused, and an automatic capture of a page showing a sign-in form is refused too. When a site has signed the login out, a BROWSER_LOGIN_LOST webhook names the login and where the site sent you, so a person can sign in again in the viewer. Capturing into the same login keeps its IP.
  • Rotated cookies: written back to the login when the session is released, so the next run starts from the freshest state.

The full flow, from signing in by hand to running signed in, is 03-saved-login.ts in our examples repo. How to scrape a website that requires a login walks through the same flow step by step.

Summary

  • Save cookies and localStorage, restore them before the first page loads, and save again at the end of a good run.
  • Come back from the same IP and timezone every time. This is the step that fails most.
  • Check that you are signed in before saving, so a sign-out never overwrites a working login.
  • Expect the login to end eventually, and make sure something tells you when it does.

More in the saved logins guide and the proxy write-up. New accounts get 100 free credits.

Keep reading