Your Scraper Can't Log In. That's Where the Good Data Is.

Here is a thing that is true of almost every lead list bought or built in the last five years: it came from pages anyone can load without signing in.

That is not a criticism. It is just the shape of the tooling. An HTTP client fetches a URL and parses the HTML. If the page needs a session, the client gets a login form instead of the data, and the pipeline quietly records nothing.

So the whole industry has scraped the same public surface over and over. Which is why so many lists look identical, and why the reply rates on them keep falling.

The data you actually want has a door in front of it

Think about where the useful signal lives. A directory your industry association runs for members. A portal your CRM exposes to licensed users. A marketplace that shows pricing only once you have an account. A regional MLS. A supplier catalogue behind a trade login.

None of that is secret. You have credentials. You are allowed to be there. It is simply unreachable to a tool that cannot hold a session.

And because it is unreachable to that tool, it is unreachable to your competitors using the same tools. The barrier that makes it annoying is the same barrier that makes it worth having.

Why "just add cookies" doesn't work

The usual first attempt is to log in by hand, copy the session cookie, and paste it into the scraper. It works for about a day.

Then it stops, for reasons that are all the same reason underneath: the site is checking whether the thing holding the session looks like the thing that created it.

• The session is pinned to context. Many sites bind a session loosely to the IP range, the user agent, or a device fingerprint. Replaying the cookie from a datacentre IP in another country invalidates it.
• The page is JavaScript. The data renders after the request your HTTP client made. There is no HTML to parse, because the HTML arrives later and only in a browser.
• The flow has steps. Real applications make you click into a record, expand a panel, page through results. A URL fetch cannot do any of that.
• Sessions expire. Yours will, usually at the least convenient moment, and a pipeline built on a pasted cookie fails silently rather than loudly.

What actually works: a real browser that stays signed in

The approach that holds up is unglamorous. Use a real browser. Sign in once, properly, like a person. Keep that session. Come back to it.

This is what Scrappy's cloud browser does. A session is a real, headful Chrome — not a headless variant — running on our infrastructure with a stable residential exit. You sign in through an interactive viewer, with your own credentials and your own second factor, and the resulting session state is saved as a profile. The next run reuses that profile: same cookies, same exit IP, same fingerprint, so the site sees the same returning user rather than a new suspicious one.

The sign-in happens once. The automation happens every day after that.

And then you can hand it to an agent

Once a browser can hold a session, something else becomes possible: you no longer need to write the selectors.

The traditional flow is a person inspecting the DOM, writing CSS selectors, and then repairing them every time the site ships a redesign. That maintenance is why so many internal scrapers end up abandoned.

Scrappy's browser sessions can be driven by Claude directly — the model sees screenshots of the page, reads its structure, and clicks, types and navigates its way through a flow it was not pre-programmed for. You describe the goal. It works out the clicks.

That matters most for exactly the gated sources above, because those are the ones nobody has written a parser for. There is no library for your industry's member portal. There is a model that can look at it.

The part worth being careful about

Being able to reach gated data does not mean you should take all of it.

Read the terms you agreed to when you got the account. "I have a login" and "I may bulk-export this" are different permissions, and the second one is often explicitly withheld. Respect rate limits — an automated session that hammers a members' portal is both rude and a fast way to lose the account. And remember that a contact's data remains theirs: GDPR and CCPA do not have an exception for data you found while signed in.

The rule we would apply: automate the things you are allowed to do by hand, at a pace a person could plausibly work at. If the answer to "would they mind?" is yes, do not.

The short version

The public web has been scraped to exhaustion. The data that still differentiates a lead list sits behind a sign-in — not because it is secret, but because the standard toolchain cannot open a door.

A real browser that stays logged in opens it. An AI agent driving that browser means you do not have to maintain a parser for every site you open.

---

Scrappy's cloud browser holds real sessions, saves your logins, and can be driven by Claude or by your own code. Try it with 100 free credits.