Platform
Credentials
Some sites only show their data to signed-in users. A credential is a stored login for one of those sites: session cookies, a username and password, or an API key. Scrappy keeps it encrypted and uses it when a scrape needs to be signed in.
What it is for
Add a credential when a source sits behind a login, or when a tool needs a third-party API key. You enter the secret once. After that Scrappy uses it for you, tracks whether it still works, and counts its use against a daily limit.
Add a credential
- Open Settings, go to the Integrations tab and choose Manage Credentials. This opens the Credentials page (
/dashboard/credentials). - Add a credential. Give it a name and enter the site it is for.
- Choose the authentication method and fill in its fields.
- Save. The credential appears in the table with its type, status and when it was last used.
Credential types
| Type | What you provide | Use it when |
|---|---|---|
COOKIE | The session cookies from a browser where you are already signed in. | The site has a sign-in step that is hard to automate, such as a second factor. |
LOGIN | A username and password, the login page URL and the selectors for the form. | The site has an ordinary sign-in form. |
API_KEY | An API key. | The service is reached through an API instead of a page. |
For a LOGIN credential, the login configuration tells Scrappy how to fill in the form: the selectors for the username field, the password field and the submit button, and an element that only appears once you are signed in.
{"name": "Directory account","domain": "example.com","type": "LOGIN","username": "user@example.com","password": "your-password","loginUrl": "https://example.com/login","loginConfig": {"usernameSelector": "#username","passwordSelector": "#password","submitSelector": "button[type='submit']","successIndicator": ".account-menu"}}
LinkedIn has its own entry on the Credentials page, with the fields LinkedIn needs. See LinkedIn.
Encryption at rest
Cookies, usernames, passwords and API keys are encrypted with AES-256-GCM before they are written to the database. AES-256-GCM is authenticated encryption, so stored data that has been tampered with fails to decrypt instead of yielding altered values.
Listing credentials returns metadata only: the name, the site, the type and the status. The secret values are not included.
Health and daily limits
Each credential has a computed status:
| Status | Meaning |
|---|---|
ACTIVE | Working normally. |
RATE_LIMITED | Temporarily blocked by the site. |
SOFT_BANNED | The site has placed a soft restriction on the account. |
INVALID | The credential no longer works. |
Set dailyLimit when you create a credential to cap how many requests it makes in a day. GET /api/credentials/{id} returns the status together with dailyLimit and requestsRemaining, so you can see how much of the day's allowance is left.
To test a credential on demand, call POST /api/credentials/{id}/validate. It checks whether the credential is still valid.
Replace a credential
Session cookies expire and passwords change. A stored credential cannot be edited, so rotation means adding the new one and removing the old:
- Add a new credential with the fresh cookies, password or key.
- Validate it.
- Delete the old credential.
# 1. Add the replacementcurl -X POST https://scrappy.gg/api/credentials \-H "Authorization: Bearer sk_your_api_key" \-H "Content-Type: application/json" \-d '{ "name": "Directory session (new)", "domain": "example.com", "type": "COOKIE", "cookies": [ ... ] }'# 2. Check that it workscurl -X POST https://scrappy.gg/api/credentials/NEW_ID/validate \-H "Authorization: Bearer sk_your_api_key"# 3. Delete the old onecurl -X DELETE https://scrappy.gg/api/credentials/OLD_ID \-H "Authorization: Bearer sk_your_api_key"
An INVALID status is the signal to replace a credential.
Use the API
A cookie credential tied to one source:
curl -X POST https://scrappy.gg/api/credentials \-H "Authorization: Bearer sk_your_api_key" \-H "Content-Type: application/json" \-d '{"name": "Directory session","domain": "example.com","type": "COOKIE","sourceId": "source_123","cookies": [{ "name": "session_id", "value": "abc123", "domain": ".example.com" }],"dailyLimit": 500}'
| Request | What it does |
|---|---|
GET /api/credentials | List credentials, with status and no secret values. |
POST /api/credentials | Create a credential. |
GET /api/credentials/{id} | Get one, with its status, daily limit and requests remaining. |
POST /api/credentials/{id}/validate | Test whether it still works. |
DELETE /api/credentials/{id} | Delete it. |
Where to go next
- Sources and AI discovery: the sources a credential signs in to.
- Saved logins: keep a sign-in for the cloud browser, where a person or an agent drives the session.
- LinkedIn: LinkedIn credentials and their health.