Platform

Credentials

Some sites only show their data to signed-in users. A credential is a stored login for one of those sites: session cookies, a username and password, or an API key. Scrappy keeps it encrypted and uses it when a scrape needs to be signed in.

What it is for

Add a credential when a source sits behind a login, or when a tool needs a third-party API key. You enter the secret once. After that Scrappy uses it for you, tracks whether it still works, and counts its use against a daily limit.

Add a credential

  1. Open Settings, go to the Integrations tab and choose Manage Credentials. This opens the Credentials page (/dashboard/credentials).
  2. Add a credential. Give it a name and enter the site it is for.
  3. Choose the authentication method and fill in its fields.
  4. Save. The credential appears in the table with its type, status and when it was last used.

Credential types

TypeWhat you provideUse it when
COOKIEThe session cookies from a browser where you are already signed in.The site has a sign-in step that is hard to automate, such as a second factor.
LOGINA username and password, the login page URL and the selectors for the form.The site has an ordinary sign-in form.
API_KEYAn API key.The service is reached through an API instead of a page.

For a LOGIN credential, the login configuration tells Scrappy how to fill in the form: the selectors for the username field, the password field and the submit button, and an element that only appears once you are signed in.

{
"name": "Directory account",
"domain": "example.com",
"type": "LOGIN",
"username": "user@example.com",
"password": "your-password",
"loginUrl": "https://example.com/login",
"loginConfig": {
"usernameSelector": "#username",
"passwordSelector": "#password",
"submitSelector": "button[type='submit']",
"successIndicator": ".account-menu"
}
}

LinkedIn has its own entry on the Credentials page, with the fields LinkedIn needs. See LinkedIn.

Encryption at rest

Cookies, usernames, passwords and API keys are encrypted with AES-256-GCM before they are written to the database. AES-256-GCM is authenticated encryption, so stored data that has been tampered with fails to decrypt instead of yielding altered values.

Listing credentials returns metadata only: the name, the site, the type and the status. The secret values are not included.

Health and daily limits

Each credential has a computed status:

StatusMeaning
ACTIVEWorking normally.
RATE_LIMITEDTemporarily blocked by the site.
SOFT_BANNEDThe site has placed a soft restriction on the account.
INVALIDThe credential no longer works.

Set dailyLimit when you create a credential to cap how many requests it makes in a day. GET /api/credentials/{id} returns the status together with dailyLimit and requestsRemaining, so you can see how much of the day's allowance is left.

To test a credential on demand, call POST /api/credentials/{id}/validate. It checks whether the credential is still valid.

Replace a credential

Session cookies expire and passwords change. A stored credential cannot be edited, so rotation means adding the new one and removing the old:

  1. Add a new credential with the fresh cookies, password or key.
  2. Validate it.
  3. Delete the old credential.
# 1. Add the replacement
curl -X POST https://scrappy.gg/api/credentials \
-H "Authorization: Bearer sk_your_api_key" \
-H "Content-Type: application/json" \
-d '{ "name": "Directory session (new)", "domain": "example.com", "type": "COOKIE", "cookies": [ ... ] }'
# 2. Check that it works
curl -X POST https://scrappy.gg/api/credentials/NEW_ID/validate \
-H "Authorization: Bearer sk_your_api_key"
# 3. Delete the old one
curl -X DELETE https://scrappy.gg/api/credentials/OLD_ID \
-H "Authorization: Bearer sk_your_api_key"

An INVALID status is the signal to replace a credential.

Use the API

A cookie credential tied to one source:

curl -X POST https://scrappy.gg/api/credentials \
-H "Authorization: Bearer sk_your_api_key" \
-H "Content-Type: application/json" \
-d '{
"name": "Directory session",
"domain": "example.com",
"type": "COOKIE",
"sourceId": "source_123",
"cookies": [
{ "name": "session_id", "value": "abc123", "domain": ".example.com" }
],
"dailyLimit": 500
}'
RequestWhat it does
GET /api/credentialsList credentials, with status and no secret values.
POST /api/credentialsCreate a credential.
GET /api/credentials/{id}Get one, with its status, daily limit and requests remaining.
POST /api/credentials/{id}/validateTest whether it still works.
DELETE /api/credentials/{id}Delete it.

Where to go next