Webhooks and integrations

What it is for

Webhooks push events to your own code. Data destinations push leads to another system. Gmail connects a mailbox. This guide covers webhooks in detail and the other two briefly.

Create a webhook

1. Open Settings, go to the Integrations tab and choose Manage Webhooks.
2. Create a webhook and give it a name. Names are unique within your account.
3. Enter the endpoint URL. It must be reachable from the public internet.
4. Enter a secret, or use Generate to create one. Copy it to your server now; you need it to verify signatures.
5. Tick the events you want, then save.

curl -X POST https://scrappy.gg/api/user/webhooks \
  -H "Authorization: Bearer sk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Job notifications",
    "url": "https://your-server.com/webhook",
    "secret": "a-long-random-string",
    "events": ["JOB_COMPLETED", "JOB_FAILED", "LEAD_CREATED"]
  }'

name, url and at least one event are required. To change a webhook, or to switch it off without deleting it, send PATCH /api/user/webhooks/{id} with the fields to change, such as {"isActive": false}.

Events

The two browser events are for integrations that drive the cloud browser through the API, and you subscribe to them through the API. They carry session and profile ids and a reason, and never page text, cookie values or query strings.

Payload

Every delivery is an HTTP POST with a JSON body in the same envelope. data depends on the event; the rest is always present.

{
  "id": "0b8f6c1e-5f2a-4c53-9d0e-3f1f2a7b9c10",
  "event": "JOB_COMPLETED",
  "timestamp": "2024-01-15T10:00:00.000Z",
  "data": {
    "job": {
      "id": "job_123",
      "name": "Company Directory Scrape",
      "status": "COMPLETED",
      "completedAt": "2024-01-15T10:00:00.000Z"
    },
    "results": { "totalPages": 12, "scrapedPages": 12, "leadsFound": 150 },
    "source": { "id": "source_123", "name": "Company Directory", "url": "https://example.com/directory" }
  },
  "metadata": { "userId": "user_123", "projectId": "project_123", "sourceId": "source_123", "jobId": "job_123" }
}

Verify the signature

Anyone who learns your endpoint URL can post to it. The signature proves a request came from Scrappy and was not changed on the way. Compute an HMAC-SHA256 of the raw request body with your secret, encode it as hex, and compare it with the X-Scrappy-Signature header.

const crypto = require("crypto");
const express = require("express");

const app = express();
const SECRET = process.env.SCRAPPY_WEBHOOK_SECRET;

// Read the raw body: the signature is computed over the exact bytes sent.
app.post("/webhook", express.raw({ type: "application/json" }), (req, res) => {
  const received = req.get("X-Scrappy-Signature") || "";
  const expected = crypto.createHmac("sha256", SECRET).update(req.body).digest("hex");

  const a = Buffer.from(received);
  const b = Buffer.from(expected);
  if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) {
    return res.status(401).send("Invalid signature");
  }

  const event = JSON.parse(req.body.toString("utf8"));
  // Respond quickly, then do the work.
  res.status(200).send("ok");
  handle(event);
});

Delivery, retries and history

• A delivery succeeds when your endpoint answers with a 2xx status within 10 seconds.
• Redirects are not followed. A 3xx answer counts as a failure.
• A failed delivery is retried automatically, up to three attempts in total, with a growing delay between them.
• After 10 consecutive failed deliveries the webhook is switched off. Fix the endpoint, then switch it back on.

Each webhook has a delivery history showing the event, the status, the response code, the number of attempts and the time. A delivery is PENDING, DELIVERED, RETRYING or FAILED. Use Retry on a failed delivery to send it again, or call POST /api/user/webhooks/{id}/deliveries/{deliveryId}/retry. The same history is in the deliveries array of GET /api/user/webhooks/{id}.

Gmail

Connecting a Gmail account lets Scrappy read and send email from that mailbox. The connection uses Google's OAuth consent screen, so Scrappy never sees your password. It asks for permission to read, send and modify mail and to see your email address and profile. You can remove a connection at any time, from the dashboard or with DELETE /api/gmail-integrations/{id}.

Data destinations

A data destination syncs leads to an external system. The supported types are HUBSPOT, SALESFORCE, WEBHOOK, SLACK, GOOGLE_SHEETS and AIRTABLE. Create one on the Integrations page (/dashboard/integrations) with a name, a type and the configuration that type needs, such as an API key. Trigger a sync with POST /api/data-destinations/{id}/sync, and set isActive to false to stop syncing without deleting the destination.

Where to go next

• Teams and API keys: create the API key the examples use.