S3 export

Set it up

Each account has one export destination. Set it in Dashboard → Settings → S3 export, or with PUT /v1/exports/s3. You need an existing bucket and an access key that may write to it.

1. Create (or pick) a bucket. Objects are written under a key prefix, scrapely/ by default.
2. Create an access key limited to s3:PutObject on that bucket and prefix (an example policy is below).
3. Enter the bucket, region and keys, then Test connection: we write scrapely-connection-test.txt under the prefix and report what the storage service answered.
4. Save. Exports switch on only after a passing test.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-scrapes/scrapely/*"
    }
  ]
}

What is written, and where

While the destination is on, every watchlist event is written to the bucket as one JSON object, in the background as each run finishes: an upload never delays or fails the run. It happens whether or not you also receive the event by webhook.

Each object also carries x-amz-meta-scrapely-watchlist-id, x-amz-meta-scrapely-run-id and x-amz-meta-scrapely-event, so you can route it without opening it.

scrapely/watchlists/wl_3c4d5e6f7a8b9c0d/wlrun_7a8b9c0d1e2f3a4b/watchlist.product.changed-1.json
scrapely/watchlists/wl_3c4d5e6f7a8b9c0d/wlrun_7a8b9c0d1e2f3a4b/watchlist.product.changed-2.json
scrapely/watchlists/wl_3c4d5e6f7a8b9c0d/wlrun_7a8b9c0d1e2f3a4b/watchlist.run.completed-1.json
scrapely/scrapely-connection-test.txt

S3-compatible stores

Choose Custom endpoint (or send endpoint) to write to any service that speaks the S3 API. Requests are signed with AWS Signature Version 4 using the region you give.

API

• GET/v1/exports/s3The destination (access key id masked) and its 20 most recent uploads. 404 when there is none.
• PUT/v1/exports/s3Create or change it. Fields you leave out keep their value. Tests the connection first unless "test": false; "enabled": true is refused (400) if the test fails, and nothing is saved.
• DELETE/v1/exports/s3Remove the destination and its stored keys. Objects already written stay in your bucket.
• POST/v1/exports/s3/testWrite the test object now. With no body it tests the saved destination; with destination fields it tests those without saving.

curl -X PUT "https://api.scrapelyserp.com/v1/exports/s3" \
  -H "Authorization: Bearer $SCRAPELY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "bucket": "my-scrapes",
    "region": "us-east-1",
    "access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "prefix": "scrapely/"
  }'

# → { "object": "s3_destination", "bucket": "my-scrapes", "access_key_id": "••••MPLE",
#     "enabled": true, "test": { "ok": true, "key": "scrapely/scrapely-connection-test.txt" }, … }

Errors and retries

An upload that fails because the storage service is busy or unreachable (a timeout, a 5xx, SlowDown) is retried after 10 s, 1 min, 5 min, 30 min and 2 h. A refusal (AccessDenied, NoSuchBucket, a wrong key) is not retried: fix the destination and later events export again. Every attempt is listed under Recent exports in Settings and in recent_exports from the API (newest 100 kept).