Skip to main content
Most calls answer in a few seconds. Some render a live page or wait for an AI answer: ChatGPT, Gemini, Perplexity and Copilot asks usually take 15–40 s and sometimes over a minute. You can run any of these calls asynchronously. You get a request_id back, and then you either poll for the result or have it sent to your server.

When to use async

Run a call asynchronously when:
  • it can take longer than your HTTP client, proxy or serverless platform allows (AI asks, JavaScript-rendered pages, AI Overviews);
  • you send many calls at once and don’t want to hold a connection open for each one;
  • you want the result pushed to your server as a webhook instead of polling for it;
  • you need to resend a submit after a timeout or a network error without running, and paying for, the call twice.
For a quick call that you need right away, a normal synchronous call is simpler.

Synchronous calls have about two minutes

A normal (synchronous) call has about 115 s to return. Long-running endpoints such as the AI asks fit their retries into that window: if they can’t finish, you get an error response instead of a dropped connection. Failed calls are not billed.
  • Set your HTTP client timeout to 130 s for AI endpoints. A shorter timeout aborts answers that would have arrived.
  • If your connection drops before the response arrives, the call is not billed.
Async calls are not bound by that window.

Run a call asynchronously

Each async option has a header form and a query-parameter form. They mean the same thing.
  • If the call finishes within the wait, you get the normal response.
  • Otherwise you get 202 Accepted with a request_id.
  • Without a wait, the call waits 25 s. A callback with no other async option waits 0 s, so the 202 comes back at once. A callback sent with Prefer: respond-async or async=true and no wait keeps the 25 s.
  • async_wait (or wait=) outside 0–120 is clamped to that range.
  • When you send both forms of the same option, the header wins.
  • Put the query parameters in the query string, also on a POST. Fields in a JSON body go to the endpoint and are not read as async options.
  • The async parameters are removed before the call reaches the endpoint, so they never change what the endpoint sees or what it costs.
X-Callback-URL used to need Prefer: respond-async and failed with 400 callback_requires_async without it. A callback now makes the call async by itself.

The 202 response

Every async-capable endpoint answers 202 in the same shape:
The response also carries a Location header with the poll URL and a Retry-After header in seconds.

Poll for the result

Poll GET /v1/requests/{request_id} with any API key on the same account. Polling is free.
  • While the call runs, it returns 202 in the shape above.
  • When the call completes, it returns the call’s own status, headers and body, as if you had made the call synchronously.
  • The result is held in memory only, never written to disk, for about 15 minutes after the call completes. After that, polling returns 404 request_not_found.
  • Wait retry_after_seconds between polls.
The SDKs poll for you. client.requests.wait() honours Retry-After and raises a timeout error if the call isn’t done in time. To check once without waiting, use client.requests.get(request_id).
When a call was submitted with a callback, the poll also reports the delivery: as a callback object in the 202 body, and as X-Callback-Status, X-Callback-Attempts and X-Callback-Error headers on the final response. For the full delivery state, use GET /v1/requests/{request_id}/callback.

Get the result by webhook

Add a callback URL to a call. When the call completes, ScrapeBadger POSTs the result to that URL as JSON, signed with your account’s webhook secret.
That returns 202 at once. If you also set a wait and the call finishes within it, you get the normal response and the webhook as well.

The payload

  • status, headers and body are the call’s own, errors included. Header names are lower case.
  • body is decoded JSON when the call returned JSON, and text otherwise.
  • A body over 1 MB is left out: you get "body": null and "body_omitted": true. Fetch it from result_url before result_expires_at.
  • batch_id is set when the call came from a batch.
  • attempt and sent_at (Unix seconds) change on every attempt, and each attempt is signed again.
Each delivery carries these headers:

Callback URL rules

  • The URL must be https, on a public address, at most 2,048 characters long, with no user name or password in it.
  • Private, loopback, link-local and cloud-metadata addresses are refused. So are localhost and names ending in .local or .internal.
  • Redirects are not followed.
  • A bad URL gets 400 invalid_callback_url before the call runs. Nothing is billed.
  • URL-encode the URL when you pass it as callback_url.

Retries and duplicates

  • Answer with any 2xx within 10 s.
  • On a timeout, a connection error, 408, 425, 429, 500, 502, 503 or 504, the delivery is retried after 10 s, 60 s and 300 s: four attempts in total. Any other status is final.
  • Delivery is at least once. The same result can arrive more than once, for example when your server handled it but the response didn’t reach us in time. Deduplicate on request_id.
  • Answer first and do the work after. A slow handler times out, and the result is sent again.
  • Calls refused before they run (invalid key, no credits, rate limit) answer synchronously and send no webhook.
  • Deliveries in progress don’t survive a restart of our gateway. If a restart interrupts retries, the delivery is reported as failed with last_error: "interrupted (gateway restart)". Fetch the result from result_url while it is still kept.

Verify the signature

X-ScrapeBadger-Signature is sha256= followed by the hex HMAC-SHA256 of the raw request body, keyed with your webhook secret. This is the same scheme stream monitors use.
  • Compute the HMAC over the exact bytes you received, before any JSON parsing or re-encoding.
  • Compare in constant time.
  • Reject stale deliveries to stop replays: check that sent_at is within a few minutes of now. Take sent_at from the body, which the signature covers. The X-ScrapeBadger-Timestamp header has the same value but is not signed.
The SDKs do all three. verify_signature and verifyWebhookSignature return true only when the signature matches and, when you pass sent_at, it is within tolerance_s (default 300 s) of now.
scrapebadger.webhooks.parse_event(raw_body) and parseWebhookEvent(rawBody) parse the payload into a typed event if you prefer that to plain JSON.

Receive webhooks

Both receivers read the raw body, verify it, skip repeats and answer before doing any work. Store the request_ids you have handled somewhere shared, such as Redis or your database, and keep them for at least an hour.
A 400 or 401 from your receiver is final, so a delivery that fails verification is not retried. You can still fetch its result from result_url.

Your webhook secret

Every account has one webhook secret. It is created the first time you ask for it.
  • Rotate it with POST /v1/account/webhook-secret/rotate, which returns the new secret.
  • The SDKs have client.account.webhook_secret() and client.account.rotate_webhook_secret() (Node: webhookSecret() and rotateWebhookSecret()).
  • Both endpoints need a full-access API key. A key restricted to some APIs gets 403 full_access_required.
  • New deliveries use a rotated secret at once. A delivery that was already retrying keeps the secret it started with, for up to about 7 minutes. Accept both secrets for that long after you rotate.
  • Keep the secret on your server. Anyone who has it can sign payloads that pass your check.

Check a delivery

Both endpoints are free.

Check one delivery

GET /v1/requests/{request_id}/callback returns the delivery state of a request that was submitted with a callback. Other requests get 404 callback_not_found.
  • state is pending (not sent yet), delivering (being sent or waiting to retry), delivered (your server answered 2xx) or failed (a final status, or all four attempts used).
  • last_status is your server’s last HTTP status, and last_error a short reason such as HTTP 503 or timeout.
  • next_attempt_at and delivered_at are Unix seconds.
  • In the SDKs: client.requests.callback_status(request_id) (Node: callbackStatus(id)).

List recent deliveries

GET /v1/account/webhook-deliveries lists your account’s deliveries of the last 24 hours, newest first. Pass limit for up to 200 (default 50). It needs a full-access API key.
endpoint is the path only, without the query. In the SDKs: client.account.webhook_deliveries(limit=50) (Node: webhookDeliveries({ limit })).

Resend safely with an idempotency key

A submit can time out, or the connection can drop, before you see the 202. Without a key, resending it starts, and bills, a second call. With a key, the resend returns the first call. Send Idempotency-Key: <key> (or idempotency_key=<key>) with an async call. A key is 1–200 visible ASCII characters, with no spaces. Make it from your own job id, such as job-4815-try-1.
  • Same key, same request: the call is not run again and not billed again. You get what GET /v1/requests/{request_id} returns now: 202 while the first call runs, or its stored result. The response carries Idempotent-Replayed: true. No second webhook is sent.
  • Same key, different request: 409 idempotency_key_reused.
  • Key on a synchronous call: ignored. The call runs as usual. Idempotency applies to async calls only (a callback counts as async), because synchronous results are never stored.
  • “The same request” means the same method, path, query and body. The async options and the response-shaping parameters (_fields, _jq and the others) don’t count, and neither do headers.
  • Keys belong to your account, so all your API keys share them. A key is kept for about 15 minutes. After that, the same key starts a new call.
  • A replay returns the stored result, errors included. To retry a call that failed, use a new key, such as job-4815-try-2.

Billing

  • An async call is billed once, when it completes, at the endpoint’s normal price. The result’s x-credits-used header says how much.
  • As with synchronous calls, only 2xx results are billed. A call that never completes is not billed.
  • Submitting, polling, delivery status, the delivery list, the webhook secret and webhook deliveries (retries included) are free.
  • A replay with an idempotency key is not billed.
  • Resending the original request without an idempotency key starts, and bills, a second call.

Errors

Async errors answer before the call runs, so they are not billed. The body is {"error": "<code>", "detail": "<what to fix>"}. These routes can’t run asynchronously. They answer 400 async_not_supported to async, async_wait, callback_url, idempotency_key and X-Callback-URL. A bare Prefer: respond-async or Idempotency-Key header is ignored there, so clients that send those on every call keep working.
  • /v1/requests/*, /v1/batch, /v1/quote, /v1/feedback and /v1/results/*;
  • /v1/account/*;
  • the Twitter stream routes under /v1/twitter/stream/*: managing monitors, webhooks and filter rules, their logs, and the live stream itself;
  • /v1/web/unblock/{job_id};
  • every call that changes something (PUT, PATCH, DELETE, and POST other than the read endpoints such as the AI ask routes and /v1/web/scrape).

Data retention

  • Results (the response body and headers) are held in memory only, never written to disk, for about 15 minutes after the call completes.
  • Idempotency keys are kept for about 15 minutes, as a hash of your key and a hash of the request. Neither the key nor the request is stored.
  • Delivery metadata is kept for 24 hours: the callback host only (never the path or query, which can hold your tokens), the endpoint path only (never the query, which holds your parameters), status codes, attempt counts, short error reasons and timestamps.
  • The webhook secret is kept with your account until you rotate it.

Submit a batch

POST /v1/batch takes up to 100 GET reads and returns a request_id for each, at once:
  • Each item is an ordinary async request under your key. It has the same authentication, permissions, credit check, rate limit and price as if you had sent it alone. Submitting the batch is free.
  • Items are checked against your balance before any of them start, at each endpoint’s base price (what POST /v1/quote reports). Items whose price no longer fits, after the items before them and your other batch items still running, come back as 402 insufficient_credits and are not started.
  • An item that can’t start (for example 402 as above, 429 over your rate limit, or 404 unknown endpoint) comes back with "accepted": false, its status and error. It is not billed, and you can resubmit just that item later.
  • Poll each poll_url, or pass callback_url to get one webhook per item, each with batch_id set.
  • At most 10 items of one batch run at a time. The others wait their turn and show as running when polled.
  • A batch over 100 items gets 413 batch_too_large.
  • Items still waiting for a slot when our gateway restarts are dropped. They are not billed, and polling them returns 404 request_not_found within about 5 minutes. Resubmit those.

Trial keys

Unclaimed trial keys can’t batch (403 trial_not_allowed) or use webhooks (trial_not_allowed). They can still run calls asynchronously and poll for them, one call at a time: a second call while the first is still running gets 429 trial_concurrency.

Concurrency

There is no per-key concurrency limit beyond your plan’s rate limit. AI asks render a live answer, so latency rises when many run at once. Start around 10 in parallel per engine and scale up while you watch the error rate. A 502 on an AI ask means the engine produced no usable answer on that attempt. It is not billed, and retrying it is safe.