request_id back, and then you either poll for the result or have it sent to your server.
When to use async
Run a call asynchronously when:- it can take longer than your HTTP client, proxy or serverless platform allows (AI asks, JavaScript-rendered pages, AI Overviews);
- you send many calls at once and don’t want to hold a connection open for each one;
- you want the result pushed to your server as a webhook instead of polling for it;
- you need to resend a submit after a timeout or a network error without running, and paying for, the call twice.
Synchronous calls have about two minutes
A normal (synchronous) call has about 115 s to return. Long-running endpoints such as the AI asks fit their retries into that window: if they can’t finish, you get an error response instead of a dropped connection. Failed calls are not billed.- Set your HTTP client timeout to 130 s for AI endpoints. A shorter timeout aborts answers that would have arrived.
- If your connection drops before the response arrives, the call is not billed.
Run a call asynchronously
Each async option has a header form and a query-parameter form. They mean the same thing.- If the call finishes within the wait, you get the normal response.
- Otherwise you get
202 Acceptedwith arequest_id. - Without a wait, the call waits 25 s. A callback with no other async option waits 0 s, so the
202comes back at once. A callback sent withPrefer: respond-asyncorasync=trueand no wait keeps the 25 s. async_wait(orwait=) outside 0–120 is clamped to that range.- When you send both forms of the same option, the header wins.
- Put the query parameters in the query string, also on a
POST. Fields in a JSON body go to the endpoint and are not read as async options. - The async parameters are removed before the call reaches the endpoint, so they never change what the endpoint sees or what it costs.
X-Callback-URL used to need Prefer: respond-async and failed with 400 callback_requires_async without it. A callback now makes the call async by itself.The 202 response
Every async-capable endpoint answers202 in the same shape:
Location header with the poll URL and a Retry-After header in seconds.
Poll for the result
PollGET /v1/requests/{request_id} with any API key on the same account. Polling is free.
- While the call runs, it returns
202in the shape above. - When the call completes, it returns the call’s own status, headers and body, as if you had made the call synchronously.
- The result is held in memory only, never written to disk, for about 15 minutes after the call completes. After that, polling returns
404 request_not_found. - Wait
retry_after_secondsbetween polls.
client.requests.wait() honours Retry-After and raises a timeout error if the call isn’t done in time. To check once without waiting, use client.requests.get(request_id).
callback object in the 202 body, and as X-Callback-Status, X-Callback-Attempts and X-Callback-Error headers on the final response. For the full delivery state, use GET /v1/requests/{request_id}/callback.
Get the result by webhook
Add a callback URL to a call. When the call completes, ScrapeBadger POSTs the result to that URL as JSON, signed with your account’s webhook secret.202 at once. If you also set a wait and the call finishes within it, you get the normal response and the webhook as well.
The payload
status,headersandbodyare the call’s own, errors included. Header names are lower case.bodyis decoded JSON when the call returned JSON, and text otherwise.- A body over 1 MB is left out: you get
"body": nulland"body_omitted": true. Fetch it fromresult_urlbeforeresult_expires_at. batch_idis set when the call came from a batch.attemptandsent_at(Unix seconds) change on every attempt, and each attempt is signed again.
Callback URL rules
- The URL must be
https, on a public address, at most 2,048 characters long, with no user name or password in it. - Private, loopback, link-local and cloud-metadata addresses are refused. So are
localhostand names ending in.localor.internal. - Redirects are not followed.
- A bad URL gets
400 invalid_callback_urlbefore the call runs. Nothing is billed. - URL-encode the URL when you pass it as
callback_url.
Retries and duplicates
- Answer with any
2xxwithin 10 s. - On a timeout, a connection error,
408,425,429,500,502,503or504, the delivery is retried after 10 s, 60 s and 300 s: four attempts in total. Any other status is final. - Delivery is at least once. The same result can arrive more than once, for example when your server handled it but the response didn’t reach us in time. Deduplicate on
request_id. - Answer first and do the work after. A slow handler times out, and the result is sent again.
- Calls refused before they run (invalid key, no credits, rate limit) answer synchronously and send no webhook.
- Deliveries in progress don’t survive a restart of our gateway. If a restart interrupts retries, the delivery is reported as
failedwithlast_error: "interrupted (gateway restart)". Fetch the result fromresult_urlwhile it is still kept.
Verify the signature
X-ScrapeBadger-Signature is sha256= followed by the hex HMAC-SHA256 of the raw request body, keyed with your webhook secret. This is the same scheme stream monitors use.
- Compute the HMAC over the exact bytes you received, before any JSON parsing or re-encoding.
- Compare in constant time.
- Reject stale deliveries to stop replays: check that
sent_atis within a few minutes of now. Takesent_atfrom the body, which the signature covers. TheX-ScrapeBadger-Timestampheader has the same value but is not signed.
verify_signature and verifyWebhookSignature return true only when the signature matches and, when you pass sent_at, it is within tolerance_s (default 300 s) of now.
scrapebadger.webhooks.parse_event(raw_body) and parseWebhookEvent(rawBody) parse the payload into a typed event if you prefer that to plain JSON.
Receive webhooks
Both receivers read the raw body, verify it, skip repeats and answer before doing any work. Store therequest_ids you have handled somewhere shared, such as Redis or your database, and keep them for at least an hour.
400 or 401 from your receiver is final, so a delivery that fails verification is not retried. You can still fetch its result from result_url.
Your webhook secret
Every account has one webhook secret. It is created the first time you ask for it.- Rotate it with
POST /v1/account/webhook-secret/rotate, which returns the new secret. - The SDKs have
client.account.webhook_secret()andclient.account.rotate_webhook_secret()(Node:webhookSecret()androtateWebhookSecret()). - Both endpoints need a full-access API key. A key restricted to some APIs gets
403 full_access_required. - New deliveries use a rotated secret at once. A delivery that was already retrying keeps the secret it started with, for up to about 7 minutes. Accept both secrets for that long after you rotate.
- Keep the secret on your server. Anyone who has it can sign payloads that pass your check.
Check a delivery
Both endpoints are free.Check one delivery
GET /v1/requests/{request_id}/callback returns the delivery state of a request that was submitted with a callback. Other requests get 404 callback_not_found.
stateispending(not sent yet),delivering(being sent or waiting to retry),delivered(your server answered2xx) orfailed(a final status, or all four attempts used).last_statusis your server’s last HTTP status, andlast_errora short reason such asHTTP 503ortimeout.next_attempt_atanddelivered_atare Unix seconds.- In the SDKs:
client.requests.callback_status(request_id)(Node:callbackStatus(id)).
List recent deliveries
GET /v1/account/webhook-deliveries lists your account’s deliveries of the last 24 hours, newest first. Pass limit for up to 200 (default 50). It needs a full-access API key.
endpoint is the path only, without the query. In the SDKs: client.account.webhook_deliveries(limit=50) (Node: webhookDeliveries({ limit })).
Resend safely with an idempotency key
A submit can time out, or the connection can drop, before you see the202. Without a key, resending it starts, and bills, a second call. With a key, the resend returns the first call.
Send Idempotency-Key: <key> (or idempotency_key=<key>) with an async call. A key is 1–200 visible ASCII characters, with no spaces. Make it from your own job id, such as job-4815-try-1.
- Same key, same request: the call is not run again and not billed again. You get what
GET /v1/requests/{request_id}returns now:202while the first call runs, or its stored result. The response carriesIdempotent-Replayed: true. No second webhook is sent. - Same key, different request:
409 idempotency_key_reused. - Key on a synchronous call: ignored. The call runs as usual. Idempotency applies to async calls only (a callback counts as async), because synchronous results are never stored.
- “The same request” means the same method, path, query and body. The async options and the response-shaping parameters (
_fields,_jqand the others) don’t count, and neither do headers. - Keys belong to your account, so all your API keys share them. A key is kept for about 15 minutes. After that, the same key starts a new call.
- A replay returns the stored result, errors included. To retry a call that failed, use a new key, such as
job-4815-try-2.
Billing
- An async call is billed once, when it completes, at the endpoint’s normal price. The result’s
x-credits-usedheader says how much. - As with synchronous calls, only
2xxresults are billed. A call that never completes is not billed. - Submitting, polling, delivery status, the delivery list, the webhook secret and webhook deliveries (retries included) are free.
- A replay with an idempotency key is not billed.
- Resending the original request without an idempotency key starts, and bills, a second call.
Errors
Async errors answer before the call runs, so they are not billed. The body is{"error": "<code>", "detail": "<what to fix>"}.
These routes can’t run asynchronously. They answer
400 async_not_supported to async, async_wait, callback_url, idempotency_key and X-Callback-URL. A bare Prefer: respond-async or Idempotency-Key header is ignored there, so clients that send those on every call keep working.
/v1/requests/*,/v1/batch,/v1/quote,/v1/feedbackand/v1/results/*;/v1/account/*;- the Twitter stream routes under
/v1/twitter/stream/*: managing monitors, webhooks and filter rules, their logs, and the live stream itself; /v1/web/unblock/{job_id};- every call that changes something (
PUT,PATCH,DELETE, andPOSTother than the read endpoints such as the AIaskroutes and/v1/web/scrape).
Data retention
- Results (the response body and headers) are held in memory only, never written to disk, for about 15 minutes after the call completes.
- Idempotency keys are kept for about 15 minutes, as a hash of your key and a hash of the request. Neither the key nor the request is stored.
- Delivery metadata is kept for 24 hours: the callback host only (never the path or query, which can hold your tokens), the endpoint path only (never the query, which holds your parameters), status codes, attempt counts, short error reasons and timestamps.
- The webhook secret is kept with your account until you rotate it.
Submit a batch
POST /v1/batch takes up to 100 GET reads and returns a request_id for each, at once:
- Each item is an ordinary async request under your key. It has the same authentication, permissions, credit check, rate limit and price as if you had sent it alone. Submitting the batch is free.
- Items are checked against your balance before any of them start, at each endpoint’s base price (what
POST /v1/quotereports). Items whose price no longer fits, after the items before them and your other batch items still running, come back as402 insufficient_creditsand are not started. - An item that can’t start (for example
402as above,429over your rate limit, or404unknown endpoint) comes back with"accepted": false, itsstatusanderror. It is not billed, and you can resubmit just that item later. - Poll each
poll_url, or passcallback_urlto get one webhook per item, each withbatch_idset. - At most 10 items of one batch run at a time. The others wait their turn and show as
runningwhen polled. - A batch over 100 items gets
413 batch_too_large. - Items still waiting for a slot when our gateway restarts are dropped. They are not billed, and polling them returns
404 request_not_foundwithin about 5 minutes. Resubmit those.
Trial keys
Unclaimed trial keys can’t batch (403 trial_not_allowed) or use webhooks (trial_not_allowed). They can still run calls asynchronously and poll for them, one call at a time: a second call while the first is still running gets 429 trial_concurrency.
Concurrency
There is no per-key concurrency limit beyond your plan’s rate limit. AI asks render a live answer, so latency rises when many run at once. Start around 10 in parallel per engine and scale up while you watch the error rate. A502 on an AI ask means the engine produced no usable answer on that attempt. It is not billed, and retrying it is safe.
