Webhook Retry Queues, Backoff and Dead Letters Explained
A practical failure-handling design for lead delivery with bounded retries, jitter, observability and safe manual recovery.
What happens after an outbound delivery fails?
An accepted lead stays stored. Each destination receives bounded attempts; exhaustion leads to investigation rather than silent loss.
- 1Persist the accepted lead independently of downstream APIs.
- 2Attempt the configured destination and record its response.
- 3Retry eligible transient failures with bounded backoff and idempotency.
- 4Investigate exhausted retries without duplicating previous successful deliveries.
Illustrative workflow: specific timeout, retry and dead-letter handling depends on the service and receiver contract.
A queue solves one problem, not every problem
The reason to put outbound delivery into a queue is to stop a slow destination from blocking intake. Once the accepted lead is safely stored, a separate worker can attempt delivery. However, queues do not guarantee the receiver is healthy, and messages may be attempted more than once.
Design a delivery as a state machine, not a Boolean flag: pending, attempted, delivered, waiting to retry or requiring investigation. A durable audit trail should explain the current state of each destination independently.
Retry only problems that might recover
HTTP codes alone are not a complete contract. Some providers have their own retryable error codes or response bodies. Follow the receiving API specification instead of assuming every non-2xx response deserves an identical retry.
| Outcome | Typical next step |
|---|---|
| Timeout / connection reset | Retry with backoff; status unknown |
| 429 Too Many Requests | Honor provider guidance and retry limits |
| 5xx server failure | Retry within a bounded budget |
| 400 invalid schema | Fix mapping, do not loop blindly |
| 401/403 authorization | Inspect credentials and access first |
| 2xx acknowledgment | Record delivered; prevent duplicate effects |
Backoff with jitter avoids synchronized storms
A common pattern is a capped exponential delay with random jitter. The cap bounds waiting time between attempts; an overall attempt or age budget stops an unrecoverable delivery from retrying forever. Treat the numbers below as a general example, not a Panther Hook configuration promise.
function nextRetryDelayMs(attempt) {
const safeAttempt = Math.max(0, Math.min(attempt, 8));
const cap = Math.min(60_000, 1000 * 2 ** safeAttempt);
return Math.floor(Math.random() * cap); // Full jitter
}
// Also enforce a maximum attempt count and total retry age.A dead-letter queue is an investigation queue
After the configured budget is exhausted, keep a traceable failed delivery for inspection instead of silently dropping it. Operators need the destination, request correlation ID, timestamp, failure category and retry history, but should not expose sensitive payloads in log messages.
Manual replay must respect idempotency and permissions. Replaying a lead to one failed destination should not accidentally resend every other successful notification.
- Monitor queue age, pending depth and repeated failures.
- Alert on stalled workers separately from unhealthy destinations.
- Define retention and access control for failed events.
- Document how to fix a configuration error before replaying.