Retries sit on both sides of a webhook. As a sender, you retry delivery when the consumer does not acknowledge. As a consumer, you retry your own downstream calls when processing an event hits a transient failure. Both need the same discipline: back off, add jitter, cap the attempts, and know which failures are even worth retrying.
Exponential backoff with jitter
Retrying immediately, or on a fixed interval, hammers a system that is probably already struggling. Exponential backoff spaces attempts out — 1s, 2s, 4s, 8s — giving the far end time to recover. The jitter is not optional: without a random component, everything that failed at the same moment retries at the same moment, and you recreate the outage. Add randomness so the retries spread.
import random, time
def backoff_delays(base=1, cap=300, tries=8):
for attempt in range(tries):
yield min(cap, base * 2 ** attempt) * (0.5 + random.random())Retry only what is retryable
A retry only helps a transient failure. Retrying a permanent one wastes effort and delays the inevitable:
- Retryable — timeouts, connection resets, 429s, 5xx. Back off and try again.
- Not retryable — 400/422 (malformed payload), 401/403 (auth). The same request will fail identically; send it straight to the dead-letter queue.
Cap attempts and total time
Retries need a hard end. Cap both the number of attempts and the total elapsed time, because an event retried forever is a queue that never drains and a resource that is never freed. When the budget is exhausted, stop and route the event to a dead-letter queue for inspection or manual replay — a failure you can see and act on, rather than one buried in an infinite loop.
Do not do slow processing inside the webhook request while the sender waits. Validate, enqueue, and return 2xx immediately, then process — and retry — from your own queue. Holding the connection open through your retries risks the sender timing out and retrying the whole delivery, multiplying the work.
The reliable pattern is the same on both sides: classify the failure, back off exponentially with jitter, cap attempts and time, and dead-letter what is left. That turns transient failures into brief delays and permanent ones into visible, recoverable records — instead of a retry storm that outlasts the outage that started it.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.