Because webhooks are delivered at-least-once, a consumer that does real work needs to recognise a repeat and drop it. Deduplication is that recognition step — distinct from, and usually paired with, idempotent processing. Dedupe stops the duplicate cheaply at the door; idempotency is the safety net for when it slips through anyway.
Choosing the dedupe store
A dedupe store answers one question fast: have I seen this event id before? The right backing depends on volume and durability needs:
| Store | Good for |
|---|---|
| Redis with TTL | High volume, dedupe only needed within the retry window; fast and self-evicting. |
| Relational unique key | Lower volume, when you want dedupe and an audit trail in one durable place. |
| Bloom filter + backing store | Very high volume where a probabilistic pre-check saves lookups. |
Size the window, then evict
You do not need to remember every event forever — only long enough to cover the provider's retry window. If a provider retries for 24 hours, a dedupe entry needs to outlive that, with margin. Set a TTL a comfortable multiple of the retry window so late duplicates are still caught, and let the store evict on its own rather than growing without bound. Keeping ids forever is a slow storage leak solving a problem you do not have.
Late and out-of-order duplicates
Retries do not arrive politely. A duplicate can land minutes after the original, and updates to the same resource can arrive out of order. Dedupe handles the exact repeat; for out-of-order updates, compare a version or timestamp on the payload and ignore an event older than the state you already hold. That is a different check from id-dedupe, and reliable consumers do both.
Put the dedupe check at the very front of the handler, before parsing, business logic, or downstream calls. The whole value is spending a single fast lookup to avoid the entire cost of reprocessing. A dedupe check buried after the work has already run is just an audit log.
Effective deduplication is a fast front-door lookup against a store sized to your retry window, backed by idempotent processing for anything that gets past it. Get both in place and the at-least-once firehose becomes a clean, exactly-once stream your business logic can trust.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.