Things go wrong: a deploy ships a handler bug, a downstream dependency is down for an hour, a consumer silently drops events. When that happens you need to re-process the affected events — replay them. The catch is that replay is only possible if you designed for it beforehand, by keeping the raw events and making processing safe to repeat.
Persist the raw event first
The foundation of replay is a durable record of every event exactly as it arrived — full payload, headers, received timestamp — written before you process it. If you only store the results of processing, a processing bug means the correct input is gone and there is nothing to replay. Treat the inbound event log as the source of truth and processing as a derivation from it.
CREATE TABLE webhook_events (
event_id TEXT PRIMARY KEY,
source TEXT NOT NULL,
payload JSONB NOT NULL,
received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
status TEXT NOT NULL DEFAULT 'received', -- received | processed | failed
processed_at TIMESTAMPTZ
);Replay from the DLQ or the log
There are two replay sources for two situations. The dead-letter queue holds events that failed processing — replay those once you have fixed the bug or the dependency recovers. The full event log lets you replay a broader slice — every event for a resource, or everything in a time window — which is what you need when a handler was silently wrong and the events never even reached the DLQ.
Scope and control the replay
- By time — replay everything received between the deploy that broke and the fix.
- By resource — replay all events for the specific customers or orders that were affected.
- By status — replay only events still marked failed, leaving the good ones untouched.
A replay re-delivers events you may have already partly processed, so without idempotent handling you will double-charge and duplicate on the way to fixing the original problem. Idempotency is what makes replay a safe recovery tool instead of a second incident. Never replay into a handler you are not certain is idempotent.
A replay capability is cheap insurance you build before you need it: store raw events durably, track their processing status, keep handlers idempotent, and give yourself a scoped, controllable way to re-run them. When the inevitable bad deploy or outage arrives, recovery is a targeted replay instead of a data-repair archaeology project.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.