Workflow Dead-Letter Queue Patterns

When a workflow run has exhausted its retries or hit a failure that retrying cannot fix, it needs somewhere to go. A dead-letter queue (DLQ) is that place: a holding area for runs that could not complete, kept with enough context to diagnose and, once fixed, replay. The alternative to a DLQ is a run that either retries endlessly or disappears — and you rarely find out which until data is missing.

Route with full context

A DLQ entry is only useful if it carries everything needed to understand the failure without re-running it: the original input payload, which step failed, the full error, the retry history, and a timestamp. A dead-letter that holds only "failed" is a graveyard; one that holds the complete run context is a recovery tool. Persist the raw input especially — you cannot replay what you did not keep.

CREATE TABLE workflow_dlq (
  id           BIGSERIAL PRIMARY KEY,
  workflow     TEXT NOT NULL,
  run_id       TEXT NOT NULL,
  failed_step  TEXT NOT NULL,
  input        JSONB NOT NULL,        -- the payload to replay
  error        TEXT NOT NULL,
  attempts     INT  NOT NULL,
  failed_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  status       TEXT NOT NULL DEFAULT 'pending'  -- pending | replayed | discarded
);

Watch the depth

DLQ depth is one of the best health signals a workflow system has. An empty or steady DLQ means things are flowing; a rising one means something broke and runs are piling up. Alert on the depth and on the rate of new entries, and treat a growing DLQ as an incident — because every entry is a business process that did not complete.

Inspect and replay

The DLQ is a starting point for recovery, not a final resting place. Give yourself tooling to inspect entries, fix the root cause — bad data, an expired credential, a downstream that is back up — and replay runs individually or in bulk. Replay leans on idempotency: you are re-running steps that may have partly executed, so the workflow must be safe to repeat. After a successful replay, mark the entry resolved so the queue reflects reality.

A dead-letter queue turns unrecoverable workflow failures from silent losses into visible, contextual, replayable records. Route exhausted runs there with full context, alarm on depth, and give the queue an owner who inspects, fixes, and replays — so nothing that fails simply disappears.