An automation workflow is a chain of calls across services you do not control, and at least one of them will be briefly unavailable during any given run. Without retries, a two-second network blip fails an entire multi-step process. With careless retries, that same blip re-runs steps that were never meant to run twice. The strategy is retries built on idempotent steps.
Make steps idempotent first
Before adding a single retry, ask what happens if a step runs twice — because a retry guarantees some step eventually will. If "create invoice" is not idempotent, a retry after a timeout that actually succeeded creates a duplicate invoice. Use upserts keyed on a business id, and pass an idempotency key to any downstream API that accepts one. Retries are only safe on top of idempotent steps; get that order wrong and retries make things worse.
Back off, with jitter
Retry with exponential backoff so a struggling downstream gets room to recover, and add jitter so many workflows failing at once do not retry in lockstep and re-cause the outage. A fixed one-second retry loop against a rate-limited API is just a slower way to stay rate-limited.
Transient vs permanent
| Failure | Action |
|---|---|
| Timeout, 429, 5xx, connection reset | Retry with backoff |
| 400 / 422 bad input | Do not retry — fix the data, dead-letter |
| 401 / 403 auth | Do not retry — fix credentials, alert |
| Business rule rejection | Route to human review, not a retry |
Retrying a permanent failure just delays the moment you admit it failed. Classify first: only transient, infrastructure-level failures are worth a retry; a malformed payload will be malformed on every attempt.
Every retry needs a limit — attempts and total time. When the budget runs out, move the run to a dead-letter queue with its full context, not into an infinite loop. A visible, replayable failed run is recoverable; a workflow silently retrying forever is a resource leak nobody notices until the queue is full.
Reliable workflow retries are a short recipe: idempotent steps underneath, exponential backoff with jitter, a transient/permanent split before every retry, and a hard cap that dead-letters what cannot recover. That turns the inevitable flaky-dependency moments into brief pauses instead of failed business processes.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.