AWS SQS for ERP Event-Driven Workflows

Most ERP integrations start as direct calls: system A calls system B's API, waits for a response, and fails the whole operation if B is slow or down. That works until B has a bad day, or a batch of 500 orders needs to sync at once and B can only handle 20 requests a second. AWS SQS fixes this by putting a durable queue between the ERP and whatever consumes its events — order created, invoice posted, inventory adjusted. The producer drops a message and moves on; the consumer pulls messages at its own pace and retries on failure without the producer ever knowing there was a problem.

Why a queue instead of a direct call

A direct HTTP call couples the availability of two systems together. If the downstream warehouse system is deploying, restarting, or just slow, the ERP either blocks or drops the event. A queue decouples that: SQS holds the message durably (by default, messages are retained up to 4 days, configurable up to 14) until a consumer is ready. The ERP's job becomes "publish the event reliably," not "guarantee the downstream system processed it."

This also absorbs bursts. An end-of-day batch job that pushes 10,000 line-item events doesn't need the consumer to keep up in real time — it needs the queue to hold them until the consumer works through the backlog.

Sending and receiving messages

Here's a minimal producer and consumer using boto3. The producer runs inside (or next to) the ERP's event hooks; the consumer is a small worker process.

import json
import boto3

sqs = boto3.client("sqs", region_name="us-east-1")
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/erp-order-events"

def publish_order_created(order):
    sqs.send_message(
        QueueUrl=QUEUE_URL,
        MessageBody=json.dumps({
            "event": "order.created",
            "order_id": order["id"],
            "total": order["total"],
        }),
        MessageAttributes={
            "eventType": {"DataType": "String", "StringValue": "order.created"}
        },
    )

def consume_orders():
    while True:
        resp = sqs.receive_message(
            QueueUrl=QUEUE_URL,
            MaxNumberOfMessages=10,
            WaitTimeSeconds=20,        # long polling
            VisibilityTimeout=30,      # hide message while we process it
        )
        for msg in resp.get("Messages", []):
            body = json.loads(msg["Body"])
            try:
                handle_order_event(body)
                sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=msg["ReceiptHandle"])
            except Exception:
                # Don't delete — let it become visible again and retry,
                # or land in the DLQ after maxReceiveCount is hit.
                pass

def handle_order_event(event):
    print(f"syncing order {event['order_id']} to warehouse system")

Long polling (WaitTimeSeconds=20) matters: without it, the consumer hammers SQS with empty receive_message calls. The visibility timeout is the window during which a received-but-not-deleted message is hidden from other consumers — if processing throws, the message reappears in the queue after that timeout and gets retried.

Retries and the dead-letter queue

Retrying forever is its own failure mode — a malformed event that always throws will loop indefinitely, burning consumer capacity and hiding real problems in noise. SQS's redrive policy caps retries: after maxReceiveCount failed receives, the message moves to a separate dead-letter queue (DLQ) instead of going back to the main queue.

Resources:
  OrderEventsDLQ:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: erp-order-events-dlq
      MessageRetentionPeriod: 1209600  # 14 days

  OrderEventsQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: erp-order-events
      VisibilityTimeout: 30
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt OrderEventsDLQ.Arn
        maxReceiveCount: 5

Ordering and duplicate delivery

Standard SQS queues don't guarantee order and can deliver a message more than once. For most ERP events that's fine if the consumer is idempotent — upserting an order by its ID rather than blindly inserting handles duplicate delivery safely. If strict per-entity ordering actually matters (for example, "created" must be processed before "cancelled" for the same order), use a FIFO queue with a MessageGroupId set to the order ID: messages within the same group are delivered in order, while different groups still process in parallel.

Don't reach for FIFO by default — it caps throughput per message group and adds constraints you don't need if your consumer is already idempotent. Start with a standard queue and an idempotent handler; add FIFO only when you've confirmed ordering is a real requirement, not a hypothetical one.

Sizing visibility timeout and retention

Two settings decide most of your queue's behavior in production. Visibility timeout should be comfortably longer than your worst-case processing time — if it's too short, a slow-but-successful job gets treated as failed and reprocessed, potentially causing duplicate side effects downstream. Message retention should cover your longest expected outage: if the downstream system might be down for a weekend deploy, a 4-day default retention could drop events before anyone notices.

Wrapping up

The value of SQS in an ERP integration isn't the queue itself — it's what the queue lets you stop worrying about. The producer no longer needs the consumer to be up, fast, or even reachable at the moment an event fires. Retries, visibility timeouts, and the DLQ turn "the integration silently dropped an event" into "there are 12 messages sitting in a dead-letter queue with an alarm attached," which is a problem you can actually see and fix.