Prefect for ERP Workflows

Most ERP integrations start life as a cron job: a script that runs at 2 AM, pulls sales orders or inventory levels from the ERP's API, and writes them into a warehouse or a downstream system. That works fine until the ERP API times out on one run out of fifty, or a schema change upstream silently breaks the transform step, or someone needs last Tuesday's data re-run because the ERP was down that night. Cron has no answer to any of that beyond "check the logs and re-run it by hand." An orchestrator like Prefect isn't a fundamentally different architecture — it's the same extract-transform-load shape — but it gives you retries, observability, and backfills as first-class features instead of things you bolt on yourself.

Why cron stops being enough

ERP APIs are usually the least reliable part of a sync pipeline. They're built for the ERP's own UI first and integrations second, rate limits are inconsistently documented, and a nightly batch job hitting them at scale will occasionally get a timeout or a 500 that has nothing to do with your code. A bare cron job either crashes and waits for someone to notice, or — worse — swallows the error and silently skips a day's data. Neither is a design decision anyone made on purpose; it's just what happens by default when the orchestration logic is a bash script and a crontab entry.

The other thing cron doesn't give you is dependency ordering. A sync pipeline is rarely one step — it's extract from the ERP, transform into the target schema, then load — and those steps genuinely depend on each other. Cron can schedule three separate jobs five minutes apart and hope the timing holds; it can't express "don't run the load step if the extract step failed."

Retries with backoff on a flaky API

The single highest-value thing an orchestrator buys you here is retry policy as configuration instead of hand-rolled logic. A transient 503 from the ERP shouldn't fail the whole nightly sync — it should back off and try again, and only page someone if it's still failing after a few attempts.

from prefect import flow, task
from datetime import date, timedelta

@task(retries=4, retry_delay_seconds=[10, 30, 60, 120])
def extract_sales_orders(sync_date: date) -> list[dict]:
    # calls the ERP's REST API for orders modified on sync_date;
    # a transient timeout or 5xx here should not kill the whole run
    ...

@task
def transform_orders(raw_orders: list[dict]) -> list[dict]:
    ...

@task(retries=2)
def load_to_warehouse(orders: list[dict], sync_date: date) -> None:
    ...

@flow(name="erp-sales-order-sync", log_prints=True)
def erp_sales_order_sync(sync_date: date = None):
    sync_date = sync_date or date.today() - timedelta(days=1)
    raw = extract_sales_orders(sync_date)
    clean = transform_orders(raw)
    load_to_warehouse(clean, sync_date)

if __name__ == "__main__":
    erp_sales_order_sync()

The transform_orders task only runs once extract_sales_orders has returned successfully, and load_to_warehouse only runs once transform has — that ordering falls out of the plain Python data dependency, no separate DAG config required. If extraction fails after all its retries, the flow run is marked failed with the actual exception attached, and the load step never fires against partial or missing data.

Observability into which run failed and why

With a cron job, "did last night's sync work" means SSH-ing into a box and grepping a log file, if the log file wasn't rotated away already. With Prefect, every flow run has a state, a duration, and the actual traceback of whatever task failed, visible in one place. When a sync fails at 2 AM, the question in the morning isn't "did it run" — it's "which task failed, on which sync_date, with what error," and that's answered in a few clicks instead of a log spelunking exercise.

This matters more than it sounds like it should, because ERP sync failures are rarely all-or-nothing. A run might succeed for 900 of 950 orders and fail on one malformed record. Task-level state means you can see exactly which task and which input caused the failure, instead of treating the whole night's sync as a binary success or failure.

Backfills without rerunning everything

ERP outages, downstream schema changes, and bad data all eventually require re-running the sync for a specific past date, not the whole history. Because the flow above takes sync_date as a parameter, a backfill is just calling the same flow with a different date — or a range of dates — rather than writing a one-off script that duplicates the extract/transform/load logic.

Dependency ordering between extraction and transformation

Real ERP sync pipelines are rarely a single flow — inventory sync, sales order sync, and customer sync often run on different schedules but sometimes need to respect ordering between them (customer records should land before the sales orders that reference them, for instance). Prefect handles the in-flow ordering naturally through task dependencies, and cross-flow ordering through deployment scheduling or by having one flow call another as a subflow when the dependency is tight enough to require it. The point either way is that the ordering is explicit in code, not an assumption about how long each cron job takes to run.

Wrapping up

None of this requires abandoning the basic extract-transform-load shape that a cron job already has — an ERP sync pipeline built on Prefect is doing the same work a bash script and crontab entry would do. What changes is that retries, failure visibility, and backfills stop being things you build yourself under pressure during an incident, and become configuration you set once. For a pipeline pulling from an API you don't control and can't make more reliable, that's the actual value: less time spent reconstructing what happened after a 2 AM failure, more confidence that a bad night doesn't quietly become a week of missing data.