Argo Workflows — A Complete Guide

Argo Workflows is a Kubernetes-native workflow engine that implements each step of a pipeline as a pod, orchestrated through a custom resource called a Workflow. Where a generic CI tool runs steps on shared runners, Argo runs each step in its own container on the cluster, which makes it a natural fit for multi-stage data pipelines, ML training runs, and CI/CD pipelines that already live in Kubernetes. It is one of the four Argo projects (alongside Argo CD, Argo Rollouts, and Argo Events) under the CNCF umbrella.

Core concepts: Workflows, templates, DAGs

A Workflow is a custom resource whose spec lists one or more templates — each template is a unit of work, typically a container step, but it can also be a script, a resource template that applies Kubernetes manifests, or a suspend step that pauses for manual approval. Steps are wired together either as a linear steps list or as a dag with explicit dependencies between tasks, which is the more common choice once a pipeline has any branching or parallel fan-out.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: etl-pipeline-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: extract
            template: extract-step
          - name: transform
            template: transform-step
            dependencies: [extract]
          - name: load
            template: load-step
            dependencies: [transform]
    - name: extract-step
      container:
        image: myrepo/etl:1.4.0
        command: [python, extract.py]

Artifacts and passing data between steps

Because each step runs in its own pod, data does not flow through shared memory or a filesystem by default — it has to be passed explicitly as an artifact or a parameter. Argo integrates with an artifact repository (S3, GCS, Azure Blob, or an in-cluster MinIO) so a step can output a file and a downstream step can declare it as an input; the controller handles the upload and download transparently. Parameters work the same way for small values like a computed count or a status string, passed via outputs.parameters and consumed with {{tasks.extract.outputs.parameters.count}}.

Retries, timeouts, and exit handlers

Production pipelines fail in the ordinary ways distributed systems fail: a pod gets evicted, an external API rate-limits a request, a node drains mid-step. Argo's retryStrategy lets you set a backoff policy per template rather than per workflow, so a flaky network call can retry three times with exponential backoff while a step that should never be retried (one with a side effect like sending an email) stays retry-free. An onExit handler runs regardless of whether the workflow succeeded or failed, which is where cleanup and notification steps belong.

Argo Events for triggering workflows

Argo Workflows on its own only runs what you submit manually or via the API. Argo Events adds event sources (webhooks, message queues, S3 bucket notifications, cron schedules) and sensors that trigger a workflow when a condition matches — the common pattern is an S3 sensor that kicks off an ETL workflow the moment a new file lands in a bucket, replacing a polling cron job with an event-driven trigger.

Resource limits and cluster cost

Every step is a pod, which means every step should carry its own resource requests and limits — omit them and the scheduler either over-packs the node or the workflow controller can't make good bin-packing decisions. For workflows with high fan-out (hundreds of parallel tasks), set parallelism at the workflow or template level to cap concurrent pods, and enable pod garbage collection (podGC) so completed step pods don't linger and consume etcd storage.

Wrapping up

Argo Workflows earns its complexity when a pipeline is already Kubernetes-native and needs per-step isolation, artifact passing, and DAG-level retries that a generic CI runner does not give you cleanly. For a simple linear build-test-deploy pipeline, a standard CI tool is usually less operational overhead — reach for Argo when the workflow has real branching, needs to scale out parallel steps independently, or should react to events rather than run on a fixed schedule.