DevOps · Kubernetes

Argo CD for GitOps — A Field Guide

How Argo CD's Application CRD, sync waves, App-of-Apps pattern, and ApplicationSet actually work in practice, and the RBAC and self-heal decisions that determine whether GitOps makes your cluster more predictable or just moves the chaos into Git.

John Kihiu12 min read

Argo CD's pitch is simple: the cluster's state should always match what's declared in a Git repository, and if it drifts, Argo CD either tells you or fixes it. The mechanics that make this work — the Application CRD, sync waves, App-of-Apps, ApplicationSet — are worth understanding individually, because most of the pain teams hit with GitOps comes from treating Argo CD as a black box that "just syncs YAML" instead of a controller with real ordering and RBAC semantics.

The Application CRD is the whole model

Everything in Argo CD starts with the Application custom resource: a pointer to a Git repo path (or Helm chart, or Kustomize overlay) as the source of truth, and a pointer to a cluster and namespace as the destination. Argo CD's controller continuously diffs the live state of that destination against the rendered manifests from the source, and reports the result as one of two independent states — Synced/OutOfSync for whether live matches desired, and Healthy/Degraded/Progressing for whether the resources are actually working. Those two axes are easy to conflate but matter separately: a Deployment can be perfectly in sync with Git and still be Degraded because the image tag it's pinned to doesn't exist.

YAML · APPLICATION CRD
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: billing-api
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://github.com/acme/billing-api-manifests.git
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: billing
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Sync waves and hooks control ordering

A raw kubectl apply -f across a directory of manifests applies everything roughly at once, which breaks the moment one resource genuinely depends on another being ready first — a CRD before the resources that use it, a database migration Job before the Deployment that assumes the schema is current. Argo CD's answer is the argocd.argoproj.io/sync-wave annotation: resources are grouped by wave number and applied lowest-to-highest, waiting for each wave's resources to reach a healthy state before starting the next. Combined with sync hooks (PreSync, Sync, PostSync, SyncFail), you get a real deployment pipeline expressed entirely as annotations on standard Kubernetes objects, no separate pipeline tool required for ordering.

Sync waves are strings, not defaults

Resources with no sync-wave annotation default to wave "0". If you're introducing waves into an existing app, make sure resources you actually need to run first get a negative wave number rather than assuming unannotated resources will politely wait their turn — they won't.

App-of-Apps and ApplicationSet for scale

A single cluster with a handful of services can be managed with individual Application resources created by hand. That stops working once you have multiple environments, multiple clusters, or dozens of services — the App-of-Apps pattern solves the first problem by making an Application's source just be a directory of other Application manifests, so one root app bootstraps everything else recursively. ApplicationSet goes further and solves the templating problem directly: a single ApplicationSet resource with a generator (list, cluster, Git directory, matrix) stamps out one Application per entry, so adding a new environment or a new cluster is a data change, not a copy-pasted YAML file.

YAML · APPLICATIONSET
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: billing-api-envs
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - env: staging
            cluster: https://staging.k8s.internal
          - env: production
            cluster: https://prod.k8s.internal
  template:
    metadata:
      name: 'billing-api-{{env}}'
    spec:
      project: platform
      source:
        repoURL: https://github.com/acme/billing-api-manifests.git
        targetRevision: main
        path: 'overlays/{{env}}'
      destination:
        server: '{{cluster}}'
        namespace: billing

Self-heal and automated sync are a real decision, not a default

Turning on automated.selfHeal means Argo CD will actively revert manual changes made directly against the cluster — someone runs kubectl scale or kubectl edit under pressure during an incident, and Argo CD quietly puts it back the way Git says it should be, sometimes seconds later. That's exactly the point of GitOps, but it surprises people the first time it happens to them mid-incident, and it means your actual emergency lever is "commit to Git and let it sync" or "pause auto-sync on the Application," not "kubectl edit and sort out Git later." Teams that skip this conversation find out about self-heal the hard way, usually at 2 AM.

RBAC lives in AppProjects, not just Applications

An AppProject is what actually constrains which repos, which destination clusters/namespaces, and which resource kinds an Application in that project is allowed to touch. If every team's Applications sit in the default project, there's effectively no isolation — a typo'd destination.namespace can sync into a namespace it should never have access to. Scope AppProjects per team or per environment before you scale past a handful of Applications.

The diff you actually need to trust

Argo CD's diff between live and desired state is what everything else depends on, and it's not always a naive field-by-field comparison — server-side apply, mutating webhooks, and fields defaulted by the API server (like some values under status or controller-managed annotations) can all make live state differ from Git in ways that are expected, not drift. Getting comfortable reading the diff view and configuring ignoreDifferences for fields that are legitimately mutated outside Git (HPA-managed replica counts are the classic example) is what keeps "OutOfSync" from becoming background noise that people learn to ignore, which defeats the entire premise of GitOps.

Wrapping up

Argo CD's value isn't the sync button, it's the model underneath: Application as the unit of desired state, sync waves for ordering that would otherwise need a separate pipeline tool, ApplicationSet for scaling that model across environments and clusters without copy-pasting YAML, and AppProject-scoped RBAC so "who can touch what" isn't left implicit. Self-heal is the feature that makes GitOps real instead of aspirational — decide deliberately whether you want it on, because the first time it reverts someone's emergency kubectl edit should not be a surprise.

John Kihiu
Acumatica ERP Developer · Laravel Engineer

Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.