Buildkite Patterns — A Field Guide

Buildkite takes a different approach from most hosted CI: the pipeline UI, web dashboard, and scheduling live in Buildkite's cloud, but the actual build agents run wherever you point them — your own VMs, your Kubernetes cluster, spot instances, whatever you already operate. That split is the whole story of Buildkite patterns: you get the orchestration convenience of a hosted service without giving up control of where code executes, which matters a lot once you have compliance requirements, GPU builds, or a cost-sensitive fleet of self-hosted runners.

Pipeline YAML and dynamic pipelines

A Buildkite pipeline is defined in YAML uploaded either from a file in the repo or generated dynamically by a script at runtime. Static YAML is fine for simple pipelines, but the dynamic upload pattern — a small script that inspects the repo and emits pipeline steps as JSON or YAML to stdout — is what lets you do things like generating one test-shard step per changed package, or skipping a deploy step entirely when only docs changed. This is the single biggest practical difference from GitHub Actions' more static workflow model.

steps:
  - label: ":mag: Detect changed packages"
    command: "scripts/generate-pipeline.sh | buildkite-agent pipeline upload"

  - wait

  - label: ":hammer: Build"
    command: "make build"
    agents:
      queue: "default"

  - label: ":test_tube: Test (parallel)"
    command: "make test"
    parallelism: 8
    agents:
      queue: "default"

  - block: ":rocket: Deploy to production?"
    branches: "main"

  - label: ":rocket: Deploy"
    command: "scripts/deploy.sh"
    agents:
      queue: "deploy"

Agent queues and self-hosted runners

Agents register against one or more queues, and steps target a queue with the agents: { queue: "..." } key. This is the mechanism for routing GPU-heavy ML training steps to a GPU queue, deploy steps to a queue with production credentials, and everything else to a general-purpose autoscaling queue. Because you run the agents, autoscaling is your responsibility — the common pattern is an agent metrics-based autoscaler (Buildkite publishes an official AWS autoscaling stack) that watches the number of scheduled jobs and adds or removes agent instances accordingly.

The block step for manual gates

The block step pauses a pipeline until someone approves it in the UI, which is the standard way to gate production deploys without building a separate approval system. Combined with branch filters, you can require manual approval only on main while feature branches run straight through to a preview environment unattended.

Secrets and credential scoping

Buildkite itself doesn't store your build secrets — the recommended pattern is pulling them at runtime from a secrets manager (AWS Secrets Manager, Vault, GCP Secret Manager) scoped to the agent's IAM role, so credentials never live in pipeline YAML or environment hooks committed to a repo. Scope deploy credentials to the deploy queue specifically; an agent that only ever runs test steps has no business holding production deploy keys.

Wrapping up

Buildkite's value is in the seam between hosted orchestration and self-hosted execution — use dynamic pipeline generation to keep YAML from becoming unmanageable, route steps to queues by what credentials or hardware they need, and keep agents ephemeral so a bad build never contaminates the next one.