Kubernetes Spot Instances — A Field Guide

Spot and preemptible instances cost 60-90% less than on-demand, and Kubernetes is one of the few platforms where that discount is actually usable in production, because the scheduler already assumes pods can be rescheduled elsewhere. The catch is that "the cloud provider can take this node back with roughly two minutes' notice" is a real constraint, not a theoretical one, and workloads that aren't built to tolerate it will have a bad time the first time a large batch of spot capacity gets reclaimed at once.

Which workloads actually belong on spot

The rule of thumb is: anything stateless, horizontally scaled, and tolerant of a pod restarting on a different node is a good spot candidate — web frontends, stateless APIs behind a load balancer, batch processing, CI runners, and most machine learning training jobs that checkpoint periodically. Anything that's a StatefulSet with local storage, a single-replica database, or a workload with a long-running in-memory computation that can't checkpoint is a poor fit — the interruption cost is either data loss or a slow, expensive restart. Mixed node pools, where stateless workloads run on spot and stateful ones stay on on-demand, is the standard shape rather than trying to force everything onto one pool type.

Taints, tolerations, and separate node pools

The standard setup is a dedicated spot node pool, tainted so that only pods explicitly tolerating it get scheduled there. This prevents a scheduler accident from placing your stateful database on a node that can vanish in two minutes. Pair the taint with a nodeAffinity or a simple node selector so spot-tolerant workloads actively prefer that pool instead of merely being allowed onto it.

spec:
  template:
    spec:
      tolerations:
        - key: "cloud.google.com/gke-spot"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: cloud.google.com/gke-spot
                    operator: In
                    values: ["true"]
      terminationGracePeriodSeconds: 25

Handling the interruption notice

Every cloud provider sends some form of interruption warning before reclaiming a spot node — AWS gives roughly a two-minute notice via the instance metadata endpoint, GCP gives about 30 seconds for preemptible/spot VMs, Azure similarly gives a short eviction notice. Kubernetes itself doesn't read these signals natively; you need a node-termination handler running as a DaemonSet (AWS Node Termination Handler, or the cloud-specific equivalent) that watches for the notice and cordons plus drains the node before the provider yanks it, so pods get a clean reschedule instead of an abrupt kill. Without this, pods still get rescheduled eventually — kubelet and the node lease will eventually mark the node NotReady — but you lose the grace period and clean shutdown hooks won't run.

PodDisruptionBudgets and instance-type diversification

A PodDisruptionBudget doesn't stop spot reclamation — that's an involuntary disruption the cloud provider initiates outside Kubernetes' control, and PDBs only govern voluntary evictions like node drains and cluster-autoscaler scale-downs. What actually helps with spot availability is diversifying instance types and availability zones in the node pool's autoscaling config, so a capacity shortage in one instance type or zone doesn't take out your entire spot fleet simultaneously. Cluster Autoscaler and Karpenter both support multi-instance-type node pools for exactly this reason.

Capacity gaps and the on-demand fallback

Spot capacity isn't guaranteed to exist — a request for spot nodes can simply fail if the cloud provider has none available at that price in that zone. Production setups that depend on spot for anything latency-sensitive need a fallback: either a small baseline of on-demand nodes that's always present, or a node pool priority scheme (via Karpenter or Cluster Autoscaler priority expander) that falls back to on-demand automatically when spot is unavailable, rather than leaving pods unschedulable.

Wrapping up

Spot instances work well in Kubernetes because the platform's rescheduling model already matches the constraint spot imposes — the work is making sure your workloads actually tolerate that constraint instead of assuming it away. Taint the spot node pool, run a termination handler that reacts to the interruption notice, keep grace periods inside the notice window, diversify instance types to reduce simultaneous capacity loss, and keep a small on-demand fallback for when spot capacity just isn't there.