Chaos engineering is the practice of deliberately injecting failure into a running system to verify it degrades the way you think it does — before an unplanned outage forces the experiment on you at 3 a.m. It only works as a discipline: a hypothesis, a small blast radius, a way to abort, and a review of what the system actually did versus what the runbook assumed.
Start with a hypothesis, not a tool
Every experiment should read like a sentence: "If we kill one instance of the checkout service, p99 latency stays under 400ms and no requests are dropped, because the load balancer removes unhealthy targets within 10 seconds." If you cannot write that sentence, you are not ready to run the experiment — you are just breaking things and hoping to learn something useful. The hypothesis also gives you your success criteria up front, so the retro is not a debate about whether the result was "bad."
Fault injection techniques
The common categories are resource exhaustion (CPU, memory, disk pressure), network faults (latency, packet loss, DNS failures, blackholing a dependency), and process-level faults (killing a pod, terminating an instance, forcing a leader election). Tools like Chaos Mesh and LitmusChaos operate at the Kubernetes level and can inject network delay or kill pods on a schedule; AWS Fault Injection Simulator does the same against EC2, ECS, and RDS from inside AWS's own control plane. Netflix's original Chaos Monkey — the one that popularized the term — simply terminated random production instances during business hours, on the theory that if you cannot survive losing an instance at 2pm, you will not survive it at 2am either.
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: checkout-latency
namespace: prod
spec:
action: delay
mode: fixed-percent
value: "25"
selector:
namespaces: [prod]
labelSelectors:
app: checkout-service
delay:
latency: "200ms"
jitter: "50ms"
duration: "5m"
scheduler:
cron: "@every 30m"Blast radius and the abort switch
The single rule that separates chaos engineering from recklessness is blast radius control. Start in staging, then move to production against a single canary instance or a small percentage of traffic, and only widen the scope once the hypothesis holds repeatedly. Every experiment needs an automatic abort condition tied to a real SLO — if error rate crosses a threshold, the experiment stops itself rather than waiting for a human to notice. Running chaos experiments without a kill switch is how you turn a controlled test into the incident you were trying to prevent.
A scheduled game day, where the team knows roughly when an experiment will run and gathers to watch, is a different tool than continuous, automated fault injection running unannounced in production. Both have value — game days build muscle memory and validate runbooks with people present; unannounced chaos catches the assumptions nobody thought to write down. Don't treat them as interchangeable.
What to measure
The interesting output of a chaos experiment is rarely "did it crash." It's whether alerts fired, whether the on-call runbook's diagnosis steps actually led somewhere useful, whether the dashboard showed the operator what was happening, and whether the automated recovery (a retry, a circuit breaker, an autoscaler) kicked in within the time your SLA assumes. A system that survives the fault but leaves no trace in monitoring has a blind spot just as dangerous as one that falls over.
Before injecting anything, define and instrument the steady state — the metrics that describe "normal" (latency percentiles, error rate, throughput). Without a measured baseline, you cannot tell whether a wobble during the experiment was caused by your fault injection or was already happening.
Where to start
Teams new to chaos engineering get the most value from the cheapest, safest experiments first: kill a single non-leader replica, add latency to one downstream dependency, or fail a non-critical background job. Save cross-AZ failover and dependency-wide outages for once the team trusts the tooling and has a track record of clean aborts. The goal is not to prove the system is fragile — most systems are, in ways nobody has looked for yet — it's to find and fix the fragility on your own schedule, not the internet's.
Chaos engineering doesn't replace testing, monitoring, or incident response — it validates that all three actually work together under real failure conditions. Start small, write the hypothesis down, keep the abort switch within reach, and let the findings drive the backlog rather than the anxiety.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.