Game Days for SRE Teams — A Field Guide

A game day is a scheduled exercise where a team deliberately breaks something in a controlled way to find out whether the systems, the alerting, and the humans respond the way the runbook says they will. The value isn't the outage — it's finding the gap between what you believe will happen during an incident and what actually happens, while it's still Tuesday afternoon and not 3am on a real one.

What a game day actually tests

A well-run game day tests three things at once: whether monitoring and alerting fire correctly for the injected failure, whether the documented runbook is accurate and findable under pressure, and whether the on-call engineer can execute it without needing the person who wrote it. Teams that only test the first — "did the alert fire" — miss most of the value. The alert firing and someone successfully mitigating are different questions, and game days exist to answer the second one.

Picking a failure that teaches you something

The best game day scenarios come from real, plausible failure modes — a dependency timing out, a region losing connectivity, a cache going cold, a queue backing up — not creative but implausible disasters. Start from your architecture's actual single points of failure and your past incident history; if a database failover has never been tested outside of production, that's a stronger game day candidate than an exotic multi-region scenario nobody's architecture actually needs to survive.

Roles: facilitator, scribe, and participants

A game day needs someone running the exercise who is not also debugging it — a facilitator who knows the injected failure and observes without giving hints, and a scribe capturing a timeline: when the failure was injected, when it was detected, when mitigation started, when it resolved. Participants respond as if it were a real incident, using the actual paging and communication channels. Skipping the facilitator/scribe split is the most common way a game day quietly turns into an unstructured debugging session that produces no usable timeline afterward.

game_day:
  name: payment-service-dependency-timeout
  date: 2026-08-05
  facilitator: sre-lead
  scribe: sre-secondary
  hypothesis: "On-call detects elevated latency within 5 minutes and
    fails over to the backup payment provider within 15 minutes,
    per the documented runbook."
  injection:
    method: inject 2000ms latency on payment-gateway calls via proxy
    blast_radius: staging environment, 10% of synthetic traffic
  abort_conditions:
    - error rate exceeds 25% in staging
    - facilitator calls "stop" at any time
  success_criteria:
    - alert fires within 5 minutes of injection
    - on-call follows runbook without escalation
    - mitigation completes within 20 minutes

The blameless retro is not optional

A game day that ends without a retrospective wastes most of its value. The retro should be blameless by design — the point is finding gaps in systems and documentation, not finding fault in the responder, who was following whatever the runbook said. Every gap found (an alert that didn't fire, a runbook step that was wrong, a dashboard that didn't exist) becomes a tracked follow-up item with an owner, and the next game day should verify the fix actually closed the gap rather than assuming it did.

Wrapping up

Game days work when the scenario is plausible, the roles are separated so the exercise produces a real timeline, and every gap surfaces as a tracked, owned follow-up rather than a war story. Run them on a cadence, not as a one-off — the value compounds as the team builds confidence that the runbooks they wrote actually work under pressure.