Acumatica · Sre

Incident Retrospectives — A Field Guide

Incident Retrospectives — A Field Guide is the work that turns a deploy into a system. The deployment is one moment; the system is the next 18 months of uptime, incidents, and.

John Kihiu12 min read

A retrospective (or postmortem) is the single highest-leverage step in incident response, and the one teams skip most readily once the fire is out. The point isn't the document — it's forcing a structured look at why the incident happened and why it took as long as it did to detect and fix, before the details fade and the team moves on to the next thing.

Blameless is a mechanism, not a mood

"Blameless postmortem" gets used as a vague cultural aspiration, but it's actually a specific analytical stance: assume the person who took an action that contributed to the incident made a reasonable decision given what they knew, what tools they had, and what pressures they were under at the time. The question isn't "why did they do that" (which implies fault) but "what made that action seem reasonable in the moment, and what would need to change so a similarly reasonable person wouldn't make the same call again." This isn't about being nice — it's that blame-focused retrospectives make people defensive and less forthcoming about what actually happened, which produces worse retrospectives and, over time, engineers who route around the process instead of engaging with it.

"Human error" is rarely a root cause worth stopping at

If the retrospective's conclusion is "the engineer ran the wrong command," ask why the system made that command easy to run in a dangerous way, why there was no confirmation step, or why staging and production looked similar enough to confuse. Human error is almost always the proximate cause, not the root one — the system let the error happen and didn't catch it.

The timeline is the foundation everything else is built on

A retrospective without an accurate timeline is speculation. This is why having a scribe capture events in real time during the incident (not reconstructed afterward from memory) matters so much — reconstructed timelines reliably compress the confusing, uncertain middle part of an incident into something that sounds cleaner and more decisive than it actually was, which hides exactly the detection and diagnosis delays you most need to learn from. The timeline should include when the problem started (not just when it was noticed), when it was detected, when the right people were engaged, when the cause was identified, and when it was actually fixed — each gap between these is a distinct thing to potentially improve.

Detection time is usually the biggest lever, not fix time

Teams instinctively focus retrospectives on "how do we fix this class of bug faster next time," but in most incidents the largest chunk of total impact duration is the gap between the problem starting and someone noticing — not the time it took to fix once found. An incident that took 5 minutes to fix but 40 minutes to detect has a monitoring and alerting problem, not primarily an engineering-quality problem, and the retrospective's action items should reflect that. Look at the timeline gaps explicitly before deciding where to invest follow-up work.

Action items need owners and dates or they don't happen

A retrospective that ends with a list of "we should improve monitoring" style bullets and no assigned owner or deadline reliably produces zero follow-through. Each action item needs a specific owner, a specific and checkable outcome, and a due date — treated with the same rigor as any other committed work, or tracked in the same backlog the team already uses.

What the document actually needs, at minimum

A useful retrospective doesn't need to be long: a brief summary, the customer/business impact (quantified if possible), the timeline, the root cause analysis (ideally using a technique like the "5 whys" to push past the first, most obvious explanation), and a short list of concrete action items with owners. Longer documents with extensive narrative rarely get read by anyone beyond the people who wrote them; a tight, scannable one is more likely to actually inform future decisions.

TEXT · MINIMAL RETRO TEMPLATE
## Summary
One paragraph: what broke, for how long, who was affected.

## Impact
Quantified where possible: users affected, revenue impact,
SLA breach, duration of degraded service.

## Timeline (UTC)
14:02 - Deploy of service X v2.14 begins
14:07 - Error rate begins climbing (not yet noticed)
14:22 - Alert fires, on-call paged
14:25 - IC declared, incident channel opened
14:41 - Root cause identified: missing feature flag default
14:48 - Rollback deployed
14:52 - Error rate back to baseline

## Root cause
(5 whys or equivalent — push past the first explanation)

## Action items
- [ ] Add default value validation to feature flag config
      loader (Owner: J. Doe, Due: next sprint)
- [ ] Reduce alert threshold latency from 15min to 5min
      (Owner: SRE team, Due: 2 weeks)
Timeline gapWhat it tells you
Start to detectionMonitoring/alerting quality
Detection to engagementOn-call/paging process quality
Engagement to root causeObservability and debugging tooling
Root cause to fixDeployment speed and rollback readiness

Wrapping up

A good retrospective is blameless in its analytical method, built on a real-time timeline rather than a reconstructed one, and honest about where the biggest gap actually was — usually detection, not the fix itself. Keep the document short, assign owners and dates to every action item, and treat those items as committed work, not aspirations.

John Kihiu
Acumatica ERP Developer · Laravel Engineer

Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.