The Incident Command System (ICS) originated in US wildfire response in the 1970s, after multiple agencies discovered their separate command structures and terminology made coordinating a large fire nearly impossible. Software engineering teams borrowed the core idea decades later, and it solves the same underlying problem: when something is on fire (literally or figuratively) and several people are trying to help, without clear roles the response is often slower and more chaotic than if fewer people were involved.
The Incident Commander role, and why it's not the fixer
The single most important idea in ICS, and the one teams most often get backwards, is that the Incident Commander (IC) does not fix the problem. The IC's job is coordination: tracking what's known, deciding who's investigating what, deciding when to escalate or pull in more people, managing communication to stakeholders, and declaring when the incident is resolved. The person best qualified to fix the underlying issue is rarely the person who should also be fielding Slack questions from six different stakeholders — splitting those roles is what lets the technical responder actually focus.
A common mistake is assuming the most senior engineer should always be IC. In practice, IC is a skill (coordination, calm communication, decision-making under uncertainty) that's somewhat independent of deep technical expertise in the specific system that's broken. Many mature incident programs train several people to serve as IC and rotate the role, keeping the most technically relevant person free to actually investigate.
Severity levels: the point is consistent triggers, not precision
Most incident programs define 3-5 severity levels (SEV1 through SEV4/5 is common), each tied to concrete triggers — a SEV1 might mean "customer-facing outage affecting all users" and automatically triggers paging the on-call IC, opening a dedicated channel, and starting a timeline. The specific labels matter less than having them defined in advance and consistently applied: the value isn't a precise severity score, it's that everyone in the organization knows what response a given severity level triggers, so no one has to argue about whether to page someone at 2am while the outage is ongoing.
Roles beyond the IC: scribe and communications
Larger incidents benefit from two more defined roles. A scribe maintains a timeline in real time — what was observed, what was tried, when — which sounds like overhead during the incident but is the single biggest lever on how useful the postmortem afterward will be, because human memory of a stressful hour is unreliable within a day. A communications lead handles updates to stakeholders and customers, freeing the IC and the responders from having to context-switch out of the technical problem to answer "any update?" every ten minutes. For a small team, one person may hold multiple roles, but naming the roles explicitly, even informally, still helps.
The instinct during a bad incident is to pull in everyone who might know something. Past a certain point this actively slows resolution — more people means more competing theories, more parallel (uncoordinated) changes to the system, and more communication overhead for the IC. ICS explicitly limits who's "in the incident" versus who's a passive observer, and the IC controls that boundary.
Declaring and closing: both need clear criteria
Teams are usually decent at declaring an incident but bad at closing one cleanly — the system recovers, everyone quietly stops paying attention, and no one formally declares it resolved or schedules the retrospective. Define closure criteria the same way you define severity triggers: what observable condition means the incident is over (error rate back to baseline for N minutes, not just "looks fine now"), and who has authority to declare it closed. This matters because an incident that's never formally closed rarely gets a retrospective, and the lessons from it are lost.
| Role | Responsibility | Not responsible for |
|---|---|---|
| Incident Commander | Coordination, decisions, escalation, declaring resolution | Directly debugging/fixing the issue |
| Technical responder(s) | Investigating and fixing the underlying problem | Stakeholder communication |
| Scribe | Real-time timeline of observations and actions | Decision-making |
| Communications lead | Stakeholder/customer updates | Technical investigation |
Wrapping up
ICS works because it separates coordination from fixing, gives every incident a named owner for decisions, and defines severity and closure criteria in advance so no one has to negotiate them mid-fire. None of it requires special tooling — a defined IC rotation, a severity rubric, and the discipline to name a scribe on anything past a SEV2 gets most of the benefit.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.