Site reliability engineering gets reduced to error budgets and SLO dashboards a lot, but the fundamentals that actually determine whether a team survives on-call are less quantifiable: how much toil is baked into the weekly routine, whether incidents produce learning or just paperwork, and whether the on-call rotation is something people dread or something they can do sustainably for years. This is the practice side of SRE — the habits and structures, not the math.
Toil is the tax you pay for not fixing things
Toil is work that's manual, repetitive, and scales linearly with the system — restarting a service that wedges every few days, manually re-running a failed batch job, clearing a disk that fills up on a predictable schedule. None of it is hard. All of it eats the time that should go to actually improving reliability. Google's original SRE workbook put a rough ceiling on this: if toil crosses about half of an engineer's time, the role has quietly turned into ops with extra steps.
The fix isn't heroic automation — it's tracking toil as a first-class backlog item. When the same manual fix shows up in an incident channel three times, that's the signal to write the runbook into code, not to get faster at doing it by hand.
The first time something breaks, write a runbook. The second time, turn the runbook into a script. Automating a one-off is wasted effort; automating a pattern you've now confirmed is recurring is the actual leverage.
On-call that people can sustain
A rotation fails long before anyone quits over it — it fails the moment engineers start dreading their week on the pager. The fundamentals that keep on-call sustainable are boring: a rotation long enough that a bad week doesn't repeat too often (a week on, several weeks off, not week-on-week-off), a secondary who can be paged if the primary doesn't ack, and a hard rule that anything paging more than once a week gets triaged as a reliability bug, not tolerated as background noise.
# Alertmanager routing: page primary, escalate to secondary after 10 minutes
route:
receiver: primary-oncall
group_wait: 30s
group_interval: 5m
routes:
- match:
severity: critical
receiver: primary-oncall
continue: true
routes:
- match:
severity: critical
receiver: secondary-oncall
group_wait: 10m # only fires if primary hasn't resolved/acked
receivers:
- name: primary-oncall
pagerduty_configs:
- routing_key: ''
- name: secondary-oncall
pagerduty_configs:
- routing_key: ''
Every alert that fires and gets acknowledged with "yeah, that's fine" trains the responder to distrust the next page. Delete or downgrade alerts that never require action — a noisy on-call rotation is a bigger reliability risk than most of the outages it's supposedly watching for.
Postmortems that change behavior
A blameless postmortem isn't about avoiding blame for its own sake — it's a mechanism to get an honest timeline out of people who'd otherwise be defensive. If an engineer expects to be blamed for a mistake, they'll describe the incident vaguely enough to protect themselves, and the report loses the detail that would have prevented a repeat. The postmortem's job is to answer: what did the system let happen, and what changes so it can't happen the same way twice.
The output that actually matters isn't the document — it's the action items, and specifically whether they get done. A postmortem with five action items that all sit in the backlog for a year is theater. Better to write one or two concrete, owned, dated changes than a long list that never gets closed.
The error budget as a decision tool, not a scoreboard
Error budgets get treated as a report card, but their actual use is simpler: they settle the recurring argument between "ship the feature" and "stop and fix reliability" before it becomes political. If the budget is spent, feature work pauses and reliability work takes priority — no debate needed, because the rule was agreed before anyone had a stake in the outcome. If the budget is healthy, teams can take on more risk without guilt. The budget's value is in being decided in advance, not in how precisely it's calculated.
What 2026 actually changed
The tooling has gotten better — more of the toil that used to require a human (correlating an alert with a recent deploy, drafting the first pass of a postmortem timeline) can now be handled by an assistant that has read access to logs, deploy history, and the incident channel. That's a genuine toil reduction. What hasn't changed is the discipline underneath it: someone still has to decide what's worth automating, someone still has to own the action items, and no tool fixes a rotation that pages too often for something that isn't actually urgent.
Wrapping up
SRE fundamentals aren't a set of metrics to compute — they're a set of habits: track toil so it doesn't quietly become the whole job, build on-call rotations people can survive for years, run postmortems that produce real changes instead of paperwork, and use error budgets to settle disagreements before they start. Get those four right and the dashboards take care of themselves.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.