Most alerting systems fail the same way: too many alerts that don't require action, so the ones that do get ignored. Good alerting is less about detection coverage and more about a small set of rules people actually trust enough to wake up for. The practices below are the ones that hold up once a system has been on-call for a year, not the ones that look good in a demo.
Alert on symptoms, not causes
The most durable rule in alerting design: page on user-visible symptoms (error rate, latency, availability), not on every possible internal cause. A disk filling up, a queue backing up, a cache miss rate spiking — these are diagnostic signals, not paging conditions, unless they're a leading indicator of imminent symptom-level failure. Google's SRE book calls this out explicitly: alert on SLO burn rate, and let the dashboards and logs carry the causal investigation once someone is already engaged.
If a system generates more than a couple of distinct paging alerts for the same underlying incident, the alerts are cause-based rather than symptom-based. Consolidate: one alert for "the SLO is burning too fast," with the cause-level detail available in the linked dashboard, not as a separate page.
Multi-window, multi-burn-rate alerts
A single threshold alert ("error rate over 5% for 5 minutes") either fires too late for fast, severe outages or too early for slow, tolerable degradations. The pattern that fixes both is a multi-window burn-rate alert: pair a short window (5m) with a long window (1h) at a high burn-rate threshold for fast-burn pages, and a longer pair (6h/3d) at a lower threshold for slow-burn tickets. This is the approach documented in Google's SRE workbook and it's now standard in most Prometheus/Grafana SLO tooling.
groups:
- name: slo-fast-burn
rules:
- alert: ErrorBudgetFastBurn
expr: |
(
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))
) > (14.4 * 0.001)
and
(
sum(rate(http_requests_total{status=~"5.."}[1h]))
/ sum(rate(http_requests_total[1h]))
) > (14.4 * 0.001)
for: 2m
labels:
severity: page
annotations:
summary: "Error budget burning 14.4x too fast — pages on-call"Every page needs a runbook link
An alert without a linked runbook is a puzzle handed to whoever is unlucky enough to be on call. The annotation on the alert itself should carry a direct link to a runbook that states what the alert means, what's already been ruled out, and the first three diagnostic steps. If a responder has to search Slack history to remember what an alert means, the alert is undocumented technical debt, not a safety mechanism.
A runbook that references a decommissioned service or an outdated dashboard link actively costs time during an incident, because the responder trusts it before discovering it's wrong. Review runbooks whenever the alert they're attached to changes, not on a separate schedule that inevitably drifts.
Routing and escalation without alert storms
Dependent failures cascade: a database going down triggers alerts on every service that depends on it. Alert grouping and inhibition rules (Alertmanager's group_by and inhibit_rules, or the equivalent in PagerDuty/Opsgenie) collapse a storm of 40 correlated alerts into one incident with 40 linked signals, instead of paging the same person 40 times in two minutes. Get this configured before the first real incident — retrofitting inhibition rules during a live outage is how storms turn into missed pages.
Reviewing alert quality over time
Track a precision metric for every paging alert: of the last N times it fired, how many led to an action versus an acknowledge-and-ignore. An alert with a low action rate should be demoted to a ticket, tuned, or deleted — not left in place because someone might theoretically need it. Teams that review this monthly keep their on-call load flat as the system grows; teams that don't accumulate alert fatigue until people start missing the alerts that matter.
| Signal type | Route to |
|---|---|
| SLO burn rate, fast window | Page immediately |
| SLO burn rate, slow window | Ticket, review next business day |
| Internal resource metric (disk, queue depth) | Dashboard only, unless it predicts imminent SLO breach |
| Known noisy alert, low action rate | Demote, tune threshold, or delete |
Wrapping up
Good alerting is a discipline of subtraction more than addition: fewer, symptom-based, well-routed alerts beat comprehensive coverage of every internal metric. Pair each page with a runbook, group correlated failures so one incident doesn't become forty pages, and review action rates regularly so the alert set stays trustworthy instead of just large.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.