Toil Reduction — A Field Guide

The Google SRE book defines toil precisely, and the precision matters: toil is work that is manual, repetitive, automatable, tactical rather than strategic, and devoid of enduring value — and it tends to scale linearly with the size of the service, so headcount growth just keeps pace with growth in pain instead of getting ahead of it. Most teams use "toil" loosely to mean "work I don't enjoy," which is a different problem with a different fix. The useful version of the definition is narrower, and narrower is what makes it actionable.

What actually counts as toil

Run a candidate task against all five criteria before you call it toil. Restarting a crashed process by hand is manual, repetitive, and automatable — clearly toil. Investigating a novel outage you've never seen before is manual and non-repetitive, but it's not toil, because it has enduring value: you learn something and probably fix a real bug. Writing a design doc is manual and non-repetitive and has enduring value — not toil, even though it's slow and sometimes tedious. The test isn't "does this feel like drudgery," it's "if I automate this away, does the service get permanently better, or does the same fire just start again next week." Provisioning a new customer environment by hand every time a sales deal closes is toil. Debugging why a specific customer's provisioning script failed for a new reason is not — that's operational work, and it's often where the next automation idea comes from.

The toil budget

The SRE book's suggestion — cap toil at roughly half of an SRE's time, with the rest protected for project work — isn't a law of physics, it's a forcing function. The number itself matters less than having one at all. Without a stated cap, toil expands to fill all available time, because it's usually the loudest, most urgent-feeling work in the queue: an alert fired, a ticket is open, someone is waiting. Project work that would eliminate next month's toil is easy to defer indefinitely because nothing breaks today if you don't do it. A budget makes toil visible as a line item instead of an ambient tax, and it gives a team backed into a corner — 80% of the week gone to firefighting — a concrete number to point at when asking for headcount or for permission to stop taking on new manual responsibilities until the automation catches up.

Measuring it honestly requires tagging work as it happens, not reconstructing it from memory at the end of the quarter — a simple label on tickets or a category in your on-call log is enough. The point isn't precision to the decimal, it's noticing the trend: is toil as a share of time going up, flat, or down as the service grows?

Automating toil: real examples

Auto-remediation for known-shape alerts is the highest-leverage place to start, because the fix is usually already documented in a runbook that a human is manually executing at 3 a.m. If your runbook for "worker process wedged, health check failing" is "SSH in, check the health endpoint, restart the service if it's down," that whole sequence can be a script triggered by the same alert that pages a human today — with the human only paged if the automated remediation itself fails.

#!/usr/bin/env bash
# auto-restart a stuck worker after confirming it's actually unhealthy
set -euo pipefail

SERVICE="worker.service"
HEALTH_URL="http://127.0.0.1:9000/healthz"
MAX_RETRIES=3

is_healthy() {
  curl -fsS --max-time 2 "$HEALTH_URL" >/dev/null 2>&1
}

for attempt in $(seq 1 "$MAX_RETRIES"); do
  if is_healthy; then
    exit 0
  fi
  echo "attempt $attempt: $SERVICE unhealthy, waiting before retry"
  sleep 5
done

echo "restarting $SERVICE after $MAX_RETRIES failed health checks"
systemctl restart "$SERVICE"
sleep 10

if is_healthy; then
  logger -t auto-remediate "$SERVICE recovered after restart"
  exit 0
else
  logger -t auto-remediate "$SERVICE still unhealthy after restart, paging human"
  exit 1
fi

Note the shape: confirm the failure isn't transient, attempt the known-good remediation, verify it worked, and only escalate to a human when the automation's own assumptions don't hold. That last branch is what keeps this from becoming a silent, flapping restart loop that masks a real problem.

Self-service tooling that replaces ticket-driven provisioning is the second big category. A new database, a new environment, a new API key — if the fulfillment is "engineer reads a ticket, runs the same five commands they ran last week, closes the ticket," the toil isn't the commands, it's the human being the interface between a request and a script. Wrapping that script in a form, a Slack command, or a self-service CLI removes the human from the loop entirely and, as a side effect, makes the request instant instead of queued behind whoever's on call.

Certificate rotation is the textbook case precisely because manual renewal fails in a specific, expensive way: nothing goes wrong for eleven months, then an outage happens because someone missed a calendar reminder. Replacing that with automated renewal — ACME clients, cert-manager in Kubernetes, or even a plain systemd timer wrapping certbot renew — turns a once-a-year manual step with a real outage risk into a boring recurring job that either succeeds silently or alerts well before expiry.

# /etc/systemd/system/cert-renew.timer
[Unit]
Description=Daily certificate renewal check

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

# /etc/systemd/system/cert-renew.service
[Unit]
Description=Renew TLS certs if due, alert on failure

[Service]
Type=oneshot
ExecStart=/usr/bin/certbot renew --quiet --deploy-hook "systemctl reload nginx"
ExecStartPost=/bin/sh -c '[ $EXIT_STATUS -eq 0 ] || curl -fsS -X POST "$ALERT_WEBHOOK" -d "cert renewal failed"'

Why zero toil is the wrong goal

It's tempting to treat "eliminate all toil" as the north star, but that goal has three problems that show up in practice. First, automation has a maintenance cost of its own — the auto-remediation script above needs to be tested when the service's failure modes change, and a stale script that "fixes" the wrong thing is worse than a human noticing something's off. Automating a task doesn't remove the work, it converts recurring manual work into occasional automation upkeep, which is a good trade only when the recurring cost was higher.

Second, some toil is genuinely cheaper to tolerate than to automate. A task that happens twice a year and takes twenty minutes is not worth a week of engineering time to eliminate, no matter how satisfying the automation would be to build. The toil budget exists partly to make this trade-off explicit instead of pretending every manual task is equally worth removing.

Third, the goal was never "no toil," it was "toil capped low enough that engineering work happens." A service with a growing on-call rotation that spends 15% of its time on manual work and is shipping reliability improvements every quarter is healthier than a service with 0% toil and a brittle, under-tested automation layer that nobody wants to touch.

Wrapping up

Toil reduction isn't a campaign to get to zero — it's a discipline of noticing which manual work is actually toil by the formal definition, automating the kind that's expensive and recurring enough to justify the build and maintenance cost, and leaving the rest alone on purpose. The toil budget is useful precisely because it forces that trade-off into the open instead of letting either extreme win by default: a team drowning in unmeasured toil, or a team that has automated itself out of the operational knowledge it needs for the next incident that doesn't fit any existing runbook.