Incident Response for LLM Apps

Bolting an LLM onto an on-call rotation sounds appealing right up until it pages the wrong team at 3am with a confident, plausible-sounding root cause that's wrong. Incident response is one of the higher-leverage places to use an agent — there's a firehose of alerts, logs, and runbooks that a human triages by pattern-matching, which is exactly what a model with good tool access can accelerate — but it's also one of the least forgiving places to get the guardrails wrong, because a bad automated action during an incident makes the incident worse.

What the agent should and should not do

Split the job into three stages: correlation, triage, and remediation. Correlation — pulling related alerts, recent deploys, and relevant logs into one timeline — is safe to fully automate; it's read-only and the worst outcome is a noisy summary. Triage — proposing a likely root cause and severity — should be agent-assisted but human-confirmed, because a wrong severity assignment either wakes people up for nothing or under-pages a real outage. Remediation — restarting a service, rolling back a deploy, scaling a pool — is where you want the agent proposing a specific, reversible action from a fixed runbook menu, not improvising a fix from first principles. An LLM that writes and executes arbitrary remediation commands during a live incident is a second incident waiting to happen.

Alert correlation is a retrieval problem, not a generation problem

The most reliable part of an incident agent is the part that does the least "thinking." Give it tool access to your alerting system, log aggregator, and deploy history, and have it assemble a timeline: what fired, in what order, what changed in the window before the first alert. This is standard function-calling — the model decides which tools to call and in what order, the orchestration layer executes them and returns structured results, and the model synthesizes a summary. Keep the summary grounded by requiring the model to cite the alert ID or log line for every claim it makes; an incident summary with unverifiable claims is worse than a shorter one you can trust.

tools = [
    {"name": "get_active_alerts", "description": "Fetch alerts firing in the last N minutes"},
    {"name": "get_recent_deploys", "description": "List deploys to a service in a time window"},
    {"name": "query_logs", "description": "Run a scoped log query against a service"},
    {"name": "get_runbook", "description": "Fetch the runbook for a known alert type"},
]

response = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=2048,
    tools=tools,
    system=(
        "You are triaging a production incident. Correlate alerts, deploys, "
        "and logs into a timeline. Every claim must cite an alert ID or log line. "
        "Propose a severity and likely cause, but do not execute remediation — "
        "hand remediation options to the on-call engineer for approval."
    ),
    messages=[{"role": "user", "content": incident_context}],
)
# Loop: execute any tool_use blocks, feed tool_result back, repeat until
# the model returns a final triage summary with no further tool calls.

Runbook execution needs a fixed menu, not a free-form shell

If the agent is allowed to take remediation actions, constrain it to a small, pre-approved set of parameterized operations — restart this service, roll back to the last known-good deploy, scale this pool to N instances — each implemented as a tool with strict argument validation, not a general "run this command" tool. The model should never be one prompt injection away from running arbitrary shell. Every remediation tool call should also be reversible or at minimum idempotent, and logged with the reasoning that led to it before it executes, so the retro afterward has a clear record of what the agent decided and why.

The agent needs its own observability

Log every tool call, every model response, and every escalation decision with a trace ID tied to the incident. When the postmortem asks "why did the agent think this was a database issue," you need the actual transcript, not a reconstruction. Track false-positive and false-negative triage rates over time per alert type — an agent that's accurate on API latency alerts but consistently wrong on database connection pool exhaustion is telling you it needs better tool access or a more specific runbook for that class of alert, not that the whole system is unreliable.

Escalate on disagreement, not just on low confidence

Rather than trusting a single self-reported confidence score, run correlation twice — once against the alerting data alone, once with the added context of recent deploys — and escalate to a human immediately if the two triages disagree on severity or likely cause. Disagreement between independent passes is a far more honest signal that something is ambiguous than asking the model to grade its own certainty, and it costs one extra API call, which is cheap compared to a wrong page.

Wrapping up

Incident-response agents earn their keep on correlation and triage assistance, where the cost of a wrong answer is a few wasted minutes. Keep remediation on a fixed, parameterized runbook menu instead of free-form execution, require human sign-off on resolution, and escalate on disagreement between independent triage passes rather than trusting a single confidence score. The goal isn't a fully autonomous on-call replacement — it's cutting the time between "something's wrong" and "a human with full context is looking at it."