Observability for LLM Apps

Standard APM tells you a request took 800ms and returned 200. For an LLM-backed agent that's almost useless — the request could have taken eight LLM calls, picked the wrong tool twice, and produced a confidently wrong answer, all while looking healthy on a normal dashboard. Observability for LLM apps means instrumenting the things that are specific to this kind of system: the prompt, the tool calls, the token spend, and the reasoning steps in between, not just the HTTP envelope around them.

Traces, not just logs

A single user request to an agent can trigger multiple LLM calls, tool invocations, and retries. Log lines scattered across that sequence are hard to reconstruct after the fact. What you actually want is a trace: one request ID that threads through every LLM call and tool call in the chain, each recorded as a span with its own duration, token count, and result. Standard distributed tracing (OpenTelemetry works fine here) applies directly — an LLM call is just a span with unusual attributes: prompt tokens, completion tokens, model name, and the tool calls it triggered.

from opentelemetry import trace

tracer = trace.get_tracer("agent.llm")

def call_model(messages, tools):
    with tracer.start_as_current_span("llm.call") as span:
        response = client.messages.create(model=MODEL, messages=messages, tools=tools)
        span.set_attribute("llm.model", MODEL)
        span.set_attribute("llm.input_tokens", response.usage.input_tokens)
        span.set_attribute("llm.output_tokens", response.usage.output_tokens)
        span.set_attribute("llm.tool_calls", len(response.tool_calls or []))
        return response

The four numbers worth a dashboard

Token spend per request (cost has a direct line to your bill, and a regression here is often the first sign something changed upstream), tool-call success rate (a tool returning an error the agent has to reason around is a hidden tax on every subsequent call), steps-to-completion (a jump from 2 average tool calls to 6 for the same task class means the agent started looping or second-guessing), and end-to-end latency broken down by span, not just totalled. The total latency number tells you something is slow; the span breakdown tells you whether it's the model, the tool, or the network — three completely different fixes.

Sampling what you store

Storing full prompts and responses for every request gets expensive and raises privacy questions fast, especially if customer or employee data flows through the prompt. A workable default: store full traces for a sampled percentage of traffic, plus 100% of traces that ended in an error, a low-confidence tool result, or a human escalation. That combination catches the interesting cases — failures and edge cases — without paying to store every routine success in full detail.

Alerting on agent-specific signals

Standard alerts (error rate, p99 latency) still matter, but add two that are specific to agents: a spike in tool-call error rate (often means an upstream API — Acumatica, a payment gateway — changed shape or started rate-limiting) and a spike in "agent gave up / asked for human help" rate (often the earliest signal of a prompt regression or a model version change, well before it shows up in a satisfaction survey). Both are cheap to compute from the same trace data you're already collecting.

Connecting observability to the runbook

A trace that shows the fifth tool call returned a 429 from Acumatica's API is only useful if someone on call knows what to do with a 429 from Acumatica's API. Attach a short runbook link to each alert type — "tool error rate spike" points to "check API gateway rate limits and the retry/backoff config," not to a generic "investigate" ticket. The instrumentation earns its cost only when it shortens the path from alert to fix.

Wrapping up

Treat each LLM call and tool call as a traced span with token, cost, and outcome attributes, not just a log line. Sample full detail for routine traffic and keep everything for failures. The dashboards only pay for themselves if every alert routes to a runbook someone can actually execute at 3am — instrumentation without that link is just a more expensive way to notice the problem after the fact.