Drift Monitoring for LLM Apps

A model doesn't need to be replaced for its behaviour to change under you. Provider-side model updates, prompt template edits by a teammate, upstream data shifts (a new product catalogue, a new customer segment), and seasonal shifts in what users ask all quietly move an agent's output distribution. Traditional software monitoring — uptime, latency, error rate — won't catch any of it, because the requests still return 200 and the agent still produces plausible-looking text. Drift monitoring is the practice of catching "still running, quietly wrong."

What actually drifts

Three things move independently and need separate tracking. Input drift: the distribution of what users ask changes — new intents, new phrasing, a new integration sending traffic you didn't design for. Output drift: for the same class of input, the agent's answers shift — tone, format, tool-call patterns, refusal rate. Task drift: the downstream outcome quality changes — more escalations to a human, more corrections, more support tickets referencing the agent. Output drift is the easiest to instrument and the least useful signal on its own; task drift is what the business actually cares about and the hardest to measure cheaply.

Cheap proxies before expensive evals

You don't need an LLM-judge pipeline running on every request to get a useful early-warning signal. Track things you already have for free: response length distribution, tool-call frequency and which tools get called, refusal/clarification rate, and latency per call. A jump in "the agent asked a clarifying question" rate, or a drop in a specific tool's call frequency, is often the first visible symptom of an upstream prompt change or a model version bump — and it costs nothing to compute from logs you're already writing.

from collections import Counter
import statistics as stats

def daily_signals(logs: list[dict]) -> dict:
    tool_calls = Counter(l["tool_name"] for l in logs if l.get("tool_name"))
    lengths = [len(l["response"]) for l in logs]
    return {
        "date": logs[0]["date"],
        "tool_call_mix": dict(tool_calls),
        "median_response_len": stats.median(lengths),
        "clarify_rate": sum(l["asked_clarifying_question"] for l in logs) / len(logs),
        "avg_latency_ms": stats.mean(l["latency_ms"] for l in logs),
    }

# Alert when today's mix deviates > 2 std devs from a 14-day rolling baseline

Sampled, LLM-judged evaluation

For the output-quality question that cheap proxies can't answer, sample a small percentage of production traffic (1-5% is usually enough) and score it against a fixed rubric with a separate judge model — not the same model generating the responses. Keep the rubric narrow and specific to the task ("did the agent correctly identify the invoice number," not "was this a good response") so scores are reproducible and comparable week over week. Run this on a schedule, not on every request; it's a monitoring signal, not a gate.

Baselines and when to page someone

Drift detection is comparative, so the first thing to build is a stored baseline — a rolling 14-to-30-day window of the metrics above, not a single "golden" snapshot that goes stale. Alert on statistically meaningful deviation from the rolling baseline, not on absolute thresholds picked once in a design meeting. Anything triggered by a known deploy (you changed the prompt, you changed model version) should be expected and doesn't need a page; anything else — especially a shift with no corresponding deploy in your change log — is the case worth waking someone up for, because it usually means the input distribution or the provider's model moved without your permission.

Closing the loop

Monitoring without a response plan is just a dashboard nobody looks at. Tie drift alerts to a concrete playbook: pin the model version if a provider update is suspected, roll back the last prompt change if it correlates with the shift, or widen the sampled-eval rate temporarily to get a faster read. The goal isn't zero drift — inputs will always shift — it's catching drift before it becomes a support queue full of tickets nobody connected back to the agent.

Wrapping up

Drift monitoring doesn't require a dedicated ML observability platform to start — a rolling baseline over metrics you already log, plus a small sampled eval, catches most of what matters. Segment before you average, alert on deviation from a rolling window rather than a fixed threshold, and make sure every alert has an owner and a playbook. The expensive tooling is worth adding once you know which signals actually predict the problems you've had, not before.