Error Handling in LLM Apps

A plain HTTP client treats an LLM call like any other request, which is why LLM features feel flaky in production. The model can time out, throttle you, return a 5xx, refuse on content grounds, overflow the context window, or — the one people forget — return a perfectly successful 200 whose body is not the JSON you asked for. Each of these wants a different response.

The failure taxonomy

Failure	Retry?	Handling
Timeout / connection	Yes	Retry with backoff; set an aggressive client timeout so you fail fast rather than hang.
429 rate limit	Yes	Honour `retry-after`, backoff with jitter.
5xx server error	Yes	Backoff; after N tries, fall back to another model or provider.
Context length exceeded	No	Trim or summarise input; retrying unchanged fails identically.
Content filter / refusal	No	Handle as a product state, not an error; show a safe message.
Malformed structured output	Once	Validate, then re-ask with the parse error; do not blind-retry.

The retry column is the important distinction. Retrying a transient network error is correct; retrying a context-length overflow just burns tokens to fail the same way. Classify before you retry.

Validate structured output

If you asked for JSON, assume you will sometimes get prose, a trailing comment, or JSON wrapped in a code fence. Validate against a schema and, on failure, re-prompt once with the validation error included — models are good at fixing a shape when told exactly what was wrong. If the second attempt also fails, fall back rather than loop.

Python · validate-then-repair

from pydantic import BaseModel, ValidationError

class Result(BaseModel):
    category: str
    confidence: float

def parse_result(raw, repair=True):
    try:
        return Result.model_validate_json(extract_json(raw))
    except (ValidationError, ValueError) as e:
        if not repair:
            raise
        fixed = call_model(REPAIR_PROMPT.format(error=e, raw=raw))
        return Result.model_validate_json(extract_json(fixed))

Fail into a known state

Every unrecoverable path needs a defined destination. A circuit breaker around a provider stops you hammering a dead endpoint. A fallback model or a cached/templated response keeps the feature usable when the primary is down. And a refusal from the content filter is not a bug to swallow — it is a product state that deserves a clear, honest message to the user.

Log the request that failed, not just the exception

When an LLM call misbehaves, the exception type is rarely enough to reproduce it. Log the model, prompt version, token counts, and a hash of the input. The failures that matter are usually input-shaped, and you cannot fix what you cannot replay.

Robust LLM error handling is mostly classification: decide what kind of failure you are looking at, retry only what is transient, validate what claims to be structured, and give everything else a defined place to land.

John Kihiu

Acumatica ERP Developer · Laravel Engineer

Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.

The failure taxonomy

Validate structured output

Fail into a known state

Related reading