Fallback Strategies for LLM Failures

If your feature calls one model from one provider, its availability is capped by that provider's — and LLM APIs have real outages and degraded periods. A fallback strategy is what turns "the AI feature is down" into "the AI feature is a little slower or a little simpler right now." The layers go from cheapest to most drastic.

Model and provider fallback

The first layer is a fallback chain: try the primary model, and on a retryable failure, try a secondary. The secondary can be a cheaper model from the same provider (survives a capacity issue on the flagship) or an equivalent model from a different provider (survives a full provider outage). Because prompts are mostly portable, a cross-provider fallback is usually a thin adapter, not a rewrite.

PROVIDERS = [
    ("anthropic", "claude-primary"),
    ("openai", "gpt-secondary"),
]

def complete(messages):
    last = None
    for provider, model in PROVIDERS:
        try:
            return call(provider, model, messages)
        except (RateLimitError, ServerError, TimeoutError) as e:
            last = e
            continue
    raise AllProvidersFailed(last)

Keep the fallback list ordered by preference and only fall through on retryable errors — a content refusal or a bad-request should not cascade down the chain, because every model will reject it the same way. Log which provider actually served each request so you can see when you are silently running on the backup.

Capability degradation

The second layer accepts a worse-but-working answer. If a summarisation call fails, return the first paragraph of the source. If a smart-reply generator is down, offer a short list of canned replies. If enrichment fails, save the record without it and enrich later. The principle is that the feature has a floor it degrades to, not a cliff it falls off.

Cached and templated fallbacks

The last layer is a response you already have. A recent cached answer for a similar query, or a deterministic template filled from structured data, keeps the surface alive during a total outage. It will be staler or blander than a live generation, but it is a page that renders instead of an error that does not.

Match the effort to the stakes: a background enrichment job might need nothing but a retry, while a user-facing feature on your pricing page earns the full chain — secondary model, degraded capability, and a cached floor beneath both.