LLM quality regressions are sneaky: the app is up, requests succeed, but answers got worse in a way no exception catches. By the time a user complains, the bad version has been serving for hours. Rollback is your recovery, and it only works if you designed for it before the incident.
A release is prompt + model + config
The unit you roll back is not your application — it is the behaviour config: prompt version, model id, and parameters. If that config is baked into a code deploy, then reverting it means a full redeploy, which is slow exactly when you are under pressure. Instead, store the active version as data — a flag or a config record — so switching it is a value change, not a build.
Decouple release from deploy
Ship new prompt versions dark: the code that can run classify@4 is deployed, but a feature flag still points production at classify@3. Flipping the flag releases the new version; flipping it back rolls it back. Deploy and release become independent, which is what lets you revert in seconds without touching the codebase.
def active_prompt(feature):
# resolved from a config store / flag service, not hardcoded
return flags.get(f"prompt.{feature}", default="classify@3")
def run(feature, input):
spec = PROMPTS[active_prompt(feature)]
return call(spec, input)
# rollback = set flags["prompt.classify"] = "classify@3" (instant, no deploy)Canary before full release
Do not flip 100% of traffic to a new version at once. Route a small slice — 5–10% — to the candidate, watch your quality and cost signals, and widen only if they hold. A canary turns a bad release into a small, contained blast radius and gives your monitoring time to catch the regression before it is everyone's problem.
Always know which version was the last one that passed evals and behaved in production. That is your rollback target. 'Roll back' should mean 'point the flag at last-known-good,' a specific known state — not 'undo the last change and hope.'
The combination — behaviour as data, releases behind flags, canary rollout, and a pinned last-known-good — makes an LLM regression a five-second flag flip instead of an emergency deploy. You will still ship the occasional bad prompt; the difference is how fast you can take it back.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.