Rollback Strategy for LLM Apps

Rolling back a bad model deployment is nothing like rolling back a bad code deployment, and treating it the same way is where teams get burned. Code rollback reverts a binary. An LLM agent's mistakes can already be sitting in an approved invoice, a sent email, or a posted GL entry by the time anyone notices the new prompt or model version has a problem — reverting the code doesn't undo any of that.

Separate two different things that both get called "rollback"

Rolling back the system (reverting to the previous prompt, model version, or agent code) is fast and should be treated like any other deploy rollback — a version-controlled artifact you can redeploy in minutes. Rolling back the consequences of what the bad version already did (unwinding an incorrect approval, correcting a miscategorized transaction, retracting a sent communication) is a completely separate problem that code rollback does nothing for. Plan for both, and don't let a fast system rollback create false confidence that the incident is over.

Version everything that shapes behavior

You can't roll back to "the version that worked" if you didn't record what that version was. Version the system prompt, the tool definitions, the model identifier (including the dated snapshot, not just "gpt" or "claude"), and any few-shot examples as a single deployable unit, the same way you'd version application code. A prompt tweak shipped outside the normal deploy pipeline — edited directly in a config UI, say — is the single most common way teams lose the ability to roll back cleanly, because nobody can say what the previous state actually was.

Make consequential actions reversible by design

The cheapest rollback strategy is designing the system so the LLM's actions are reversible before they happen. An agent that drafts an email for human send, rather than sending it directly, gives you a free undo. An agent that proposes a GL entry as a pending batch rather than auto-posting it gives finance a chance to catch a mistake before it's in the ledger. Reserve direct, irreversible execution for actions with a low cost of being wrong; require a human step or a soft-delete/undo window for everything else.

def apply_agent_action(action, risk_tier):
    if risk_tier == "low":
        return execute_immediately(action)  # e.g. draft saved, no external effect

    # medium/high risk: land in a pending state, never auto-committed
    pending_id = queue_for_review(action)
    log_agent_decision(action, pending_id, model_version=CURRENT_MODEL_VERSION)
    return {"status": "pending_approval", "id": pending_id}

Have a kill switch, not just a rollback

When an agent is actively producing bad output, the fastest safe response is often to disable it entirely — route to a human queue or a simple non-LLM fallback — rather than waiting for a proper rollback deploy to go through CI. Build this as a feature flag checked before every agent invocation, cheap to flip, tested in advance so it isn't the first time it's exercised during an actual incident.

Write the unwind runbook before you need it

For each consequential action type the agent can take, document in advance how to identify affected records (query by the model-version tag logged with each decision) and how to reverse them (void the transaction, recall the email, re-open the ticket). Writing this after an incident, under pressure, produces worse decisions than writing it calmly beforehand — and the version tagging from the earlier section is what makes "which records were touched by the bad version" answerable at all.

Wrapping up

A rollback strategy for an LLM app has to cover both the fast code-level revert and the slower work of unwinding real-world consequences already taken — version everything that shapes behavior, pin model snapshots, bias toward reversible actions, keep a kill switch, and write the unwind runbook before an incident forces you to write it in a hurry.