Regression Testing for LLM Apps

Regression testing an LLM app is harder than regression testing normal code because the same input doesn't reliably produce the same output, and "close enough" isn't a concept most CI pipelines are built to evaluate. The teams that avoid silent quality decay treat this as a measurement problem — build a fixed dataset, define what "correct" means for each case, and run it on every change, not just the ones that look risky.

Build the eval set from real failures

The most valuable regression cases aren't invented — they're pulled from actual production incidents and support tickets: the time the agent misread a vendor bill's currency, the time it approved a duplicate payment request, the time it summarized a project status incorrectly. Every confirmed bug becomes a permanent test case with the correct expected output, the same discipline as adding a unit test for every bug fix in traditional code. Over a year, this set naturally grows to cover your system's actual failure modes rather than a generic list someone guessed at upfront.

Grading output that isn't exact match

Free-text output can't be graded with string equality, but that doesn't mean grading has to be manual. Use a combination of: structural checks (does the output parse as valid JSON matching the schema, are required fields present), rule-based checks (does the extracted amount match the source document, does the referenced GL account exist), and, only where those don't suffice, a separate LLM call as a grader against an explicit rubric. Reserve the LLM-grader approach for genuinely subjective quality (is this summary clear) — never for anything with a factually correct answer, where a deterministic check exists and is cheaper and more reliable.

Run it on every prompt and model change

The two changes most likely to cause silent regressions are prompt edits and model version upgrades, and both are exactly the changes teams are tempted to ship without re-running the full eval set because "it's just a wording tweak." Wire the eval run into the same CI gate as code tests — a prompt change that drops accuracy on the fixed set by more than a defined threshold blocks the merge, the same way a failing unit test would.

def run_eval_suite(agent, cases, baseline_accuracy):
    results = [grade(case, agent.run(case.input)) for case in cases]
    accuracy = sum(r.passed for r in results) / len(results)
    regressions = [r for r in results if r.was_passing_before and not r.passed]

    if accuracy < baseline_accuracy - 0.02:  # 2-point drop fails the gate
        raise EvalGateFailure(f"Accuracy {accuracy:.2%} vs baseline {baseline_accuracy:.2%}",
                               regressions=regressions)
    return accuracy

Separate the deterministic parts from the fuzzy parts

Most agent pipelines have a mix of deterministic code (tool implementations, validation, formatting) and model calls. Test the deterministic parts with ordinary unit tests — fast, exact, no flakiness — and reserve the eval-set approach for the genuinely model-dependent behavior. Conflating the two means slow, flaky test suites for logic that didn't need an LLM in the loop to verify.

Track accuracy over time, not just at release

Model providers update models behind the same API endpoint without always notifying every downstream user in time. Run the eval suite on a schedule against production, not just at deploy time, so a silent upstream model change that degrades your specific use case shows up as a trend line rather than a customer complaint three weeks later.

Wrapping up

Treat every confirmed LLM-app bug as a permanent regression case, grade with the cheapest reliable method for each case type, and gate merges on the full eval set rather than a spot check — that's what turns "the AI got worse" from a mystery into a number you can watch and act on before customers notice.