Traditional tests assume a deterministic function: given this input, assert exactly this output. LLM outputs vary run to run, so that assertion breaks immediately and teams conclude LLM features "can't be tested." They can — you just test properties of the output and aggregate over a dataset, the same way you would evaluate a noisy classifier.
Build a golden dataset
The foundation is a set of representative inputs with known-good expectations. Seed it from real traffic: the queries users actually send, especially the ones that went wrong. Every production bug becomes a new test case. A few dozen well-chosen examples that cover your edge cases are worth more than thousands of synthetic ones.
Assertion-based evals
For each case, assert the properties that must hold rather than an exact string. Cheap, deterministic checks catch most regressions:
- Structural — valid JSON, required fields present, values in range.
- Content — must-contain / must-not-contain substrings, no leaked PII, correct label from a fixed set.
- Safety — no disallowed content, refusals where refusals are expected.
import pytest, json
CASES = json.load(open("evals/classify.json"))
@pytest.mark.parametrize("case", CASES)
def test_classification(case):
out = run_feature(case["input"])
assert out["label"] in LABELS # structural
assert out["label"] == case["expected_label"] # correctness
assert "ssn" not in out["explanation"].lower() # safetyLLM-as-judge for open-ended output
When the output is a summary or an answer with no single right form, use a second model as a grader: give it the input, the output, and a rubric, and have it score correctness, relevance, or tone. It is not perfect, so calibrate it against a sample of human-graded cases and keep the rubric specific. Used well, LLM-as-judge scales the subjective checks you could never assert with a substring.
The whole point is to run the evals in CI on every prompt, model, or parameter change, and block the merge if the aggregate score drops. A prompt tweak that fixes one case and quietly breaks five is the most common LLM regression — and the eval gate is the only thing that catches it before users do.
Track the pass rate over time, not a single run. LLM testing is statistical: you are asking "did this change make the system better or worse on our dataset," and a version-controlled eval suite is what makes that question answerable instead of a matter of opinion.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.