Evaluating LLM Apps — A Complete Guide

Shipping an LLM feature without an eval harness is like shipping a payment integration without tests — it works in the demo and then quietly regresses the first time someone tweaks the prompt or the vendor rotates a model version underneath you. Evaluation for LLM apps splits into two distinct problems: catching regressions before you ship (offline) and knowing what's actually happening once real users hit it (online). Most teams build neither until something embarrassing happens in production.

Building a golden dataset

Start with 50-100 real examples, not synthetic ones — pull actual user queries, actual documents, actual edge cases from support tickets or logs, and hand-label the expected output. A golden set built from imagined inputs tends to be easier than production traffic, which means it stops catching regressions right around the time you need it most. Keep the set version-controlled next to the code, and grow it every time production surfaces a case the eval set didn't cover — a failure that isn't turned into a test case will recur.

Task-specific metrics beat generic ones

Match the metric to the task instead of reaching for a single "quality score." For structured extraction (invoice fields, entity extraction), use exact match or field-level accuracy against the labeled output — it's cheap, deterministic, and catches silent schema drift immediately. For an agent that calls tools, score tool-call correctness separately from the final answer: did it call the right tool, with the right arguments, in the right order — a wrong answer reached via the right tool calls is a different bug than a right answer reached via the wrong ones, and conflating them hides which one you actually have. For RAG systems, score groundedness/faithfulness — whether the answer's claims are actually supported by the retrieved context — separately from answer relevance, since a fluent, well-cited hallucination and an unsupported-but-correct guess fail your users differently.

LLM-as-judge, and why to distrust it a little

For open-ended outputs where exact match doesn't apply (summary quality, tone, helpfulness), using a second LLM call to score the first model's output is standard practice — but it has known failure modes worth designing around. Judges show position bias (favoring whichever answer appears first in a pairwise comparison), length bias (favoring longer answers regardless of quality), and self-preference bias (a model rating its own family's outputs more favorably). Mitigate by randomizing answer order in pairwise comparisons, using a different model family as judge than the one under test, and periodically spot-checking judge scores against human ratings to confirm they still agree — judge-human correlation drifts as both the judged model and the judge model get updated.

def test_invoice_vendor_extraction(golden_case):
    result = extract_invoice_fields(golden_case.input_pdf_text)
    assert result.vendor_name == golden_case.expected.vendor_name
    assert abs(result.total - golden_case.expected.total) < 0.01

def test_ap_agent_tool_call_sequence(golden_case):
    trace = run_agent(golden_case.prompt)
    calls = [c.tool_name for c in trace.tool_calls]
    assert calls == golden_case.expected.tool_sequence
    # Correct final answer via wrong tool path still fails this case
    assert trace.final_answer == golden_case.expected.answer

def test_rag_groundedness(golden_case, judge_model):
    answer = rag_pipeline(golden_case.query)
    score = judge_model.score_groundedness(answer, answer.sources)
    assert score >= 0.8  # threshold tuned against human-labeled sample

Regression testing on prompt and model changes

Every prompt edit and every model version bump is a deploy — run the full golden set against both before and after, and diff the results, not just the aggregate pass rate. A 2% drop in aggregate score can hide a 100% regression on one important subcategory (say, non-English invoices) that's small enough to wash out in the average. Tools like promptfoo exist specifically for this loop — define test cases and assertions in config, run them against multiple prompt/model variants side by side, and get a diff view rather than rebuilding the harness by hand each time. A custom pytest-style harness works just as well if you already have CI wired up and want evals to run alongside your normal test suite.

Online evaluation: what golden sets can't tell you

Offline evals only cover what you thought to test. Production traffic will find inputs you didn't imagine, so pair offline evals with lightweight online signals: thumbs up/down on responses, a sampled queue where a human reviews some percentage of live interactions weekly, and automatic flagging of low-confidence or tool-error traces for review. Feed anything a reviewer marks bad back into the golden dataset — that's how the offline eval set stays representative instead of stagnating at whatever it looked like on day one.

Wrapping up

None of this needs to be elaborate to be useful — a 50-example golden set checked in CI catches most embarrassing regressions, and a weekly human-review sample catches most of what the golden set missed. The mistake is treating evaluation as a one-time pre-launch activity instead of a permanent piece of infrastructure that grows every time production surprises you.