A/B Testing LLM Apps

Your eval set is a proxy for reality, and proxies drift. A prompt that scores higher offline can perform worse live because real users phrase things your dataset never anticipated. A/B testing closes that gap by measuring variants against each other on live traffic with a real outcome metric.

Split traffic cleanly

Assign each user (or session) deterministically to a variant — hash the user id so the same person always gets the same version — and log which variant served every request. Deterministic assignment matters: if a user flips between variants mid-session, you cannot attribute their behaviour to either one. The variant tag on each logged request is what makes the whole analysis possible later.

import hashlib

def variant(user_id, experiment, split=0.5):
    h = hashlib.sha256(f"{experiment}:{user_id}".encode()).hexdigest()
    bucket = int(h[:8], 16) / 0xFFFFFFFF
    return "B" if bucket < split else "A"

v = variant(user_id, "classify_prompt_v4")
spec = PROMPTS["classify@4" if v == "B" else "classify@3"]

One primary metric, guarded

Choose a single primary metric that reflects real value — task success, a thumbs-up rate, downstream conversion, reduced escalation — and commit to it before the test. Then track guardrail metrics you must not regress: cost per request and latency. A variant that lifts quality 2% while doubling cost is usually not a win, and only side-by-side guardrails reveal that.

Run until it means something

LLM output is noisy, so small samples lie. Decide the sample size or duration up front and resist calling the result early because it looks good on day one. If the difference is not statistically meaningful, the honest conclusion is "no detectable difference" — which is still useful, because it means you can pick the cheaper or faster variant with confidence.

The workflow that scales is offline evals as the fast gate for every change, and A/B tests as the slower, higher-fidelity check for the changes that matter. Together they let you improve an LLM feature continuously without guessing whether each change actually helped.