AI · LLM

Benchmarking LLM Apps

Public leaderboards tell you which model is good at a benchmark. They cannot tell you which model is good at your task — only your own eval set can do that.

John Kihiu12 min read

When a new model launches, the leaderboards light up and the temptation is to pick whatever tops MMLU or the latest arena. But a model's rank on a general benchmark is a weak predictor of how it performs on your specific task with your prompts and your data. The benchmark that matters is the one you build.

Build a task-specific eval set

Collect real inputs from your domain — the tickets you classify, the documents you summarise, the questions users actually ask — and pair them with known-good outputs. A hundred representative cases from your own traffic will separate models more reliably than any public score, because they test the thing you are actually shipping.

Measure more than accuracy

A model choice is a trade-off across at least three axes, and a benchmark that reports only quality hides two of them:

DimensionWhat to measure
QualityTask success rate on your eval set — pass/fail assertions or judge scores.
Latencyp50 and p95 end to end, including streaming time-to-first-token.
CostAverage tokens and USD per request at your real prompt sizes.

The best model on quality is often the slowest and most expensive. Seeing all three side by side is what lets you say "the cheaper model is 3% worse but half the cost and twice as fast, and that trade is right for this feature."

Pairwise comparison

For open-ended tasks where there is no single correct answer, absolute scores are noisy. Pairwise comparison — show a judge (or a human) two models' outputs for the same input and ask which is better — is more reliable and produces an intuitive win-rate. Run it blind so the judge cannot tell which model produced which answer.

Beware benchmark contamination

Popular public benchmarks leak into training data, so a high public score can partly reflect memorisation rather than capability. Your private eval set, drawn from your own recent data, is immune to this — which is another reason to trust it over the leaderboard.

Re-run your eval set whenever you consider a model change, a prompt change, or a new provider. A benchmark is not a one-time model-selection ritual; it is the standing measurement that makes every future change a decision backed by numbers rather than vibes.

John Kihiu
Acumatica ERP Developer · Laravel Engineer

Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.