When a new model launches, the leaderboards light up and the temptation is to pick whatever tops MMLU or the latest arena. But a model's rank on a general benchmark is a weak predictor of how it performs on your specific task with your prompts and your data. The benchmark that matters is the one you build.
Build a task-specific eval set
Collect real inputs from your domain — the tickets you classify, the documents you summarise, the questions users actually ask — and pair them with known-good outputs. A hundred representative cases from your own traffic will separate models more reliably than any public score, because they test the thing you are actually shipping.
Measure more than accuracy
A model choice is a trade-off across at least three axes, and a benchmark that reports only quality hides two of them:
| Dimension | What to measure |
|---|---|
| Quality | Task success rate on your eval set — pass/fail assertions or judge scores. |
| Latency | p50 and p95 end to end, including streaming time-to-first-token. |
| Cost | Average tokens and USD per request at your real prompt sizes. |
The best model on quality is often the slowest and most expensive. Seeing all three side by side is what lets you say "the cheaper model is 3% worse but half the cost and twice as fast, and that trade is right for this feature."
Pairwise comparison
For open-ended tasks where there is no single correct answer, absolute scores are noisy. Pairwise comparison — show a judge (or a human) two models' outputs for the same input and ask which is better — is more reliable and produces an intuitive win-rate. Run it blind so the judge cannot tell which model produced which answer.
Popular public benchmarks leak into training data, so a high public score can partly reflect memorisation rather than capability. Your private eval set, drawn from your own recent data, is immune to this — which is another reason to trust it over the leaderboard.
Re-run your eval set whenever you consider a model change, a prompt change, or a new provider. A benchmark is not a one-time model-selection ritual; it is the standing measurement that makes every future change a decision backed by numbers rather than vibes.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.