A/B testing lets you settle product and growth debates with data instead of authority: show variant A to half your users, B to the other half, and measure which performs better. Done rigorously it is how you improve conversion, activation, and pricing with confidence. Done casually — small samples, called early, on trivial changes — it produces a stream of false wins that are worse than no testing at all, because you act on them.
One primary metric, chosen first
Decide before the test what single metric defines success — signup conversion, activation, trial-to-paid, expansion. Commit to it in advance so you cannot rationalise a favourable secondary metric after the fact. Watch guardrail metrics you must not harm, but let one primary metric decide the winner. A test with a metric chosen after the results are in is not an experiment; it is a search for a story.
Run it long enough
SaaS traffic is noisy and conversion differences are often small, so tests need real sample sizes and real duration to distinguish signal from luck. Estimate the sample you need up front based on your baseline rate and the effect size worth detecting, and run until you reach it. Run for full weeks, too, to average over day-of-week effects — a test that spans only weekdays measures weekday users, not all of them.
The traps that fake wins
- Peeking — checking results repeatedly and stopping when it looks significant inflates false positives; decide the duration and wait.
- Multiple comparisons — test enough variants or metrics and one will look significant by chance; correct for it or focus.
- Trivial changes — button colours rarely move a business metric; test changes big enough to matter.
- Ignoring segments — a variant can win overall while hurting a key segment; check the breakdown.
If a test does not reach significance, the honest conclusion is 'no detectable difference,' not 'let it run until it does.' That is genuinely useful — it means the change did not matter, so ship whichever variant is simpler or cheaper and move on. Chasing significance by extending a flat test is how you manufacture false positives.
SaaS A/B testing turns funnel decisions into evidence when run with discipline: one pre-chosen primary metric, a sample size and duration decided up front, and vigilance against peeking, multiple comparisons, and trivial tests. Test things big enough to move the business, respect the statistics, and accept a null result as an answer — that is how experimentation compounds into real growth instead of a pile of noise.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.