Glossary
A/B Testing
Meaning and details
An A/B test is a bet settled by reality: show version A to half your visitors, version B to the other half, keep everything else identical, and let the conversion metric declare the winner. The rigor lives in the details...
A/B Testing meaning
A controlled experiment comparing two versions of a page or flow on live traffic to see which performs better on a chosen metric.
What is A/B Testing?
An A/B test is a bet settled by reality: show version A to half your visitors, version B to the other half, keep everything else identical, and let the conversion metric declare the winner. The rigor lives in the details, one changed variable, a predefined success metric, enough traffic for the difference to be statistically real rather than noise, and a fixed duration so you cannot stop the moment luck tilts the chart.
Why it matters to an indie builder: opinion is the most expensive thing in a small team. Founders argue about headlines, pricing pages and button colors because there is no evidence, and the argument recurs every redesign forever. A/B testing converts those recurring debates into one-time experiments with permanent answers. It is also the discipline that protects you from yourself: everyone's intuition, including yours, is wrong about roughly half of what it feels certain about, and the test is the only instrument that says which half.
A concrete example: a pricing page at 2.9 percent trial conversion. Hypothesis: annual-first display beats monthly-first. A: monthly $19 prominent; B: annual $190 prominent with two months free. The test needs both arms to reach about 1,700 visitors each before reading. Result: B converts 3.8 percent and shifts 41 percent of buyers to annual, which also cuts effective churn in half because annual subscribers cancel far less. One experiment, two compounding wins, and the decision is never argued again, the losing variant is retired with the evidence attached.
Common mistakes: peeking early and stopping at the first significant-looking gap, which turns noise into confident lies, testing five changes at once and learning nothing actionable when one wins, running tests on traffic volumes that cannot reach significance in months (below roughly 1,000 conversions a month, test bigger swings or skip testing and ship), measuring vanity endpoints like clicks when the business metric is signups or revenue, and ignoring novelty effects, the new version often spikes and regresses, which is why duration is fixed. PostHog, one of the analytics blocks on IndieBoot, ships experiments natively.