A/B test
Also called split test, AB test.
An experiment that shows two versions of something to random groups of users and measures which one does better on a chosen metric.
Last updated
What it is.
Users are split at random: group A sees the current version, group B sees the change. After enough people have taken part, you compare a metric you chose up front, such as sign-ups or checkout completion, and check whether the difference is bigger than chance would explain.
A/B tests are usually run through a feature flag system, which decides who sees which version and records it, so results can be tied back to each group.
Why it matters
Opinions about what users want are often wrong. An A/B test replaces a debate with evidence and stops a team from shipping changes that quietly hurt the numbers. It only works with enough traffic and a clear question.
In a product build
We assign users to groups on the server so they stay in the same group across devices, log each exposure as an event, and send results to the data warehouse for analysis. Every test has a written metric, a minimum sample size and an end date before it starts.
Common mistakes.
- 01
Stopping the test the moment it looks good, before the planned sample size.
- 02
Watching many metrics and reporting the one that happened to move.
- 03
Running tests on pages with too little traffic to ever give a clear answer.
Related terms.
- Feature flagA switch in the code that turns a feature on or off for chosen users at runtime, without deploying new code.
- Canary releaseA release method that sends a small share of users to a new version first, watches for problems, and widens the rollout step by step.
- Data warehouseA database built for analysis and reporting, where cleaned data from many systems is kept together in one place.
- ObservabilityHow well you can understand what a running system is doing, and why, from the data it gives off: logs, metrics and traces.
- MVP (minimum viable product)The smallest version of a product that real users can use, built to test whether the core idea works.
Guides and pages
Browse the full glossary or read answers to common questions.
