{"head":{"title":"What Is an A/B Test? Definition and Example | OceanLab","description":"An A/B test shows two versions of a feature to random groups of users and compares a result. How to run one well, an example and the usual mistakes.","canonical":"https://oceanlab.in/glossary/ab-test"},"crumbs":[{"name":"Home","path":"/"},{"name":"Glossary","path":"/glossary"},{"name":"A/B test","path":"/glossary/ab-test"}],"slug":"ab-test","term":"A/B test","aka":["split test","AB test"],"category":"Hosting and delivery","short":"An experiment that shows two versions of something to random groups of users and measures which one does better on a chosen metric.","body":[["Users are split at random: group A sees the current version, group B sees the change. After enough people have taken part, you compare a metric you chose up front, such as sign-ups or checkout completion, and check whether the difference is bigger than chance would explain."],["A/B tests are usually run through a ",{"href":"/glossary/feature-flag","text":"feature flag"}," system, which decides who sees which version and records it, so results can be tied back to each group."]],"why":["Opinions about what users want are often wrong. An A/B test replaces a debate with evidence and stops a team from shipping changes that quietly hurt the numbers. It only works with enough traffic and a clear question."],"example":["A food delivery app tests showing the delivery fee on the menu page instead of at checkout. Group B sees the fee earlier. Fewer people start checkout, but more of them finish, and total orders per visitor rise, so B is kept."],"inBuild":["We assign users to groups on the server so they stay in the same group across devices, log each exposure as an event, and send results to the ",{"href":"/glossary/data-warehouse","text":"data warehouse"}," for analysis. Every test has a written metric, a minimum sample size and an end date before it starts."],"mistakes":[["Stopping the test the moment it looks good, before the planned sample size."],["Watching many metrics and reporting the one that happened to move."],["Running tests on pages with too little traffic to ever give a clear answer."]],"related":[{"href":"/glossary/feature-flag","label":"Feature flag","detail":"A switch in the code that turns a feature on or off for chosen users at runtime, without deploying new code."},{"href":"/glossary/canary-release","label":"Canary release","detail":"A release method that sends a small share of users to a new version first, watches for problems, and widens the rollout step by step."},{"href":"/glossary/data-warehouse","label":"Data warehouse","detail":"A database built for analysis and reporting, where cleaned data from many systems is kept together in one place."},{"href":"/glossary/observability","label":"Observability","detail":"How well you can understand what a running system is doing, and why, from the data it gives off: logs, metrics and traces."},{"href":"/glossary/mvp","label":"MVP (minimum viable product)","detail":"The smallest version of a product that real users can use, built to test whether the core idea works."}],"tools":[],"links":[{"href":"/pricing/categories/analytics","label":"Analytics pricing","detail":"Product analytics prices, with sources."},{"href":"/software","label":"Software by industry","detail":"Systems we build per industry, with guides and tools."}],"sources":[],"legal":false,"updatedAt":"2026-10-25"}