A/B Testing

Status: public · Confidence: medium (0.78) · Basis: verified_sources

## TL;DR

A/B testing uses randomized controlled experiments to compare product or marketing variants against predefined metrics.

## Core Explanation

Reliable A/B testing depends less on a magic p-value and more on randomization, predefined metrics, trustworthy instrumentation, and careful interpretation.

## Source-Mapped Facts

- Kohavi, Longbotham, Sommerfield, and Henne describe controlled web experiments as randomized tests that compare variants using user behavior and metrics. ([source](https://doi.org/10.1007/s10618-008-0114-1))
- Trustworthy Online Controlled Experiments recommends defining key metrics and an overall evaluation criterion before interpreting experiment results. ([source](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/trustworthy-online-controlled-experiments/BFFD7CC0B7325B5DCEEFFB1DD9401F7E))
- The Microsoft Research summary of online experimentation emphasizes that randomized experiments are used to evaluate product changes and learn from deployed systems. ([source](https://www.microsoft.com/en-us/research/publication/online-experimentation-at-microsoft/))

## Further Reading

- [Controlled Experiments on the Web: Survey and Practical Guide](https://doi.org/10.1007/s10618-008-0114-1)
- [Trustworthy Online Controlled Experiments](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/trustworthy-online-controlled-experiments/BFFD7CC0B7325B5DCEEFFB1DD9401F7E)
- [Online Experimentation at Microsoft](https://www.microsoft.com/en-us/research/publication/online-experimentation-at-microsoft/)