ABSTRACT
A/B experiments are commonly used in research to compare the
effects of changing one or more variables in two different experimental groups – a control group and a treatment group. While the
benefits of using A/B experiments are widely known and accepted,
there is less agreement on a principled approach to creating software infrastructure systems to assist in rapidly conducting such
experiments. We propose a four-principle approach for developing
a software architecture to support A/B experiments that is domain
agnostic and can help alleviate some of the resource constraints currently needed to successfully implement these experiments: the
software architecture (i) must retain the typical properties of A/B
experiments, (ii) capture problem solving activities and outcomes,
(iii) allow researchers to understand the behavior and outcomes of
participants in the experiment, and (iv) must enable automated
analysis. We successfully developed a software system to encapsulate these principles and implement it in a real-world A/B
experiment.
Conducting AB Experiments with a Scalable Architecture – 2023
