Online Experimentation and A/B Testing

In this tutorial, we provide a quick introduction to online experimentation and A/B testing and give an overview of stats fundamentals needed to understand A/B testing. We then explain how A/B testing is done in an online business. We will conclude by mentioning some of the pitfalls that are common when running online experiments.

What you’ll learn

What is A/B Testing?

• A/B testing on web traffic

• A/B testing on newsletter and email

• A/B testing vs. multivariate testing

Online Experimentation Fundamentals

• Understanding the terminology: Control, treatment, factor, level, A/B, A/A

• Null and alternate hypothesis; Type I and Type II error

• Power and sample size selection; Confidence intervals

• Metrics for evaluation and categories

• Segmentation