A/B Testing on Harlan D. Harris

A Collaborative Template for A/B Tests

harlan@harris.name (Harlan Harris) — Thu, 12 Oct 2023 00:00:00 +0000

As I've written about before, as a data scientist supporting a product or marketing team with A/B testing, the job is communication -- helping to translate between business requirements and what we can learn from statistics. I (and many, many others) have found that there is a lot of value in having a document, shared among the team that is running the test. Some A/B testing tools include some limited workflow for collaboration on testing (e.g., Amplitude Experiments has pre-test, monitoring, and results tabs with useful graphs and summary statistics), but I think the most important thing is to have a tool that helps the team get aligned by forcing them to write things down.

The Five Types of A/B Test Decisions

harlan@harris.name (Harlan Harris) — Fri, 07 Jul 2023 00:00:00 +0000

Recently I wrote a blog post that mentioned “Superiority” as a type of A/B test decision. In this post I want to talk about all five types of A/B test decision that I think are relevant. This is an adaptation and extension of a talk I gave last year at the Quant UX conference (it’s a great event, you should check it out.) Note that I go into a little more statistical detail here, although most of the below is readable by non-data scientists.

p Values Are Useful for A/B Tests, Sometimes

harlan@harris.name (Harlan Harris) — Tue, 02 May 2023 00:00:00 +0000

The "best practice", when evaluating the results of an online controlled experiment (A/B test), is to use classical statistical tests, proceeding with a change if (and only if) the result of the test includes a p value of less than 0.05. But, the American Statistical Association (ASA) said in a prominent 2016 statement that "...business... decisions should not be based only on whether a p-value passes a specific threshold." Wait, what? Are we making bad decisions from A/B tests? Should we stop using p values and do something else?

Communicating A/B Test Results for Conversion Rates with Ratios and Uncertainty Intervals

harlan@harris.name (Harlan Harris) — Sat, 20 Aug 2022 00:00:00 +0000

A/B testing is a tool for supporting decision-making in business, and so in addition to getting the statistics right, it’s really important to communicate well with the non-statisticians who will have the final say on the go/no-go decision. Most A/B tests in practice are testing ratios, conversion rates of various sorts – say, the proportion of people who visit your web site who buy at least one pair of shoes. Although the underlying data is just four numbers (visitors and purchasers, from the control group and the test group), there are lots of ways to compute statistics and present those numbers. Not all of those ways are ideal for supporting communication.

A/B Testing and Product Ratings, Part 1: Delays and Bias

harlan@harris.name (Harlan Harris) — Sun, 22 May 2022 00:00:00 +0000

Suppose you’re a data scientist at an e-commerce web site that sells shoes, responsible for supporting A/B tests. Many A/B tests are easy, and there are a number of companies that sell tools that make the easy cases as simple as clicking a few buttons and looking at pretty graphs. But A/B tests can get statistically complex surprisingly quickly, which is why hiring data scientists with a strong statistics background can make a big difference in the quality of decisions.