A/B testing report

(Brian Foley) #1

Understanding results of A/B test email broadcast

We conducted an A/B test with an email broadcast that went to 81,683 subscribers. I applied the A/B test to 100% of recipients and the winning metric was the open rate.

When I look at the report, it’s a bit confusing. First of all, it’s listed twice in my batch of reports. When I click on one of them, it says:

Variation A: (25%) 20421
Variation B: (25%) 20421
Winning email (50%) 40841

And then it says:

A (winner):
3560 Opens (Unique) 5071 Opens (Total) 18.7%
246 Clicks (Unique) 367 Clicks (Total) 1.3%

3541 Opens (Unique) 5022 Opens (Total) 9.2%
245 Clicks (Unique) 366 Clicks (Total) 0.6%

3382 Opens (Unique) 4832 Opens (Total) 17.8%
217 Clicks (Unique) 425 Clicks (Total) 1.1%

Two questions:

  1. How come this report says that the “winning” email went to 50% when I had selected that the test be applied to 100% of recipients?

  2. The percentages appear off. When I take 3,560 from variation A as a percentage of 20,421, I get 17.4%, not 18.7. Is there something wrong with the percentages in our email broadcast reports? Why does the report say 18.7%? From what I can tell, all of the percentages are off.

Any help is very much appreciated as I try to explain these numbers to my supervisor.