Understanding results of A/B test email broadcast
We conducted an A/B test with an email broadcast that went to 81,683 subscribers. I applied the A/B test to 100% of recipients and the winning metric was the open rate.
When I look at the report, it’s a bit confusing. First of all, it’s listed twice in my batch of reports. When I click on one of them, it says:
Variation A: (25%) 20421
Variation B: (25%) 20421
Winning email (50%) 40841
And then it says:
A (winner):
3560 Opens (Unique) 5071 Opens (Total) 18.7%
246 Clicks (Unique) 367 Clicks (Total) 1.3%
3541 Opens (Unique) 5022 Opens (Total) 9.2%
245 Clicks (Unique) 366 Clicks (Total) 0.6%
B:
3382 Opens (Unique) 4832 Opens (Total) 17.8%
217 Clicks (Unique) 425 Clicks (Total) 1.1%
Two questions:
-
How come this report says that the “winning” email went to 50% when I had selected that the test be applied to 100% of recipients?
-
The percentages appear off. When I take 3,560 from variation A as a percentage of 20,421, I get 17.4%, not 18.7. Is there something wrong with the percentages in our email broadcast reports? Why does the report say 18.7%? From what I can tell, all of the percentages are off.
Any help is very much appreciated as I try to explain these numbers to my supervisor.