Averages can be misleading. For instance, if we look at results from Campaign A and Campaign B seen below, we are tempted to say with certainty that Campaign B is the more successful campaign. After all the average revenue and the response rate from Campaign B is considerably higher than that of Campaign A. However, we must examine whether the higher average and the higher response rate are significant or if they are quite possibly due to random chance.
Looking at the graphs below showing the percentage of customers at each revenue level, we see that customers responding to Campaign A are tightly grouped between $10 and $30, while those responding to Campaign B are considerably more spread out. There are a large number of customers at the $5 and $10 mark in Campaign B, but a small number of customers with large revenue amounts that pulls the average up. When statistical tests are run, they tell us that Campaign B's average revenue is not statistically significantly better than Campaign A's average revenue.
Delving further into the campaign data, we see that Campaign A was sent to 1200 customers, while Campaign B was sent to 100 customers. Analyzing the response rates for statistical significance, we again find that Campaign B is not significantly better than Campaign A. Because of the large difference in number of customers in the two campaigns, the difference in response rate must be greater than 3.3% to be significant. It is important to keep this in mind when selecting customer lists for test and control groups of a campaign or any two or more campaigns you wish to directly compare.
These statistics tell us that Campaign B's higher average revenue and higher response rate was almost as likely to be pure chance as it was to be because of superior campaign tactics. Had a business owner made decisions to scrap the tactics used in Campaign A in favour of those used in Campaign B based on average revenue and response rate alone, they may have been disregarding an excellent campaign.
Find out how 3 Sheep Analytics can help improve your organization with post-campaign analysis and the insights it provides.