TL;DR: Conclusion

Traditional A/B Testing is still the best and most robust approach for determining which web page, text copy, or button color yields the optimal click-through and conversions rates!

For those who are not familiar with A/B tests, they typically show two versions, A and B, of a page randomly (i.e., 50% of the time). Overtime, either version A or B will attract more button clicks (CTRs) or class sign-ups (conversions).

Why This Post?

Back in December 2014, we began researching A/B testing as a way to test the effectiveness of various squeeze pages for an upcoming Google AdWords campaign. We defined “effectiveness” as course sign-ups (conversions) and Click Through Rates (CTR) as the quality of our Adwords copy. Fortunately, Google auto-optimizes the display of AdWords copy and over time only displays ads with the best CTRs.


The problem that A/B or (A, B, C… n) testing attempts to solve is known as the multi-armed bandit problem. It attempts to determine which object (or machine) produces the highest reward (and least amount of losses or regrets). Where the “object” can be a drug that may cure a fatal disease, a casino machine, or the color of a “click to buy” button on a web page!

Statisticians have numerous algorithms to determine which “object” will produce the highest reward with the least amount of regrets. Some of these algorithms are epsilon-first (traditional A/B testing), epsilon-greedy (brilliantly articulated by Steve Hanov in a 2012 blog post), and Thompson Sampling (better known the as Bayesian Control Rule or Bayesian Bandits when applied to dynamic, very real-world environments).