Uberized A/B Testing for Websites… NO!

TL;DR: Conclusion

Traditional A/B Testing is still the best and most robust approach for determining which web page, text copy, or button color yields the optimal click-through and conversions rates!

For those who are not familiar with A/B tests, they typically show two versions, A and B, of a page randomly (i.e., 50% of the time). Overtime, either version A or B will attract more button clicks (CTRs) or class sign-ups (conversions).

Why This Post?

Back in December 2014, we began researching A/B testing as a way to test the effectiveness of various squeeze pages for an upcoming Google AdWords campaign. We defined “effectiveness” as course sign-ups (conversions) and Click Through Rates (CTR) as the quality of our Adwords copy. Fortunately, Google auto-optimizes the display of AdWords copy and over time only displays ads with the best CTRs.

Terminology

The problem that A/B or (A, B, C… n) testing attempts to solve is known as the multi-armed bandit problem. It attempts to determine which object (or machine) produces the highest reward (and least amount of losses or regrets). Where the “object” can be a drug that may cure a fatal disease, a casino machine, or the color of a “click to buy” button on a web page!

Statisticians have numerous algorithms to determine which “object” will produce the highest reward with the least amount of regrets. Some of these algorithms are epsilon-first (traditional A/B testing), epsilon-greedy (brilliantly articulated by Steve Hanov in a 2012 blog post), and Thompson Sampling (better known the as Bayesian Control Rule or Bayesian Bandits when applied to dynamic, very real-world environments).

Battle of the Statisticians: My Multi-armed Bandit Algo is Better than Yours

From 2012 to January 2015 there was an interesting discussion happening online about which multi-arm bandit algo was the best. It was an often heated but highly educational debate. It provided much needed insight into statistics and the pros and cons of various algorithms.

Timeline and Details

The debate (or should we say meme?) started on May 28, 2012 when Steve Hanov published his epic blog post called 20 Lines of Code That Will Beat A/B Testing Every Time. Steve asserted that an epsilon-greedy algo was superior to the traditional A/B testing (i.e., espsilon-first algo). One enthusiastic reader even created a WordPress plugin that allowed the casual user to use the algo on their own blog!

Afterwards, on Jun 1, 2012 Paras Chopra of Visual Website Optimizer published a counter-argument titled Why Multi-armed Bandit Algorithm is Not “Better” than A/B Testing. Interestingly, Paras was correct in his conclusion, but the path (statistical experiments) he used to arrive at his conclusion may have been somewhat flawed. Do read the comments section of his post. To Paras’ credit he released his Python code for the world to see and pick apart!

As if this weren’t enough excitement, on June 3, 2012 Chris Tuccio published a response to Paras called Why Multi-armed Bandit Algorithms are Superior to A/B Testing.

Conclusion: All Things Being Equal

After researching this ad nauseam, here are our takeaways:

For rewards that depend on a slew of varying inputs that we cannot control, the simplistic and non-stochastic algorithm (i.e., our vanilla, epsilon-first, A/B testing) is the best choice. Here are a few of the variables that we cannot control but impact CTRs:

Variance in sample behavior over time (particular days of the week and times of day may yield differing CTRs and conversions)
A significant time delay between exposure (i.e, first time visiting a page) and result (i.e., finally deciding to click a button or make a purchase)
Sample behaviour is influenced by factors not related to page design (e.g., a well televised documentary that causes a specific user demographic to visit your website more. Or alternatively longer page load times causing users to abandon your site before the page renders).

Ben Tilly’s comment on one of the A/B testing Hacker News threads captures many of these issues.

Interestingly on January 27, 2015, statistician Chris Stuccio finally concludes that A/B testing is good enough for web page optimization since one or more of the following assumptions may fail:

The samples drawn from each arm of the bandit are Independent and Identically Distributed

The conversion rates for each arm do not change over time

There is no delay between pulling an arm and observing the result. Or, more precisely, if there is a delay, it’s shorter than the delay between opportunities to pull an arm.

– Chris Stuccio on Don’t use Bandit Algorithms – they probably won’t work for you

So perhaps for highly controlled, scientific experiments (e.g, drug efficacy clinical trials) where the above stipulations are true, a probability-based algo such as Bayesian Bandits would be superior.