What is Bayesian A/B Testing and Why is it the Best Choice for Marketers?4th Sept 2017
On launching a new campaign, marketers need to know whether it works.
In response, A/B testing has become widespread in marketing. It is an intuitive and effective method to evaluate a particular change to the customer experience.
A/B testing requires statistical inference. As you collect data and compare the performance of your A and B variants, you need to assess the probability that your data accurately reflect the difference (or lack thereof) between the two.
This is easily understood with a simple and familiar example. Let us say you flip a coin (Coin A) 100 times and turn up exactly 50 heads. Then you flip another coin (Coin B) 100 times and turn up 59 heads. You have a very strong vested interest in knowing whether you just got lucky in those 100 flips of Coin B because otherwise, Coin B really is a special coin and you can keep flipping it all the way to the bank.
Evaluating Coin B’s good luck poses much the same problem as evaluating variants of a campaign, or in the initial pilot stage simply the comparison of a campaign to its absence. As the variants generate performance data, we need to be able to tell our clients not simply which is doing better but how likely it is that the better variant is better by luck or by design.
Getting Answers Quickly
We have designed Amigo to enable agile marketing. This is because marketers today don’t just need to know whether a campaign achieves its stated objectives, but they need to know this as quickly as possible. When we conduct A/B testing therefore, we use a Bayesian framework.
To explain further it is worth noting that Bayesian statistics are generally understood in contrast to frequentist statistics.
The differences between the two are the subject of deeply complex theoretical debate among statisticians. However, in practice we have found that for the A/B testing of marketing campaigns, Bayesianism possesses several advantages over frequentism.
For a start, Bayesian statistics simply give more useful results. Frequentist A/B testing works by attempting to disprove the hypothesis that A and B are truly the same (calculating the p-value). This means that frequentism cannot answer a number of questions that a Bayesian framework can, such as “how likely is it that B is better than A and by how much?” or “what is the downstream effect on revenue?”
Even more important than the quality of the results is that a Bayesian approach is much better suited to agile methods of working. As a Bayesian framework can constantly assess not just the answers but the certainty of our answers, it can provide the quickest reliable answer to the key question, “is the test done yet?”
While frequentist testing requires the length of the test to be defined in advance, Bayesian testing does not. It can calculate the potential dangers of ending the test (the loss value) at any point, and gives a constantly updated probability of either variant being better and by how much. Ending the test early can be disastrous for frequentist A/B testing. A Bayesian approach therefore provides us with much greater flexibility during the experiment.
Bayesian A/B testing is therefore an integral part of our ability to enable marketers to be agile by closing the marketing execution gap.
To find out more about how Amigo enables marketers, read our analysis of the marketing execution gap.
Founders like Larry Page and Zuck ask “why is this important strategically?” and “what is the user experience?” with good reason. Innovation is necessarily strategic and intimately linked with user experience.read more
In our latest breakfast, we addressed how theories of customer psychology can be harnessed in finance.read more
We invited a select group of travel marketers to breakfast to discuss the use of customer psychology to win the last mile.read more