What is Bayesian A/B Testing and Why is it the Best Choice for Marketers?

4th Sept 2017

On launching a new campaign, marketers need to know whether it works.

In response, A/B testing has become widespread in marketing. It is an intuitive and effective method to evaluate a particular change to the customer experience.

A/B testing requires statistical inference. As you collect data and compare the performance of your A and B variants, you need to assess the probability that your data accurately reflect the difference (or lack thereof) between the two.

This is easily understood with a simple and familiar example. Let us say you flip a coin (Coin A) 100 times and turn up exactly 50 heads. Then you flip another coin (Coin B) 100 times and turn up 59 heads. You have a very strong vested interest in knowing whether you just got lucky in those 100 flips of Coin B because otherwise, Coin B really is a special coin and you can keep flipping it all the way to the bank.

Evaluating Coin B’s good luck poses much the same problem as evaluating variants of a campaign, or in the initial pilot stage simply the comparison of a campaign to its absence. As the variants generate performance data, we need to be able to tell our clients not simply which is doing better but how likely it is that the better variant is better by luck or by design.

Getting Answers Quickly

We have designed Amigo to enable agile marketing. This is because marketers today don’t just need to know whether a campaign achieves its stated objectives, but they need to know this as quickly as possible. When we conduct A/B testing therefore, we use a Bayesian framework.

Bayes Theorem
wikimedia commons: mattbuck
A Bayesian approach uses what we already know to help evaluate our campaign variants in light of their performance. Crucially, it enables experiments to benefit from existing data while remaining hypothesis-driven. This makes a Bayesian framework perfect for marketers who need to draw conclusions as soon as enough data has been collected.

To explain further it is worth noting that Bayesian statistics are generally understood in contrast to frequentist statistics.

xkcd Bayesian v Frequentist


The differences between the two are the subject of deeply complex theoretical debate among statisticians. However, in practice we have found that for the A/B testing of marketing campaigns, Bayesianism possesses several advantages over frequentism.

For a start, Bayesian statistics simply give more useful results. Frequentist A/B testing works by attempting to disprove the hypothesis that A and B are truly the same (calculating the p-value). This means that frequentism cannot answer a number of questions that a Bayesian framework can, such as “how likely is it that B is better than A and by how much?” or “what is the downstream effect on revenue?”

Even more important than the quality of the results is that a Bayesian approach is much better suited to agile methods of working. As a Bayesian framework can constantly assess not just the answers but the certainty of our answers, it can provide the quickest reliable answer to the key question, “is the test done yet?”

While frequentist testing requires the length of the test to be defined in advance, Bayesian testing does not. It can calculate the potential dangers of ending the test (the loss value) at any point, and gives a constantly updated probability of either variant being better and by how much. Ending the test early can be disastrous for frequentist A/B testing. A Bayesian approach therefore provides us with much greater flexibility during the experiment.

Bayesian A/B testing is therefore an integral part of our ability to enable marketers to be agile by closing the marketing execution gap.

Amigo A/B Testing

To find out more about how Amigo enables marketers, read our analysis of the marketing execution gap.

Further reading

020 3940 4650