{"id":9951,"date":"2017-05-24T18:46:38","date_gmt":"2017-05-24T13:16:38","guid":{"rendered":"http:\/\/ucanalytics.com\/blogs\/?p=9951"},"modified":"2017-06-04T14:53:50","modified_gmt":"2017-06-04T09:23:50","slug":"bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3","status":"publish","type":"post","link":"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/","title":{"rendered":"Bayesian Statistics to Improve A\/B Testing – Digital Marketing Case Study Example (Part 3)"},"content":{"rendered":"
\n

\"\"<\/a>Welcome back to the\u00a0digital marketing case study example<\/a><\/strong>. In this case study, you are helping your client to identify an ad that will generate them the maximum return on marketing investment (ROMI). In the previous part, we discussed A\/B testing<\/strong><\/a> which is the\u00a0prevalent method used to compare the effectiveness of marketing effort.\u00a0You are an expert in digital analytics in this case study example. Based on your past experience, you have a strong belief about the performance of this digital marketing effort (i.e. click rates etc.). You expect the click rate to be between 12-18% against the possible 0-100% range. However, the biggest frustration for you with A\/B testing is that you can’t use that prior information or belief to improve your results.<\/p>\n

In this part, you will use Bayesian statistics to improve the performance of A\/B testing using your prior knowledge. The next part will extend the concepts of Bayesian statistics to create an intelligent campaign and digital marketing design. This is kind of similar to creating an artificial intelligence for the purpose.<\/p>\n

Before we start our case study example, let’s understand the underlying principals of Bayesian thinking and statistics<\/a><\/strong>.<\/p>\n

Bayesian Statistics to Improve A\/B Testing<\/span><\/h2>\n

Bayesian statistics may seem intimidating to many but the ideas behind Bayesian thinking are extremely intuitive and simple. For me, this quote by Isaac Newton captures the essence of Bayesian thinking.<\/p>\n

If I have seen further than others, it is by standing upon the shoulders of giants.<\/p>\n

– Isaac Newton<\/p>\n<\/blockquote>\n

Newton was referring to giants like Galileo, Copernicus, Aristotle, and many others for his work on celestial motion and gravity. It makes sense\u00a0because learning and knowledge are\u00a0incremental processes. Even the greatest scientists use the work of their predecessors to develop new theories and knowledge. The next generation is usually an extension of the previous generations.<\/p>\n

\"Bayesian<\/a>

Bayesian statistics and standing upon the shoulders of giants – by Roopam<\/p><\/div>\n

You may find it surprising that the prevalent methods for analysis don’t work in a similar incremental fashion of building on top of the existing\u00a0knowledge. These methods are based on Fisherian statistics aka the frequentist approach. In the frequentist approach, each\u00a0analysis is an isolated island and it is left\u00a0to humans to make the connections between different analyses. Traditional A\/B testing or hypothesis testing is a form of the frequentist approach. Bayesian statistics, on the other hand, use priors similar to how Newton used the works of other scientists prior to him. Bayesian statistics incorporate the prior knowledge and the new evidence (data) to generate new knowledge. This new knowledge is known as posterior in the Bayesian parlance.<\/p>\n

Despite being intuitive, the biggest challenge for Bayesian statistics is that,\u00a0unlike traditional\u00a0statistics, it is computationally expensive. This means that Bayesian statistics requires higher computation power for some of its applications using Markov Chain Monte Carlo methods (MCMC). We will explore MCMC in some other posts on YOU CANalytics.\u00a0For now, let’s explore methods that make Bayesian calculation less expensive and easy to implement i.e.<\/p>\n

Conjugate Priors<\/span><\/h2>\n

Let’s try to understand conjugate priors in an intuitive way. Look at this illustration:<\/p>\n

\"\"<\/a>

Illustration for Conjugate Priors- by Roopam<\/p><\/div>\n

Conjugate priors (read prior probability distributions) form a harmonic relationship with the distributions of data (evidence) to produce easy to decipher posterior distributions. This is similar to mixing distilled water from two different rivers and getting more distilled water which has the same properties as the prior distribution. \u00a0Non-conjugate priors, on the other hand, is like mixing soil and water. In this case, the posterior is like sticky mud which is hard to work with.<\/p>\n

If we work with conjugate priors, our job with Bayesian inference becomes quite easy and less computationally expensive. For non-conjugate priors, we use computationally expensive methods such as Markov Chain Monte Carlo (MCMC).<\/p>\n

In this case study example, you will work with the conjugate prior of the email campaign performance. You had identified the click rate as the performance metric for success of the digital ads. A click is a binary event that can have just two possible values i.e. clicked or not clicked. Such binary events are described as Bernoulli experiments. Yes, the famous coin tosses are also Bernoulli experiments as are so many other applications in business.<\/p>\n

\"\"<\/a><\/p>\n

For clicks, the likelihood or evidence\/data distribution of Bernoulli experiments is defined by this function.<\/p>\n

\"p(\#Clicks|\theta<\/pre>\n
Here,\u00a0\u03b8 is the real click rate.\u00a0Bernoulli experiments take Beta distribution as the conjugate prior.<\/p>\n
Beta Distribution – Conjugate Prior for\u00a0Click Rate<\/span><\/h2>\n
If you are panicking at the mention of beta distribution, I could totally understand. But trust me, it is a really cool function. The shape of a beta distribution depends on the values of two constants \u03b1 and \u03b2. Different values of \u03b1 and \u03b2 produce different shapes of probability distributions. Moreover, a beta distribution always lies between the value 0 and 1. This is perfect for us since click rate also always lies between 0 and 1.<\/p>\n
\"p(\theta<\/pre>\n
Here, B(\u03b1,\u03b2) is a\u00a0constant to normalize the distribution.\u00a0In this diagram, 4 different beta distributions are displayed with different values of\u00a0\u03b1 and \u03b2.<\/p>\n
 <\/p>\n
\"\"<\/a><\/p>\n
You must have noticed how beta distributions starts to look like a bell curve or normal distribution as the value of \u03b1 and \u03b2 starts to get bigger. Also notice that with the greater values of\u00a0\u03b1 and \u03b2, the bell curves are getting thinner or have smaller standard deviations. Now, let me reveal the coolest property of beta distributions for prior and posterior probabilities when the data is obtained\u00a0from a Bernoulli experiment. Remember, prior is old knowledge and posterior is new knowledge in presence of new evidence or data. For click rates,\u00a0p(\u03b8) is the prior probability; p(#Click|\u03b8) is the probability of the evidence with the given prior or \u03b8; p(\u03b8|#Click) is the posterior probability. Bayes rule creates a straightforward multiplicative relationship between prior, evidence, and posterior as:<\/p>\n
\"p(\theta|\#Click)=p(\#Click|\theta)\times<\/pre>\n
The posterior, thus, can be approximated by\u00a0p(\u03b8) and\u00a0p(#Click|\u03b8) equation discussed earlier.<\/p>\n
\"p(\theta|\#Click)\sim\u00a0<\/pre>\n
This is simplified to:<\/p>\n
\"p(\theta|\#Click)\sim\u00a0<\/pre>\n
Notice, the posterior in this case is another beta distribution. The posterior beta distribution has a very simple relationship with the prior and number of clicks!<\/p>\n
\"\alpha_{posterior}=\alpha_{prior}+\#Clicks<\/pre>\n
\"\beta_{posterior}=\beta_{prior}+\#No\<\/pre>\n
We will experience the power of this simple equation in the next segment when we will go back to our digital marketing case study example.<\/p>\n
Digital Marketing – Case Study Example<\/span><\/h2>\n
In this case study example, you are helping your client to improve their digital marketing efforts. Your client is a charity organization that is working on an email campaign to ask people to donate money to the needy in Africa. These are the three ads that they had sent across to three different sets of recipients. The performance measure, as discussed in the last post on A\/B testing<\/a><\/strong>, is the click rate. Before you read further, just look at these 3 ads and decide which one will perform better than others.\"\"<\/a><\/p>\n
You are an expert in digital marketing and advertising. In your experience, you have noticed that ads with humans\u00a0perform much better than ads with abstract images. The abstract image, in this case, is the map of Africa in A. Your opinion is that ad C will outperform ad B and A. Also you believe, B will do better than A.<\/p>\n
Prior Distributions\u00a0– Old Knowledge<\/span><\/h4>\n
You expect\u00a0click rate for\u00a0C to follow a bell curve with the mean of 17% and standard deviation of 4%. Similarly, you expect A and B to have the respective mean click rates of 15% and 16%. They also have the standard deviation of 4%. This expert opinion (bell curves or normal distributions) can be easily approximated as beta distributions as displayed in the graph below.<\/p>\n
You could find the complete\u00a0R Code for Bayesian AB Testing<\/a>\u00a0<\/strong>on this link. All the plots and analyses to follow are captured in this code.\u00a0\"\"<\/a><\/p>\n
These 3 prior distributions for ad A, B and C are beta distributions (they look identical to bell curves assumptions you had made). For ad A, you had estimated \u03b1 = 11.8 and \u03b2 = 66.9 to produce the beta distribution with the assumption of a bell curve with the mean of 15% and sd of 4%. Similarly, ad B has\u00a0 \u03b1 = 13.3 and \u03b2 = 69.7. Moreover,\u00a0ad c has\u00a0 \u03b1 = 14.8 and \u03b2 =\u00a072.4. You could get these numbers by running the R-code<\/a><\/strong>\u00a0I had shared earlier.<\/p>\n
You have also estimated with the help of Monte Carlo simulation that ad C will outperform B\u00a0in 57% cases. Similarly, C will do better than A for click rates in 64% cases. This prior information is not good enough for you to be conclusive about the performance of these ads. This is precisely the reason you why had launched this campaign\/experiment with these ads being tested on a large number of recipients.<\/p>\n
Posterior Distribution – New Knowledge in the Presence of New Evidence (Data)<\/span><\/h4>\n
If you recall, last week the IT team of you client shared these results for the click rate in the first hour after the launch of the campaign.<\/p>\n\n\n\n\n\n\n
\u00a0Ads<\/span><\/strong><\/td>\nTotal Emails Sent<\/span><\/strong><\/td>\nClicks (1st hour)<\/span><\/strong><\/td>\nOpens (1st hour)<\/span><\/strong><\/td>\n<\/tr>\n
A<\/strong><\/td>\n5500<\/td>\n4<\/td>\n58<\/td>\n<\/tr>\n
B<\/strong><\/td>\n9000<\/td>\n14<\/td>\n101<\/td>\n<\/tr>\n
C<\/strong><\/td>\n12500<\/td>\n23<\/td>\n129<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Here for ad A, you didn’t have a big enough sample of clicks, just 4, to use them for A\/B testing by the frequentist approach. Bayesian A\/B testing, however, because of the prior (old knowledge), can still incorporate these samples to produce meaningful results or the posterior distribution (new knowledge).<\/p>\n
You easily incorporated these results in the prior beta distributions (old knowledge) to produce the posterior distributions (new knowledge) with the formula we discussed in a previous segment on beta distributions. For ad A, #click or number of clicks = 4. Similarly, #No click = 58-4=54.<\/p>\n
\"\alpha_{posterior\<\/pre>\n
\"\alpha_{posterior\<\/pre>\n
Similarly, you had calculated posterior \u03b2 for ad A as shown below.<\/p>\n
\"\beta_{posterior\<\/pre>\n
You did the same calculation for other posterior distributions of B and C as well. Once you knew the parameters (\u03b1 and \u03b2) for the posteriors, you easily plotted the beta distributions for posteriors.\"\"<\/a><\/p>\n
In this case, B is performing better than A in 81% cases. Already, with the limited sample size, we have an improvement on our priors (old knowledge). C will outperform A in 94% cases. Hence, with the new evidence, it is safe to say that Elisa is receiving higher empathy from the recipient of the emails then the map of Africa.<\/p>\n
Posterior Distributions after 5-days<\/span><\/h4>\n
After collecting data for five days you had all the possible results from the campaign. These were the results you had seen in the previous part as well while doing A\/B testing by the frequentist method.<\/p>\n\n\n\n\n\n\n
\u00a0Ads<\/span><\/strong><\/td>\nTotal Emails Sent<\/span><\/strong><\/td>\nClicks (5 days)<\/span><\/strong><\/td>\nOpens (5 days)<\/span><\/strong><\/td>\n<\/tr>\n
A<\/strong><\/td>\n5500<\/td>\n41<\/td>\n554<\/td>\n<\/tr>\n
B<\/strong><\/td>\n9000<\/td>\n98<\/td>\n922<\/td>\n<\/tr>\n
C<\/strong><\/td>\n12500<\/td>\n230<\/td>\n1235<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Using this new and improved evidence, you calculated the new beta distributions for the click rates for the 3 ads.<\/p>\n
\"\"<\/a><\/p>\n
Now, you know almost conclusively that ad C had outperformed both ad B and A in terms of click rates. You could create new sets of experiments with variants of ad C to further improve the performance on the click rates.<\/p>\n
Sign-off Note<\/span><\/h4>\n
You showed your results for the click rates to your client. She found it useful. After looking at the results one more time, she said -“As an organization, we care more about donation amount and not so much about the click rates”. She is a smart woman and even though you are doing this work for free or pro bono<\/em>, she will not shy away from asking the right questions.<\/p>\n
Incidentally, you had thought of these issues and designed the campaign accordingly. You reminded her that in the previous part of this case study, she had asked you another question about the disproportional distribution of email recipients across the 3 ads. She noticed that you had identified 5500 recipients for ad A and 12500 recipients for ad C. You told her that both her questions about maximizing the donation amount and the disproportional distribution of recipients are actually linked. You promised her that you will reveal the link in the next part of the case study. Moreover, you also promised that you are developing an intelligent campaign design for her that will learn from the previous campaigns to generate maximum returns in the form of donations.<\/p>\n
She was excited to learn the links between her questions and also to identify digital marketing ways to generate the maximum donation to help the needy in Africa. She immediately marked a date in her busy calendar to learn all this.<\/p>\n","protected":false},"excerpt":{"rendered":"
Welcome back to the\u00a0digital marketing case study example. In this case study, you are helping your client to identify an ad that will generate them the maximum return on marketing investment (ROMI). In the previous part, we discussed A\/B testing which is the\u00a0prevalent method used to compare the effectiveness of marketing effort.\u00a0You are an expert<\/p>\n
Read More…<\/a><\/p>\n","protected":false},"author":1,"featured_media":10108,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_newsletter_tier_id":0,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[82,1],"tags":[],"jetpack_publicize_connections":[],"yoast_head":"\nBayesian Statistics to Improve A\/B Testing - Digital Marketing Case Study<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Bayesian Statistics to Improve A\/B Testing - Digital Marketing Case Study\" \/>\n<meta property=\"og:description\" content=\"Welcome back to the\u00a0digital marketing case study example. In this case study, you are helping your client to identify an ad that will generate them the maximum return on marketing investment (ROMI). In the previous part, we discussed A\/B testing which is the\u00a0prevalent method used to compare the effectiveness of marketing effort.\u00a0You are an expertRead More...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/\" \/>\n<meta property=\"og:site_name\" content=\"YOU CANalytics |\" \/>\n<meta property=\"article:author\" content=\"roopam\" \/>\n<meta property=\"article:published_time\" content=\"2017-05-24T13:16:38+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-06-04T09:23:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Bayesian-Statistics-and-Knowledge.jpg?fit=920%2C574&ssl=1\" \/>\n\t<meta property=\"og:image:width\" content=\"920\" \/>\n\t<meta property=\"og:image:height\" content=\"574\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Roopam Upadhyay\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https:\/\/ucanalytics.com\/blogs\/#organization\",\"name\":\"YOU CANalytics\",\"url\":\"https:\/\/ucanalytics.com\/blogs\/\",\"sameAs\":[],\"logo\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/ucanalytics.com\/blogs\/#logo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2015\/11\/YOU-CANalytics-Logo.jpg?fit=607%2C120\",\"contentUrl\":\"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2015\/11\/YOU-CANalytics-Logo.jpg?fit=607%2C120\",\"width\":607,\"height\":120,\"caption\":\"YOU CANalytics\"},\"image\":{\"@id\":\"https:\/\/ucanalytics.com\/blogs\/#logo\"}},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ucanalytics.com\/blogs\/#website\",\"url\":\"https:\/\/ucanalytics.com\/blogs\/\",\"name\":\"YOU CANalytics |\",\"description\":\"Explore the Power of Data Science\",\"publisher\":{\"@id\":\"https:\/\/ucanalytics.com\/blogs\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ucanalytics.com\/blogs\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Bayesian-Statistics-and-Knowledge.jpg?fit=920%2C574&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Bayesian-Statistics-and-Knowledge.jpg?fit=920%2C574&ssl=1\",\"width\":920,\"height\":574},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#webpage\",\"url\":\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/\",\"name\":\"Bayesian Statistics to Improve A\/B Testing - Digital Marketing Case Study\",\"isPartOf\":{\"@id\":\"https:\/\/ucanalytics.com\/blogs\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#primaryimage\"},\"datePublished\":\"2017-05-24T13:16:38+00:00\",\"dateModified\":\"2017-06-04T09:23:50+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ucanalytics.com\/blogs\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Bayesian Statistics to Improve A\/B Testing – Digital Marketing Case Study Example (Part 3)\"}]},{\"@type\":\"Article\",\"@id\":\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#webpage\"},\"author\":{\"@id\":\"https:\/\/ucanalytics.com\/blogs\/#\/schema\/person\/55961a1cea272ecdf290cb387be069b6\"},\"headline\":\"Bayesian Statistics to Improve A\/B Testing – Digital Marketing Case Study Example (Part 3)\",\"datePublished\":\"2017-05-24T13:16:38+00:00\",\"dateModified\":\"2017-06-04T09:23:50+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#webpage\"},\"wordCount\":2066,\"commentCount\":6,\"publisher\":{\"@id\":\"https:\/\/ucanalytics.com\/blogs\/#organization\"},\"image\":{\"@id\":\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Bayesian-Statistics-and-Knowledge.jpg?fit=920%2C574&ssl=1\",\"articleSection\":[\"Digital Marketing - Case Study\",\"Marketing Analytics\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#respond\"]}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/ucanalytics.com\/blogs\/#\/schema\/person\/55961a1cea272ecdf290cb387be069b6\",\"name\":\"Roopam Upadhyay\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/ucanalytics.com\/blogs\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/dd1aa0b0e813f7639800bcfad6a554f1?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/dd1aa0b0e813f7639800bcfad6a554f1?s=96&d=mm&r=g\",\"caption\":\"Roopam Upadhyay\"},\"description\":\"This blog contains my personal views and thoughts on predictive Analytics and big data. - Roopam Upadhyay\",\"sameAs\":[\"roopam\"],\"url\":\"https:\/\/ucanalytics.com\/blogs\/author\/roopam\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Bayesian Statistics to Improve A\/B Testing - Digital Marketing Case Study","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/","og_locale":"en_US","og_type":"article","og_title":"Bayesian Statistics to Improve A\/B Testing - Digital Marketing Case Study","og_description":"Welcome back to the\u00a0digital marketing case study example. In this case study, you are helping your client to identify an ad that will generate them the maximum return on marketing investment (ROMI). In the previous part, we discussed A\/B testing which is the\u00a0prevalent method used to compare the effectiveness of marketing effort.\u00a0You are an expertRead More...","og_url":"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/","og_site_name":"YOU CANalytics |","article_author":"roopam","article_published_time":"2017-05-24T13:16:38+00:00","article_modified_time":"2017-06-04T09:23:50+00:00","og_image":[{"width":920,"height":574,"url":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Bayesian-Statistics-and-Knowledge.jpg?fit=920%2C574&ssl=1","type":"image\/jpeg"}],"twitter_misc":{"Written by":"Roopam Upadhyay","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Organization","@id":"https:\/\/ucanalytics.com\/blogs\/#organization","name":"YOU CANalytics","url":"https:\/\/ucanalytics.com\/blogs\/","sameAs":[],"logo":{"@type":"ImageObject","@id":"https:\/\/ucanalytics.com\/blogs\/#logo","inLanguage":"en-US","url":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2015\/11\/YOU-CANalytics-Logo.jpg?fit=607%2C120","contentUrl":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2015\/11\/YOU-CANalytics-Logo.jpg?fit=607%2C120","width":607,"height":120,"caption":"YOU CANalytics"},"image":{"@id":"https:\/\/ucanalytics.com\/blogs\/#logo"}},{"@type":"WebSite","@id":"https:\/\/ucanalytics.com\/blogs\/#website","url":"https:\/\/ucanalytics.com\/blogs\/","name":"YOU CANalytics |","description":"Explore the Power of Data Science","publisher":{"@id":"https:\/\/ucanalytics.com\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ucanalytics.com\/blogs\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"ImageObject","@id":"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#primaryimage","inLanguage":"en-US","url":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Bayesian-Statistics-and-Knowledge.jpg?fit=920%2C574&ssl=1","contentUrl":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Bayesian-Statistics-and-Knowledge.jpg?fit=920%2C574&ssl=1","width":920,"height":574},{"@type":"WebPage","@id":"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#webpage","url":"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/","name":"Bayesian Statistics to Improve A\/B Testing - Digital Marketing Case Study","isPartOf":{"@id":"https:\/\/ucanalytics.com\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#primaryimage"},"datePublished":"2017-05-24T13:16:38+00:00","dateModified":"2017-06-04T09:23:50+00:00","breadcrumb":{"@id":"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ucanalytics.com\/blogs\/"},{"@type":"ListItem","position":2,"name":"Bayesian Statistics to Improve A\/B Testing – Digital Marketing Case Study Example (Part 3)"}]},{"@type":"Article","@id":"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#article","isPartOf":{"@id":"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#webpage"},"author":{"@id":"https:\/\/ucanalytics.com\/blogs\/#\/schema\/person\/55961a1cea272ecdf290cb387be069b6"},"headline":"Bayesian Statistics to Improve A\/B Testing – Digital Marketing Case Study Example (Part 3)","datePublished":"2017-05-24T13:16:38+00:00","dateModified":"2017-06-04T09:23:50+00:00","mainEntityOfPage":{"@id":"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#webpage"},"wordCount":2066,"commentCount":6,"publisher":{"@id":"https:\/\/ucanalytics.com\/blogs\/#organization"},"image":{"@id":"https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Bayesian-Statistics-and-Knowledge.jpg?fit=920%2C574&ssl=1","articleSection":["Digital Marketing - Case Study","Marketing Analytics"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/ucanalytics.com\/blogs\/bayesian-statistics-to-improve-ab-testing-digital-marketing-case-study-example-part-3\/#respond"]}]},{"@type":"Person","@id":"https:\/\/ucanalytics.com\/blogs\/#\/schema\/person\/55961a1cea272ecdf290cb387be069b6","name":"Roopam Upadhyay","image":{"@type":"ImageObject","@id":"https:\/\/ucanalytics.com\/blogs\/#personlogo","inLanguage":"en-US","url":"https:\/\/secure.gravatar.com\/avatar\/dd1aa0b0e813f7639800bcfad6a554f1?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/dd1aa0b0e813f7639800bcfad6a554f1?s=96&d=mm&r=g","caption":"Roopam Upadhyay"},"description":"This blog contains my personal views and thoughts on predictive Analytics and big data. - Roopam Upadhyay","sameAs":["roopam"],"url":"https:\/\/ucanalytics.com\/blogs\/author\/roopam\/"}]}},"jetpack_featured_media_url":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Bayesian-Statistics-and-Knowledge.jpg?fit=920%2C574&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p3L0jT-2Av","jetpack-related-posts":[{"id":9910,"url":"https:\/\/ucanalytics.com\/blogs\/ab-testing-explained-challenged-digital-marketing-case-study-example-part-2\/","url_meta":{"origin":9951,"position":0},"title":"A\/B Testing Explained and Challenged – Digital Marketing Case Study Example (Part 2)","author":"Roopam Upadhyay","date":false,"format":false,"excerpt":"Welcome back to our digital marketing case study example. In the previous part, you had initiated an email donation campaign for your client Helping Hand, an NGO, with three different ads for the same cause to support the needy in Africa. The idea is to identify the ad that generates\u2026","rel":"","context":"In "Digital Marketing - Case Study"","block_context":{"text":"Digital Marketing - Case Study","link":"https:\/\/ucanalytics.com\/blogs\/category\/digital-marketing-case-study\/"},"img":{"alt_text":"A\/B Testing and Photo Finish - by Roopam","src":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Race.jpg?fit=626%2C421&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Race.jpg?fit=626%2C421&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Race.jpg?fit=626%2C421&ssl=1&resize=525%2C300 1.5x"},"classes":[]},{"id":3129,"url":"https:\/\/ucanalytics.com\/blogs\/marketing-analytics-retail-case-study-part-1\/","url_meta":{"origin":9951,"position":1},"title":"Marketing Analytics – Retail Case Study Example (Part 1)","author":"Roopam Upadhyay","date":false,"format":false,"excerpt":"Today we are going to start a new case study example on YOU CANalytics. In this case study example, we will examine different facets of marketing analytics and customer relationship management (CRM). We will use the example of online retail to explore more about marketing analytics - an area of\u2026","rel":"","context":"In "Marketing Analytics"","block_context":{"text":"Marketing Analytics","link":"https:\/\/ucanalytics.com\/blogs\/category\/marketing-analytics\/"},"img":{"alt_text":"Marketing Analytics Modeling","src":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/06\/Marketing-Analytics-Modeling-Blog.jpg?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/06\/Marketing-Analytics-Modeling-Blog.jpg?resize=350%2C200 1x, https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/06\/Marketing-Analytics-Modeling-Blog.jpg?resize=525%2C300 1.5x, https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/06\/Marketing-Analytics-Modeling-Blog.jpg?resize=700%2C400 2x"},"classes":[]},{"id":10135,"url":"https:\/\/ucanalytics.com\/blogs\/thompson-sampling-for-artificial-intelligence-digital-marketing-case\/","url_meta":{"origin":9951,"position":2},"title":"Thompson Sampling for Artificial Intelligence – Digital Marketing Case Study Example (Part 4)","author":"Roopam Upadhyay","date":false,"format":false,"excerpt":"This is the final part of the digital marketing case study example. In this case study, you are a digital analytics consultant. Here, you are working with your client Helping Hand, an NGO, to extract the maximum value out of their marketing campaign. For this, you are using principles of\u00a0reinforcement\u2026","rel":"","context":"In "Digital Marketing - Case Study"","block_context":{"text":"Digital Marketing - Case Study","link":"https:\/\/ucanalytics.com\/blogs\/category\/digital-marketing-case-study\/"},"img":{"alt_text":"Thompson Sampling and Artificial Intelligence - by Roopam","src":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/05\/Mother-Teresa-and-Stalin.jpg?fit=1015%2C475&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/05\/Mother-Teresa-and-Stalin.jpg?fit=1015%2C475&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/05\/Mother-Teresa-and-Stalin.jpg?fit=1015%2C475&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/05\/Mother-Teresa-and-Stalin.jpg?fit=1015%2C475&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":9797,"url":"https:\/\/ucanalytics.com\/blogs\/reinforcement-learning-and-artificial-intelligence-digital-marketing-case-study-example-part-1\/","url_meta":{"origin":9951,"position":3},"title":"Reinforcement Learning and Artificial Intelligence – Digital Marketing Case Study Example (Part 1)","author":"Roopam Upadhyay","date":false,"format":false,"excerpt":"How to make machines learn on their own similar to humans? This is the pivotal question for the development of artificial intelligence. To develop intelligent machines and systems (artificial intelligence), we need to understand how human intelligence and learning work. For this, we will explore the ideas behind reinforcement learning.\u2026","rel":"","context":"In "Digital Marketing - Case Study"","block_context":{"text":"Digital Marketing - Case Study","link":"https:\/\/ucanalytics.com\/blogs\/category\/digital-marketing-case-study\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Digital-Marketing-Campaign.jpg?fit=1200%2C654&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Digital-Marketing-Campaign.jpg?fit=1200%2C654&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Digital-Marketing-Campaign.jpg?fit=1200%2C654&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Digital-Marketing-Campaign.jpg?fit=1200%2C654&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2017\/04\/Digital-Marketing-Campaign.jpg?fit=1200%2C654&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":3195,"url":"https:\/\/ucanalytics.com\/blogs\/marketing-analytics-retail-case-study-example-part-2\/","url_meta":{"origin":9951,"position":4},"title":"Marketing Analytics – Retail Case Study Example (Part 2)","author":"Roopam Upadhyay","date":false,"format":false,"excerpt":"Last week we started a case study from the online retail industry to learn more about marketing analytics (Read Part 1). Before we continue with the same case, let me share a few factors that enhance the quality of analysis for marketing or customer analytics. The obvious factors, of course,\u2026","rel":"","context":"In "Marketing Analytics"","block_context":{"text":"Marketing Analytics","link":"https:\/\/ucanalytics.com\/blogs\/category\/marketing-analytics\/"},"img":{"alt_text":"Noir - by Roopam","src":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/02\/photo.jpg?fit=713%2C1024&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/02\/photo.jpg?fit=713%2C1024&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/02\/photo.jpg?fit=713%2C1024&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/02\/photo.jpg?fit=713%2C1024&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":3973,"url":"https:\/\/ucanalytics.com\/blogs\/model-selection-retail-case-study-example-part-7\/","url_meta":{"origin":9951,"position":5},"title":"Model Selection – Retail Case Study Example (Part 7)","author":"Roopam Upadhyay","date":false,"format":false,"excerpt":"Model Selection This is a continuation of our retail case study example for campaign and marketing analytics. In the previous two parts, we discussed a couple of decision tree algorithms (CART and C4.5)\u00a0for classification. Recall a previous case study example on\u00a0banking and risk management where we discussed logistic regression\u00a0which is\u2026","rel":"","context":"In "Marketing Analytics"","block_context":{"text":"Marketing Analytics","link":"https:\/\/ucanalytics.com\/blogs\/category\/marketing-analytics\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/09\/photo.jpg?fit=1200%2C1029&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/09\/photo.jpg?fit=1200%2C1029&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/09\/photo.jpg?fit=1200%2C1029&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/09\/photo.jpg?fit=1200%2C1029&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/09\/photo.jpg?fit=1200%2C1029&ssl=1&resize=1050%2C600 3x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/ucanalytics.com\/blogs\/wp-json\/wp\/v2\/posts\/9951"}],"collection":[{"href":"https:\/\/ucanalytics.com\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ucanalytics.com\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ucanalytics.com\/blogs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ucanalytics.com\/blogs\/wp-json\/wp\/v2\/comments?post=9951"}],"version-history":[{"count":0,"href":"https:\/\/ucanalytics.com\/blogs\/wp-json\/wp\/v2\/posts\/9951\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ucanalytics.com\/blogs\/wp-json\/wp\/v2\/media\/10108"}],"wp:attachment":[{"href":"https:\/\/ucanalytics.com\/blogs\/wp-json\/wp\/v2\/media?parent=9951"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ucanalytics.com\/blogs\/wp-json\/wp\/v2\/categories?post=9951"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ucanalytics.com\/blogs\/wp-json\/wp\/v2\/tags?post=9951"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}