![\"Model](\"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/09\/photo.jpg\")
<\/a>Model Selection – by Roopam<\/p><\/div>\n
Model Selection<\/span><\/h2>\nThis is a continuation of our retail case study example for campaign and marketing analytics. In the previous two parts, we discussed a couple of decision tree algorithms (CART<\/a> and C4.5<\/a>)\u00a0for classification. Recall a previous case study example on\u00a0banking and risk management where we discussed logistic regression<\/a>\u00a0which is another approach to solving classification problems. Additionally, there are several other\u00a0statistical and machine learning algorithms that are equally powerful for classification tasks such as:<\/p>\n\n- Support Vector Machines<\/li>\n
- Random Forest<\/li>\n
- Artificial Neural Networks<\/li>\n
- Discriminant analysis<\/li>\n
- Boosting<\/li>\n
- Na\u00efve Bayes\u00a0Classifiers<\/li>\n<\/ul>\n
This list is definitely not complete but covers some of the commonly used approaches. We will discuss all these approaches in later articles\u00a0on YOU CANalytics. Now the question is: why there are so\u00a0many different approaches to solving the same problem?\u00a0A\u00a0more important question which everybody asks is: which one approach of\u00a0these\u00a0is the best? The answer to the second question is none! Yes, the best approach depends on the kind of data you are working with,\u00a0and since data come in all shapes and sizes hence you can’t have one\u00a0best approach for all problems. Hence, development of models with different approaches, and the best model selection for your data is an important exercise in data science and analytics. In this article, we will discuss the factors that influence the process of model selection. However, before that let us quickly examine some of the tasks data scientists perform, \u00a0as this will help us when we will make our transition to next parts of this case study example.<\/p>\n
Tasks for Data Science<\/span><\/h2>\nPrimarily, tasks that data scientists perform could be grouped into the following six broad categories as displayed\u00a0below. Please note that even the modern data science tasks such as web & social media analytics, text mining, image analytics, and sound\u00a0pattern detection are some the used cases of these six broad categories.<\/p>\n
![\"Data](\"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/09\/Data-Science-Tasks-by-Roopam.jpg?resize=640%2C377\")
Data Science Tasks – by Roopam<\/p><\/div>\n
As you might have noticed, in this\u00a0case study, we have so far performed 3 tasks from the above list\u00a0i.e. ‘Description’ (exploratory data analysis<\/a>), ‘Association’ (association analysis<\/a>), and ‘Classification’ (decision trees:\u00a0CART<\/a> and C4.5<\/a>). I must say, exploratory data analysis (EDA) is an integral part of every data science project. EDA is a crucial exercise that\u00a0drives predictive models in the right direction.<\/p>\nIn the last few\u00a0parts of this case study, we will do some ‘Estimation’ (i.e. regression analysis to estimate revenue\u00a0generated by customers through campaigns). Let’s go back to model selection for our classification problem.<\/p>\n
Model Selection – Retail Case Study Example<\/span><\/h2>\nBack to our retail case study example, where you are\u00a0the Chief Analytics Officer & Business Strategy Head at an online shopping store called DresSMart Inc. that specializes in apparel and clothing.\u00a0Through\u00a0your rigorous exploratory data analysis you have found several factors\u00a0that play crucial roles in marketing campaigns’\u00a0response\u00a0for customers, some of these factors are:<\/p>\n
\n- Recency: # recent visits to the company’s website and purchases<\/li>\n
- Frequency: time lag between purchases in the last 6 months<\/li>\n
- Payment mode used: cash on delivery, credit card, internet banking etc.<\/li>\n
- Marketing data aggregator’s:\u00a0life-stage segmentations (i.e. luxury buffs, up-scale ageing, first-time earners etc.)<\/li>\n
- Last year’s expenditure trend: amount spent last year<\/li>\n
- Coupon usage pattern of customer<\/li>\n<\/ul>\n
You have tried several multivariate models mentioned above (i.e. logistic regression, SVM, decision trees etc.) to model customers’ behaviour and generate purchase propensity scores. The choice of right model selection depends on the following 2\u00a0factors i.e.<\/p>\n
\n- Predictive power of models<\/li>\n
- Business &\u00a0operations integration<\/li>\n<\/ol>\n
1 Predictive power of models<\/span><\/h4>\nThe first factor for model selection\u00a0is the overall predictive power that the model has in comparison to other models. For this classification problem, the area under receiver operating curve (AUROC) is possibly the best way to assess the predictive power of models (read more about AUROC<\/a>). Sometimes people also use Gini coefficient for assessing predictive power of models, Gini is another variant\u00a0of AUROC and mathematically represented as:<\/p>\n![\"Gini=2\times](\"https:\/\/s0.wp.com\/latex.php?latex=Gini%3D2%5Ctimes+AUROC-1+&bg=ffffff&fg=000&s=0&c=20201002\")
<\/pre>\n![\"Area](\"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2014\/09\/image001.png?resize=344%2C324\")
<\/a>Area Under ROC for Different Models<\/p><\/div>\n
In the adjacent plot, AUROC is displayed for artificial neural networks, logistic regression, and CART decision tree. Notice, the perfect model curve (in green) here is with 100% predictive power, and random model (in red) represents prediction through the flip of a coin. The AUROC values for the\u00a0test sample for the 3 models are:<\/p>\n
\n\n\nModel<\/td>\n AUROC<\/td>\n<\/tr>\n \nDecision Tree<\/td>\n 72%<\/td>\n<\/tr>\n \nLogistic Regression<\/td>\n 76%<\/td>\n<\/tr>\n \nArtificial Neural Networks<\/td>\n 77%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nDecision tree here is performing much below the other two models. This is often the case with decision trees, but they are still very\u00a0useful and popular because of their highly intuitive and easy-to-explain solutions. Artificial neural networks\u00a0are performing a notch above logistic regression in this case with a slightly higher\u00a0area under ROC. Hence by the first criteria, artificial neural networks\u00a0offer the best model among the 3 models.<\/p>\n
2 \u00a0Business &\u00a0operations integration:<\/span><\/h4>\nThis aspect of model selection is equally important as the above factor if not more. The model selection must be based on productization\u00a0of the model for business usage in the long run. The following factors\u00a0are useful to keep in mind at the beginning of modeling process:<\/p>\n
- \n
- Support Vector Machines<\/li>\n
- Random Forest<\/li>\n
- Artificial Neural Networks<\/li>\n
- Discriminant analysis<\/li>\n
- Boosting<\/li>\n
- Na\u00efve Bayes\u00a0Classifiers<\/li>\n<\/ul>\n
This list is definitely not complete but covers some of the commonly used approaches. We will discuss all these approaches in later articles\u00a0on YOU CANalytics. Now the question is: why there are so\u00a0many different approaches to solving the same problem?\u00a0A\u00a0more important question which everybody asks is: which one approach of\u00a0these\u00a0is the best? The answer to the second question is none! Yes, the best approach depends on the kind of data you are working with,\u00a0and since data come in all shapes and sizes hence you can’t have one\u00a0best approach for all problems. Hence, development of models with different approaches, and the best model selection for your data is an important exercise in data science and analytics. In this article, we will discuss the factors that influence the process of model selection. However, before that let us quickly examine some of the tasks data scientists perform, \u00a0as this will help us when we will make our transition to next parts of this case study example.<\/p>\n
Tasks for Data Science<\/span><\/h2>\n
Primarily, tasks that data scientists perform could be grouped into the following six broad categories as displayed\u00a0below. Please note that even the modern data science tasks such as web & social media analytics, text mining, image analytics, and sound\u00a0pattern detection are some the used cases of these six broad categories.<\/p>\n
Data Science Tasks – by Roopam<\/p><\/div>\n
As you might have noticed, in this\u00a0case study, we have so far performed 3 tasks from the above list\u00a0i.e. ‘Description’ (exploratory data analysis<\/a>), ‘Association’ (association analysis<\/a>), and ‘Classification’ (decision trees:\u00a0CART<\/a> and C4.5<\/a>). I must say, exploratory data analysis (EDA) is an integral part of every data science project. EDA is a crucial exercise that\u00a0drives predictive models in the right direction.<\/p>\n
In the last few\u00a0parts of this case study, we will do some ‘Estimation’ (i.e. regression analysis to estimate revenue\u00a0generated by customers through campaigns). Let’s go back to model selection for our classification problem.<\/p>\n
Model Selection – Retail Case Study Example<\/span><\/h2>\n
Back to our retail case study example, where you are\u00a0the Chief Analytics Officer & Business Strategy Head at an online shopping store called DresSMart Inc. that specializes in apparel and clothing.\u00a0Through\u00a0your rigorous exploratory data analysis you have found several factors\u00a0that play crucial roles in marketing campaigns’\u00a0response\u00a0for customers, some of these factors are:<\/p>\n
- \n
- Recency: # recent visits to the company’s website and purchases<\/li>\n
- Frequency: time lag between purchases in the last 6 months<\/li>\n
- Payment mode used: cash on delivery, credit card, internet banking etc.<\/li>\n
- Marketing data aggregator’s:\u00a0life-stage segmentations (i.e. luxury buffs, up-scale ageing, first-time earners etc.)<\/li>\n
- Last year’s expenditure trend: amount spent last year<\/li>\n
- Coupon usage pattern of customer<\/li>\n<\/ul>\n
You have tried several multivariate models mentioned above (i.e. logistic regression, SVM, decision trees etc.) to model customers’ behaviour and generate purchase propensity scores. The choice of right model selection depends on the following 2\u00a0factors i.e.<\/p>\n
- \n
- Predictive power of models<\/li>\n
- Business &\u00a0operations integration<\/li>\n<\/ol>\n
1 Predictive power of models<\/span><\/h4>\n
The first factor for model selection\u00a0is the overall predictive power that the model has in comparison to other models. For this classification problem, the area under receiver operating curve (AUROC) is possibly the best way to assess the predictive power of models (read more about AUROC<\/a>). Sometimes people also use Gini coefficient for assessing predictive power of models, Gini is another variant\u00a0of AUROC and mathematically represented as:<\/p>\n
<\/a>Area Under ROC for Different Models<\/p><\/div>\n
In the adjacent plot, AUROC is displayed for artificial neural networks, logistic regression, and CART decision tree. Notice, the perfect model curve (in green) here is with 100% predictive power, and random model (in red) represents prediction through the flip of a coin. The AUROC values for the\u00a0test sample for the 3 models are:<\/p>\n
\n\n
\n Model<\/td>\n AUROC<\/td>\n<\/tr>\n \n Decision Tree<\/td>\n 72%<\/td>\n<\/tr>\n \n Logistic Regression<\/td>\n 76%<\/td>\n<\/tr>\n \n Artificial Neural Networks<\/td>\n 77%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Decision tree here is performing much below the other two models. This is often the case with decision trees, but they are still very\u00a0useful and popular because of their highly intuitive and easy-to-explain solutions. Artificial neural networks\u00a0are performing a notch above logistic regression in this case with a slightly higher\u00a0area under ROC. Hence by the first criteria, artificial neural networks\u00a0offer the best model among the 3 models.<\/p>\n
2 \u00a0Business &\u00a0operations integration:<\/span><\/h4>\n
This aspect of model selection is equally important as the above factor if not more. The model selection must be based on productization\u00a0of the model for business usage in the long run. The following factors\u00a0are useful to keep in mind at the beginning of modeling process:<\/p>\n