![\"Data](\"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2015\/01\/photo-5.jpg\")
<\/a>Data Preparation and Master Chef – by Roopam<\/p><\/div>\n
Every data scientist knows that in any business analytics and data science exercise 70-80% of the time is consumed in data preparation and data preprocessing. This is usually considered a drudgery in\u00a0comparison to the actual statistical modeling, machine learning, and business insights part. However, every good data scientist understands that data preparation is an art and a highly intellectual exercise. We will discover the art and science of data preparation in this article. However, before that let’s experience some culinary delights\u00a0in the…<\/p>\n
Master Chef’s Kitchen<\/span><\/h2>\nMy wife is a huge fan of MasterChef Australia. I think, she has watched all the seasons of the show till date since its inception. For me, I enjoy eating good food more than watching someone cook, and judges relishing their meal. However, over the years I have watched a few episodes because of my wife. I must say that there are a lot of good lessons for data scientists in cooking.\u00a0In virtually every episode of MasterChef I have seen, the participants spent a larger part of their time preparing ingredients for their final dishes. They all runaround and collect the appropriate supplies for their dishes from the larder. To me, all this seemed quite similar to data scientists\u00a0running around and getting the right data fields from data sources & data warehouses. After this, the act of peeling, chopping, cutting, roasting etc. is similar to data preparation in data science.<\/p>\n
I use to think\u00a0that\u00a0master chef’s in big restaurants usually just order around sous chefs and assistants to prepare ingredients for their main dish. However, in one of the episodes of MasterChef Marco Pierre White, a celebrity chef, asks the participants to chop onions really fine to test their skills with the knife. He\u00a0first demonstrated his meticulous knife skills by chopping the onion into perfect microscopic pieces within a few seconds. For Marco Pierre White, this was not an act of showcasing his supremacy but a necessity for the dish he had in mind. Similarly, a senior data scientist\u00a0needs to be completely involved in the process of data preparation and preprocessing to produced desirable models and business results.<\/p>\n
A data scientist can’t\u00a0expect a delicious model unless she has spent enough time preparing the right ingredients for the model through data preparation. In the next section, let’s explore some key elements of<\/p>\n
Data Preparation<\/span><\/h2>\nContinuous improvement to generate competitive advantage is the only way for companies to survive in the current times of cut-throat business competition. For the modern businesses, Analytics is an essential instrument to generate\u00a0competitive advantage. Analytics and data science generate novel business insights for improved business actions to keep the company ahead in the race. Even in the age of big data, \u00a0companies\u00a0can dismiss the fundamental rules of rigorous research and analysis design at their own peril. As we will see, intelligent data preparation is a fundamental aspect of\u00a0rigorous data science design. Unfortunately, till date, there is no software or tool that will create an intelligent data preparation design across the\u00a0industry. Hence the onus of data preparation is still with creative human minds.<\/p>\n
Before I\u00a0share a few key aspects of data preparation, let us have a look at a simplistic data schema from banking. The following schema is from banking however the discussion we will have about data preparation will suggest\u00a0general strategies for all other\u00a0industries including healthcare, telecom, retail etc.<\/p>\n
![\"A](\"https:\/\/i0.wp.com\/ucanalytics.com\/blogs\/wp-content\/uploads\/2015\/01\/Banking-Databases.jpg?resize=597%2C397\")
<\/a>A Simple Schematic of Banking Datasets<\/p><\/div>\n
Continuous improvement to generate competitive advantage is the only way for companies to survive in the current times of cut-throat business competition. For the modern businesses, Analytics is an essential instrument to generate\u00a0competitive advantage. Analytics and data science generate novel business insights for improved business actions to keep the company ahead in the race. Even in the age of big data, \u00a0companies\u00a0can dismiss the fundamental rules of rigorous research and analysis design at their own peril. As we will see, intelligent data preparation is a fundamental aspect of\u00a0rigorous data science design. Unfortunately, till date, there is no software or tool that will create an intelligent data preparation design across the\u00a0industry. Hence the onus of data preparation is still with creative human minds.<\/p>\n
Before I\u00a0share a few key aspects of data preparation, let us have a look at a simplistic data schema from banking. The following schema is from banking however the discussion we will have about data preparation will suggest\u00a0general strategies for all other\u00a0industries including healthcare, telecom, retail etc.<\/p>\n
<\/a>A Simple Schematic of Banking Datasets<\/p><\/div>\n