{"id":11290,"date":"2018-10-01T09:49:12","date_gmt":"2018-10-01T04:19:12","guid":{"rendered":"http:\/\/ucanalytics.com\/blogs\/?p=11290"},"modified":"2019-04-21T18:30:41","modified_gmt":"2019-04-21T13:00:41","slug":"math-of-deep-learning-neural-networks-simplified-part-2","status":"publish","type":"post","link":"https:\/\/ucanalytics.com\/blogs\/math-of-deep-learning-neural-networks-simplified-part-2\/","title":{"rendered":"Math of Deep Learning Neural Networks – Simplified (Part 2)"},"content":{"rendered":"
\n
\"\"<\/a>

The Math of Deep Learning Neural Networks – by Roopam<\/p><\/div>\n

Welcome back to this series of articles on deep learning and neural networks<\/a>. In the last part, you learned how training a\u00a0deep learning network is similar to a plumbing job<\/a>. <\/strong>This time you will learn the math of deep learning. We will continue to use the plumbing analogy to simplify the seemingly complicated math. I believe you will find this highly intuitive. Moreover, understanding this will provide you with a good idea about the inner workings of deep learning networks and artificial intelligence (AI) to build an AI of your own. We will use the math of deep learning to make an image recognition AI in the next part. But before that let’s create the links between…<\/p>\n

The Math of Deep Learning and Plumbing<\/span><\/h2>\n

Last time we noticed that neural networks are like the networks of water pipes. The goal of neural networks is to identify the right settings for the knobs (6 in this schematic) to get the\u00a0right output given the input.<\/p>\n

\"\"<\/a><\/p>\n

Shown below is a familiar schematic of neural networks almost identical to the water pipelines above. The only exception is the additional bias terms (b1<\/sub>,b2<\/sub>, and b3<\/sub>) added to the nodes.<\/p>\n

\"\"<\/a>In this post, we will solve this network to understand the math of deep learning. Note that a deep learning model has multiple hidden layers, unlike this simple neural network. However, this simple neural network can easily be generalized to the deep learning models. The math of deep learning does not change a lot with additional complexity and hidden layers. Here, our objective is to identify the values of the parameters {W (W1<\/sub>,…, W6<\/sub>) and b (b1<\/sub>,b2<\/sub>, and b3<\/sub>)}. We will soon use the backpropagation algorithm along with gradient descent optimization<\/a><\/strong> to solve this network and identify the optimal\u00a0values of these weights.<\/p>\n

\"\"<\/p>\n

Backpropagation and Gradient Descent<\/span><\/h2>\n

In the previous post, we discussed that the backpropagation algorithm works similar to me shouting back at my plumber while he was working in the duct. Remember, I was telling the plumber about the difference in actual water pressure from the expected. The plumber of neural networks, unlike my building’s plumber,\u00a0learns from this information to optimize the positions of the knobs. The method that the neural networks plumber uses to iteratively correct the weights or settings of the knobs is called gradient descent.<\/p>\n

We have discussed the gradient descent algorithm<\/strong><\/a> in an earlier post to solve a logistic regression model. I recommend that you read that article to get a good grasp of the things we will discuss in this post. Essentially, the idea is to iteratively correct the value of the weights (Wi<\/sub>) to produce the least difference between the actual and the expected values of the output. This difference is measured mathematically by the loss function i.e \"\ell. The weights (Wi<\/sub> and bi<\/sub>) are then iteratively improved using the gradient of the loss function wrt weights using this expression:<\/p>\n

\"W_{i}<\/pre>\n
Here, \u03b1 is called the learning rate – it’s a hyperparameter and stays constant. Hence, the overall problem boils down to the identification of partial derivatives of the loss function with respect to the weights i.e. \"\frac{\partial. For our problem, we just need to solve the partial derivatives for W5<\/sub> and W1<\/sub>. The partial derivatives for other weights can then be easily derived using the same method used for W5<\/sub> and W1<\/sub>.<\/p>\n
Before we solve these partial derivatives, let’s do some more plumbing jobs and look at a tap to develop intuitions about the results we will get from the gradient descent optimization.<\/p>\n
Intuitive Math of Deep Learning for W<\/span>5<\/span>\u00a0<\/sub>& A Tap<\/span><\/h2>\n
\"\"<\/p>\n
We will use this simple tap to identify an optimal setting for its knob. In\u00a0this process, we will develop intuitions about gradient descent and the math of deep learning. Here, the input is the water coming from the pipe on the left of the image. Moreover, the output is the water coming out of the tap. You use the knob, on the\u00a0top of the\u00a0tap, to regulate the quantity of the output water given the input.\u00a0Remember, you want to turn the knob in such a way that you get the desired output (i.e the quantity of water) to wash your hands. Keep in mind, the position of the knob is similar to the weight of a neural networks’ parameters. Moreover, the input\/output water is similar to the input\/output variables.<\/p>\n
Essentially, in math terms, you are trying to identify how the position of the\u00a0knob\u00a0influences the output water. The mathematical equation for the same is:<\/p>\n
\"\frac{\partial<\/pre>\n
If you understand the influence of the knob on the output flow of water you can easily turn it to get the desired output. Now, let’s develop an intuition about how much to twist the knob. When you use a tap you twist the knob until you get the right flow or the output. When the difference between the desired output and the actual output is large then you need a lot of twisting. On the other hand, when the difference is less then you turn the knob gently.<\/p>\n
Moreover, the other factor on which your decision depends on is the input from the left pipe. If there is no water flowing from the left pipe then no matter how much you twist the\u00a0knob it won’t help. Essentially, your action depends on these two factors.<\/p>\n
Your decision to turn the knob depends on\n<\/strong>Factor 1:<\/span> Difference between the actual output and the desired Output and<\/span>\n<\/span>Factor 2:<\/span> Input from the grey pipe on the left\n<\/span><\/strong><\/span><\/pre>\n
Soon you will get the same result by doing a seemingly complicated math for the gradient descent to solve the neural network.<\/p>\n
\"\frac{\partial<\/pre>\n
For our network, the output difference is \"(\ltimes_{3}-y)\" and input is \"\ltimes_{1}. Hence,<\/p>\n
\"\frac{\partial<\/pre>\n
Disclaimer<\/span><\/h4>\n
Please note,\u00a0to make the\u00a0concepts easy for you to understand, I had taken a few liberties while defining the factors in the previous section. I will make these factors much more theoretically grounded at the end of this article when I will discuss the chain rule to solve derivatives. For now, I will continue to take more liberties in the next section when I discuss the other weight modification for other parameters of neural networks.<\/p>\n
Add More Knobs to Solve W1<\/sub> –\u00a0Intuitive Math of Deep Learning<\/span><\/h2>\n
Neural networks, as discussed earlier, have several parameters (Ws and bs). To develop an intuition about the math to estimate the other parameters further away from the output (i.e. W1<\/sub>), let’s add another knob to the tap.\u00a0\"\"<\/a>
\nHere, we have added a red regulator knob to the tap we saw in the earlier section. Now, the output from the tap is governed by both these knobs. Referring to the neural network’s image shown earlier, the red knob is similar to the parameters (W1<\/sub>, W2,<\/sub> W3<\/sub>, W4, <\/sub>b1, <\/sub>and b2<\/sub>) added to the hidden layers. The knob on top of the brass tap is like the parameters to the output layer (i.e. W5<\/sub>, W6<\/sub>, and b3<\/sub>).<\/p>\n
Now, you are also using the red knob, in addition to the knob on the tap, to get the desired output from the tap. Your effort of the red knob will depend on these factors.<\/p>\n
Your decision to turn the red knob depends on\n<\/strong>Factor 1:<\/span> Difference between the actual and the desired final output and<\/span>\n<\/span>Factor 2: Position \/ setting of the knob on the brass tap and\nFactor 3:<\/span> <\/span>Change in input to the brass tap caused by the red knob and\nFactor 4:<\/span> Input from the pipe on the left into the red knob\n<\/span><\/span><\/span><\/strong><\/span><\/pre>\n
Here, as already discussed earlier, factor 1 is \"(\ltimes_{3}-y)\". W5<\/sub> is the setting\/weight for the knob of the brass tap. Factor 3 is \"(\ltimes_{1}-1)\cdot\ltimes_{1}. Finally, the last factor is the input or X1<\/sub>. This completes our equation as:<\/p>\n
\"\frac{\partial<\/pre>\n
Now, before we do the math to get these results, we just need to discuss the components of our neural network in mathematical terms. We already know how it relates to the water pipelines discussed earlier.<\/p>\n
\"\"<\/a><\/p>\n
Let’s start with the nodes or the orange circles in the network diagram.<\/p>\n
Nodes of\u00a0<\/span>Neural Networks<\/span><\/h2>\n
Here, these two networks are equivalent except the additional b1<\/sub> or bias for the neural networks.<\/p>\n
\"\"<\/a><\/p>\n
The node for the neural network has two components i.e. sum and non-linear. The sum component (Z1<\/sub>) is just a linear combination of the input and the weights.<\/p>\n
\"Z_{1}<\/pre>\n
The next term, i.e. non-linear, is the non-linear sigmoid activation function (\"\ltimes_{1}). As discussed earlier, it is like a regulator of a fan that keeps the value of \"\ltimes_{1}\u00a0 between 0 and 1 or on\/off.<\/p>\n
\"\"<\/a><\/p>\n
The mathematical expression for this sigmoid activation function (\"\ltimes_{1}) is:<\/p>\n
\"\ltimes_{1}<\/pre>\n
The nodes in both the hidden and output layer behave the same as described above. Now, the last thing is to define the loss function (\"\ell) which is to measure the difference between the\u00a0expected and actual output. We will define the loss function for most common business problems.<\/p>\n
Classification Problem –\u00a0Math of Deep Learning<\/span><\/h2>\n
In practice, most business problems are about classification. They have binary or categorical outputs\/answers such as:<\/p>\n