Neural Network and Deep Learning Models - YOU CANalytics

Import all the required libraries and the MNIST dataset

In [1]:
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix
from sklearn.externals import joblib
import matplotlib.pyplot as plt
from sklearn import metrics
import keras
from keras.datasets import mnist
(Image_train, Number_train), (Image_test, Number_test) = mnist.load_data()

Using TensorFlow backend.

Randomly select images of numbers to display in the next step

In [2]:
def plot_mnist_image(instances, images_per_row=10, **options):
    size = 28
    images_per_row = min(len(instances), images_per_row)
    images = [instance.reshape(size,size) for instance in instances]
    n_rows = (len(instances) - 1) // images_per_row + 1
    row_images = []
    n_empty = n_rows * images_per_row - len(instances)
    images.append(np.zeros((size, size * n_empty)))
    for row in range(n_rows):
        rimages = images[row * images_per_row : (row + 1) * images_per_row]
        row_images.append(np.concatenate(rimages, axis=1))
    image = np.concatenate(row_images, axis=0)
    plt.imshow(image, cmap = matplotlib.cm.binary, **options)
    plt.axis("off")

Display the images to explain the complexity associated with the classification task

In [3]:
values = np.asarray(Number_train)
i0=np.random.choice(np.where(values == 0)[0],size=5,replace=False)

for i in range(1, 10):
    i1=np.random.choice(np.where(values == i)[0],size=5,replace=False)
    i0=np.concatenate((i0,i1),axis=0, out=None)
    
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt    
plt.figure(figsize=(10,10))
images = np.r_[Image_train[i0]]
plot_mnist_image(images, images_per_row=10)

plt.show()

Reshape and normalize data for modeling (popeye effect)

In [4]:
Image_train = Image_train.reshape((60000, 28 * 28))
Image_train = Image_train.astype('float32') / 255

Image_test = Image_test.reshape((10000, 28 * 28))
Image_test = Image_test.astype('float32') / 255

Model 1: Logistic regression Model

In [5]:
from sklearn.linear_model import LogisticRegression
LR=LogisticRegression()
LR.fit(Image_train, Number_train)
Number_pred_LR = LR.predict(Image_test)
from sklearn.metrics import accuracy_score
accuracy_score(Number_test, Number_pred_LR)
Out[5]:
0.9202

Logistic regression - estimated probabilities of digits (a test image)

In [6]:
plt.imshow(Image_test[10].reshape(28,28), cmap = matplotlib.cm.binary)
LR.predict_proba(Image_test[10].reshape(1,-1))
Out[6]:
array([[9.77658295e-01, 5.84542143e-10, 5.47154617e-03, 2.68364552e-04,
        3.23482989e-06, 1.15918666e-02, 2.98931274e-05, 1.11649278e-08,
        4.97560185e-03, 1.18648361e-06]])

Confusion matrix - logistic regression

In [7]:
import seaborn as sns
cm = metrics.confusion_matrix(Number_test, Number_pred_LR)
plt.figure(figsize=(9,9))
sns.heatmap(cm, annot=True, fmt=".0f", linewidths=.5, square = True, cmap = 'Blues_r');
plt.ylabel('Actual Number');
plt.xlabel('Predicted Number');
all_sample_title = 'Confusion Matrix : Logistic Regression'
plt.title(all_sample_title, size = 15);

Model 2: a random forest classifier to classify the numbers

Define the model parameters and train the random forest model

In [8]:
classifier = RandomForestClassifier(n_estimators = 100, criterion = 'entropy', random_state = 42)
classifier.fit(Image_train, Number_train)
plt.imshow(classifier.feature_importances_.reshape(28,28))
Out[8]:
<matplotlib.image.AxesImage at 0x234803810f0>

Find the accuracy of the random forest model on the test dataset

In [9]:
Number_pred_RF = classifier.predict(Image_test)
from sklearn.metrics import accuracy_score
accuracy_score(Number_test, Number_pred_RF)
Out[9]:
0.9686

Make a confusion matrix for the random forest model on the test set

In [10]:
import seaborn as sns
cm = metrics.confusion_matrix(Number_test, Number_pred_RF)
plt.figure(figsize=(9,9))
sns.heatmap(cm, annot=True, fmt=".0f", linewidths=.5, square = True, cmap = 'Blues_r');
plt.ylabel('Actual Number');
plt.xlabel('Predicted Number');
all_sample_title = 'Confusion Matrix : Random Forest'
plt.title(all_sample_title, size = 15);

Model 3 : a neural networks model using Keras / Tensorflow

In [11]:
from keras import models
from keras import layers

mnist_nn = models.Sequential()
mnist_nn.add(layers.Dense(512, activation='relu', input_shape=(28 * 28,)))
mnist_nn.add(layers.Dense(512, activation='relu'))
mnist_nn.add(layers.Dense(10, activation='softmax'))

mnist_nn.compile(optimizer='adam',
                loss='categorical_crossentropy',
                metrics=['accuracy'])

Transform the output variables to categories since they are defined as numeric (0,1,..,9)

In [12]:
from keras.utils import to_categorical

Number_train = to_categorical(Number_train)
Number_test = to_categorical(Number_test)

Train the model

In [13]:
mnist_nn.fit(Image_train, Number_train, epochs=5, batch_size=128)

Epoch 1/5
60000/60000 [==============================] - 18s 302us/step - loss: 0.2194 - acc: 0.9353
Epoch 2/5
60000/60000 [==============================] - 16s 261us/step - loss: 0.0808 - acc: 0.9746
Epoch 3/5
60000/60000 [==============================] - 15s 253us/step - loss: 0.0501 - acc: 0.9843
Epoch 4/5
60000/60000 [==============================] - 15s 242us/step - loss: 0.0356 - acc: 0.9887
Epoch 5/5
60000/60000 [==============================] - 14s 238us/step - loss: 0.0272 - acc: 0.9914
Out[13]:
<keras.callbacks.History at 0x23488ed1668>

Make the predictions for test dataset using the neural networks model

In [14]:
test_loss, test_acc = mnist_nn.evaluate(Image_test, Number_test)

10000/10000 [==============================] - 1s 143us/step

Test the accuracy of the model on the test dataset

In [15]:
print('test_acc:', test_acc)

test_acc: 0.9796

Make a confusion matrix for the test dataset to evaluate the prediction accracy

In [16]:
Number_predict = mnist_nn.predict_classes(Image_test)
import seaborn as sns
cm = metrics.confusion_matrix(Number_test.argmax(1), Number_predict)
plt.figure(figsize=(9,9))
sns.heatmap(cm, annot=True, fmt=".0f", linewidths=.5, square = True, cmap = 'Blues_r');
plt.ylabel('Actual Number');
plt.xlabel('Predicted Number');
all_sample_title = 'Confusion Matrix : Neural Network'
plt.title(all_sample_title, size = 15);