Evaluation Metrics for Machine Learning Model

Evaluation metrics play an important role in achieving the optimal model during model training. It is a way to quantify the performance of the Machine learning Model.
For simplicity, I'll be focusing on binary classification. But all the below metrics can be extended for multi-class classification. Let us first understand the basic building block of metrics used to evaluate a classification model. Suppose we have trained a binary image classifier model that predicts whether there is a cat or not in the image.
Positive Class: Cat is a Positive class.
Negative Class: No Cat is the Negative class, It can be anything(dog, horse, background, etc.) except cat in the image.

Fig. 1

Fig. 2

• True Positive (TP): It is an outcome when the model correctly predicts the positive class. i.e the actual image is "Cat" and the model also predicts "Cat".
• False Positive (FP):  It is an outcome when the model incorrectly predicts the positive class. i.e the actual image is "Cat" but the model predicts it a "Not Cat".
• True Negative (TN): It is an outcome when the model correctly predicts the negative class. i.e the actual image is a "Not Cat" and the model also predicts it a "Cat".
• False Negative (FN): It is an outcome when the model incorrectly predicts the negative class. i.e the actual image is a "Cat" but the model predicts it as a "Not Cat".

Classification Accuracy 

It is the ratio of correct prediction to the total number of samples.

Example: If we have 150 testing samples, and our classifier predicts 120 samples correctly. Then accuracy is 120/150 i.e 0.8 or 80%.
For a binary classification problem, accuracy can also be defined as

Recall

\[Recall=\frac{TP}{TP+FN}\]

It is the total number of TP divided by actual positives( TP + FN ).
From the above equation, we can infer the following points

• If a model is not producing any False Negative then recall is 1.
• If a model is not producing any True Positive then recall is 0.

Precision

\[Precision =\frac{TP}{TP+FP}\]

It is the total number of TP divide by the total number of predicted positives( TP + FP ).

From the above equation, we can infer the following points

• If a model is not producing any False Positive then precision is 1.
• If a model is not producing any True Positive then precision is 0.

F1 Score

\[F1=2 . \frac{Precision . Recall }{Precision + Recall }\]

It is the harmonic mean of precision and recall.

Confusion Matrix

It is also known as error Matrix, The term 'confusion' is used because it checks if the system is getting confused between classes. It gives more insight into the performance of a classification model. Fig. 2 is a matrix for multi class classification problems.

Let's again take an example of multi-class classification problem where confusion matrix helps a lot in understanding the performance and cause of error in our trained model.
If we have a trained model on CIFAR-10 data, and perform classification on test dataset. The confusion matrix is plotted in Fig.3

• The vertical axis denotes the actual class and horizontal axis denotes the predicted class. 
• The sum of all the numbers in any row reflects the total number of images used for testing of corresponding class. Sum of all values in any row is 1000 because we have used 1000 samples of each class for testing.
• The sum of all the numbers in any column reflects the total number of prediction for corresponding class.

Now, look at the first row corresponding to "Cat" class. We can infer that 826 samples are predicted as "Cat",48 are predicted as "Dog", 12 are predicted as "Horse" and so on... all of which are actually "Cat". It means we have predicted 826 samples correctly.

We can also infer that Cat is getting confused mostly with "Dog".

From second row we can infer that 801 "Dog" images samples are correclty predicted, but 111 samples of "Dog" are predicted as "Cat".

From this we infer that "Cat" and "Dog" are confused with each other, and our accuracy is dropping  beacause of this isssue.
Using the above insights we can do necessary changes in our model and data.

Thanks for reading.