To measure the performance of a prediction model, we use the Confusion Matrix.
Let’s say we have a 2 variable classification problem. (Yes/no)
Confusion matrix:
Look at the above picture.
- If the value is actually positive and your model predicted it as positive, then it is a True Positive. The count of TPs will go into TP box
- If the value is actually negative and your model predicted it as positive, then it is a False Positive. The count of FPs will go into FP box
- If the value is actually negative and your model predicted it as negative, then it is a True Negative. The count of TNs will go into TN box
- If the value is actually positive and your model predicted it as negative, then it is a False Negative. The count of FNs will go into FN box
In common parlance, Correct Hits is used to refer to TP, False Alarm is used to refer to FP.
Measures to understand the efficiency of our prediction model:
1. Accuracy: It is the proportion of correct hits in our prediction.
Accuracy = (TP+TN)/TP+TN+FP+FN
2. True Positivity rate(TPR): It is also called Recall or sensitivity.
It means, how many positives out of total positives did your model recall?
Sensitivity = TP/Actual positive = TP/TP+FN
3. Specificity/True Negative Rate: It means how many negatives your model predicted out of total negatives.
Specificity = SR/TNR = TN/TN+FP
4. Precision: Tells us about how less false alarms your model buzzes. Meaning, how less false positives are given.
Precision = TP/TP+FP
Though accuracy tells us how correct our model is, it is important to see how less false alarms are buzzed by our mode. Also, The true positivity rate will be important to see if our model is good enough.
Questions:
- If Sensitivity is high, is our model good?
- If accuracy is very high, without looking at any other value we can approve our model. True of false?
- What is the difference between precision and sensitivity?
- If precision is high, what does this mean? (Hint: Look at the denominator)
Comment your Answers below.
Happy Learning! 🙂
Vipanchi 