Tutorial 6
Evaluation Metrics –Part 2 TA:
How to Choose an Appropriate Performance Metrics?
Copyright By PowCoder代写 加微信 powcoder
1. Look at model objective and target classes
• The effect of false positives and false negatives can be very different
• The costs associated with misclassifying each of the classes can give insight into the appropriate performance metric
Example 1:
spam detection in emails
• Appropriate metric: precision or specificity
• Costs associated with false positives can be significantly higher than those associated with false negatives
Example 2:
Cancer detection
• Appropriate metric: recall
• Costs associated with false negatives can be significantly higher than those associated with false positives
Example 3:
• Anomaly detection in transactions
• Appropriate metric: recall
• Costs associated with false negatives can be significantly higher than those associated with false positives
How to Choose an Appropriate Performance Metrics?
2. Look at the distribution of the target classes
Example: A classification model is developed to detect a rare disease. Based on historical data, this disease has occurred in approximately 2% of the cases. Which performance metric would you think is better for evaluating the performance of the model, accuracy or f1-score?
Answer: f1-score
How to Choose an Appropriate Performance Metrics?
• Accuracy cannot be a reliable performance metric in cases where there is high imbalance in target classes.
• If the goal of a model is to optimize the accuracy on a highly- imbalanced dataset, it can learn to classify all datapoints into the major class.
• Because a high proportion of the datapoints belong to the predicted class, the accuracy would be very high.
• However, as this model does not identify the minority class well, we cannot interpret it a good model.
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com