Area under the ROC Curve has the most consistent evaluation for binary classification
2024
Evaluating Model Performance in Binary Classification
Sample size: 156
publication
10 minutes
Evidence: high
Author Information
Author(s): Jing Li
Primary Institution: Department of Political Science, University of Illinois at Urbana-Champaign
Hypothesis
How consistent are different metrics at evaluating models across data of varying prevalence?
Conclusion
Metrics less influenced by prevalence provide more consistent evaluations of model performance.
Supporting Evidence
- Evaluation metrics that are less influenced by prevalence offer more consistent evaluation of individual models.
- AUC has the smallest variance in evaluating individual models and ranking of a set of models.
- Metrics that are more influenced by prevalence show larger variance in ranking of different models.
Takeaway
This study looks at how well different ways of measuring model performance work when the data changes, showing that some methods are better than others.
Methodology
Statistical simulation analyzing 18 evaluation metrics across 156 data scenarios with 5 machine learning models.
Limitations
The study focuses on one dataset and may not generalize to others with different characteristics.
Statistical Information
P-Value
p<0.05
Statistical Significance
p<0.05
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website