Bias in Cross-Validation for Model Selection
Author Information
Author(s): Varma Sudhir, Simon Richard
Primary Institution: Biometric Research Branch, National Cancer Institute, Bethesda MD, USA
Hypothesis
Using cross-validation error estimates for optimized classifiers may lead to biased estimates of true error on independent data.
Conclusion
Using cross-validation to compute error estimates for tuned classifiers results in significantly biased estimates of true error, but a nested cross-validation procedure can provide almost unbiased estimates.
Supporting Evidence
- The CV error estimate for the classifier with optimal parameters was found to be a substantially biased estimate of the true error.
- Performance of the optimized classifiers on the independent test set was no better than chance.
- The nested CV procedure reduces the bias considerably and gives an estimate of the error that is very close to that obtained on the independent testing set.
Takeaway
When we try to make predictions using a computer model, we need to be careful about how we check if it's working. If we don't do it right, we might think it's better than it really is.
Methodology
The study used cross-validation to optimize classifier parameters for Shrunken Centroids and Support Vector Machines, and compared error estimates from standard and nested cross-validation methods.
Potential Biases
The study highlights the risk of overestimating classifier performance due to biased error estimates from cross-validation.
Limitations
The study primarily focused on 'null' datasets, which may not represent all real-world scenarios.
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website