Bias in error estimation when using cross-validation for model selection
2006

Bias in Cross-Validation for Model Selection

Sample size: 40 publication Evidence: moderate

Author Information

Author(s): Varma Sudhir, Simon Richard

Primary Institution: Biometric Research Branch, National Cancer Institute, Bethesda MD, USA

Hypothesis

Using cross-validation error estimates for optimized classifiers may lead to biased estimates of true error on independent data.

Conclusion

Using cross-validation to compute error estimates for tuned classifiers results in significantly biased estimates of true error, but a nested cross-validation procedure can provide almost unbiased estimates.

Supporting Evidence

  • The CV error estimate for the classifier with optimal parameters was found to be a substantially biased estimate of the true error.
  • Performance of the optimized classifiers on the independent test set was no better than chance.
  • The nested CV procedure reduces the bias considerably and gives an estimate of the error that is very close to that obtained on the independent testing set.

Takeaway

When we try to make predictions using a computer model, we need to be careful about how we check if it's working. If we don't do it right, we might think it's better than it really is.

Methodology

The study used cross-validation to optimize classifier parameters for Shrunken Centroids and Support Vector Machines, and compared error estimates from standard and nested cross-validation methods.

Potential Biases

The study highlights the risk of overestimating classifier performance due to biased error estimates from cross-validation.

Limitations

The study primarily focused on 'null' datasets, which may not represent all real-world scenarios.

Digital Object Identifier (DOI)

10.1186/1471-2105-7-91

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication