Improved Variance Estimation for Cancer Classification Performance
Author Information
Author(s): Wickenberg-Bolin Ulrika, Göransson Hanna, Fryknäs Mårten, Gustafsson Mats G, Isaksson Anders
Primary Institution: Uppsala University
Hypothesis
Can we reduce bias in variance estimation of classifier performance caused by small sample sizes?
Conclusion
Modeling and reducing small sample bias allows for improved variance estimates of classifier performance, but the method is not directly applicable to small datasets due to high uncertainty.
Supporting Evidence
- Simulations showed that small test set sizes lead to biased variance estimates.
- The RIDT method provides a way to reduce bias in variance estimates.
- Confidence intervals based on small sample sizes are often misleading.
- Using larger test sets can stabilize confidence intervals.
- Traditional methods like RRS may yield overly conservative estimates.
Takeaway
This study shows that when we try to classify cancer using small amounts of data, we can make better guesses about how well our methods work by reducing bias, but it's still tricky with small datasets.
Methodology
The study used simulations to evaluate the performance of a new method called Repeated Independent Design and Test (RIDT) against traditional methods.
Potential Biases
The study highlights that traditional methods may yield biased variance estimates due to dependencies between design and test sets.
Limitations
The method cannot be directly applied to small datasets due to large uncertainty in variance estimates.
Participant Demographics
The study used a colon cancer microarray dataset with 22 normal and 40 cancer cases.
Statistical Information
Confidence Interval
95%
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website