Improved variance estimation of classification performance via reduction of bias caused by small sample size
2006

Improved Variance Estimation for Cancer Classification Performance

Sample size: 62 publication 10 minutes Evidence: moderate

Author Information

Author(s): Wickenberg-Bolin Ulrika, Göransson Hanna, Fryknäs Mårten, Gustafsson Mats G, Isaksson Anders

Primary Institution: Uppsala University

Hypothesis

Can we reduce bias in variance estimation of classifier performance caused by small sample sizes?

Conclusion

Modeling and reducing small sample bias allows for improved variance estimates of classifier performance, but the method is not directly applicable to small datasets due to high uncertainty.

Supporting Evidence

  • Simulations showed that small test set sizes lead to biased variance estimates.
  • The RIDT method provides a way to reduce bias in variance estimates.
  • Confidence intervals based on small sample sizes are often misleading.
  • Using larger test sets can stabilize confidence intervals.
  • Traditional methods like RRS may yield overly conservative estimates.

Takeaway

This study shows that when we try to classify cancer using small amounts of data, we can make better guesses about how well our methods work by reducing bias, but it's still tricky with small datasets.

Methodology

The study used simulations to evaluate the performance of a new method called Repeated Independent Design and Test (RIDT) against traditional methods.

Potential Biases

The study highlights that traditional methods may yield biased variance estimates due to dependencies between design and test sets.

Limitations

The method cannot be directly applied to small datasets due to large uncertainty in variance estimates.

Participant Demographics

The study used a colon cancer microarray dataset with 22 normal and 40 cancer cases.

Statistical Information

Confidence Interval

95%

Digital Object Identifier (DOI)

10.1186/1471-2105-7-127

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication