Optimally Splitting Cases for Training and Testing High Dimensional Classifiers
Author Information
Author(s): Kevin K Dobbin, Richard M Simon
Primary Institution: University of Georgia, College of Public Health
Hypothesis
What proportion of the samples should be devoted to the training set, and how does this proportion impact the mean squared error of the prediction accuracy estimate?
Conclusion
The optimal proportion of cases for the training set tends to be in the range of 40% to 80%, depending on the dataset characteristics.
Supporting Evidence
- The optimal proportion was found to depend on the full dataset size and classification accuracy.
- The commonly used strategy of allocating 2/3rd of cases for training was close to optimal for reasonable sized datasets.
- Smaller total sample sizes led to larger proportions devoted to the training set being optimal.
Takeaway
This study helps figure out how to split data into training and testing parts to make better predictions in studies using lots of data, like gene expression.
Methodology
The study developed a non-parametric algorithm and performed simulations to evaluate optimal data splitting strategies.
Potential Biases
Potential bias from using small training sets or not properly separating training and test data.
Limitations
The study may not apply to all types of classifiers and focuses on linear classifiers.
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website