Evaluating Peer Review for Funding Applications
Author Information
Author(s): Bornmann Lutz, Wallon Gerlind, Ledin Anna
Primary Institution: ETH Zurich
Hypothesis
Does the peer review process select the best applicants for funding based on their scientific performance?
Conclusion
The study found a statistically significant association between selection decisions and the scientific achievements of applicants, although there were notable errors in the selection process.
Supporting Evidence
- Approved applicants published more papers than rejected ones.
- Median citation counts for approved applicants were higher than for rejected applicants.
- Type II errors were more frequent than type I errors in the selection process.
Takeaway
The study looked at how well a committee picks scientists for funding. It found that while they usually choose good scientists, sometimes they make mistakes and reject talented ones.
Methodology
The study analyzed publication and citation data for applicants to two EMBO funding programs using negative binomial regression models.
Potential Biases
There is a risk of type I and type II errors in the selection process, leading to both overestimation and underestimation of applicants' future performance.
Limitations
The findings may not be generalizable as the study only included applicants from specific funding programs.
Participant Demographics
The study included 668 applicants to the Long-Term Fellowship program and 297 applicants to the Young Investigator program, primarily in the life sciences.
Statistical Information
P-Value
p<0.001
Confidence Interval
[15%–50%]
Statistical Significance
p<0.05
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website