Anonymizing Reviewers in Peer Review Discussions
Author Information
Author(s): Charvi Rastogi, Xiangchen Song, Zhijing Jin, Ivan Stelmakh, Hal Daumé III, Kun Zhang, Nihar B. Shah
Primary Institution: Carnegie Mellon University
Hypothesis
Does anonymizing reviewers to each other during peer review discussions affect the dynamics and outcomes of the review process?
Conclusion
The study finds small, significant differences favoring anonymous discussions in terms of reviewer engagement and influence of seniority on final decisions.
Supporting Evidence
- Reviewers in the anonymous condition posted more discussion comments than those in the non-anonymous condition.
- Senior reviewers had a greater influence on final decisions in non-anonymous discussions.
- Politeness scores of discussions were similar across both conditions.
Takeaway
This study looked at whether keeping reviewers' identities secret helps them discuss papers better. It found that when reviewers are anonymous, they talk a bit more and are less influenced by senior reviewers.
Methodology
A randomized controlled trial was conducted with reviewers assigned to either anonymous or non-anonymous discussion conditions, followed by an anonymous survey to gather their experiences.
Potential Biases
Potential bias due to the influence of senior reviewers in non-anonymous settings.
Limitations
The study's response rate was approximately 20%, which may introduce selection bias, and the categorization of reviewers into senior and junior may oversimplify their expertise.
Participant Demographics
The study included 289 reviewers, with a mix of senior and junior researchers.
Statistical Information
P-Value
0.051
Statistical Significance
p<0.05
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website