Reviewer Agreement Trends in Conference Abstract Submissions
Author Information
Author(s): Brian H Rowe, Trevor L Strome, Carol Spooner, Sandra Blitz, Eric Grafstein, Andew Worster
Primary Institution: University of Alberta
Hypothesis
What is the level of inter-rater agreement between reviewers on the quality of abstract submissions?
Conclusion
The study found moderate agreement among reviewers regarding the overall quality of submitted abstracts, with better agreement on objective criteria.
Supporting Evidence
- The mean score for submitted abstracts over the four years was 14.4.
- ICC values for total scores increased from fair to moderate over the years.
- Reviewers agreed less on individual criteria than on the total score.
Takeaway
This study looked at how well reviewers agree on the quality of research abstracts submitted to a conference, finding that they generally agree but sometimes disagree on specific details.
Methodology
Reviewers scored abstracts based on nine criteria using an online form, and the final score was the mean of three reviewers' scores calculated using Intraclass Correlation Coefficient (ICC).
Potential Biases
There is a risk of bias due to the small community of emergency medicine researchers and the potential for reviewers to score based on personal preferences.
Limitations
The study faced limitations such as a small pool of reviewers, potential biases due to the small research community, and variability in reviewer expertise.
Participant Demographics
Reviewers were primarily from the Canadian Association of Emergency Physicians, with some submitters from outside Canada.
Statistical Information
P-Value
0.23
Confidence Interval
95% CI: 14.1–14.6
Statistical Significance
p = 0.23
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website