Investigating the accuracy of adjusting for examiner differences in multi-centre Objective Structured Clinical Exams (OSCEs). A simulation study of video-based Examiner Score Comparison and Adjustment (VESCA)
2024

Examiner Score Comparison and Adjustment in OSCEs

Sample size: 113 publication 10 minutes Evidence: moderate

Author Information

Author(s): Peter Yeates, Gareth McCray

Primary Institution: Keele University

Hypothesis

The study aims to investigate the accuracy of adjusted scores produced by Video-based Examiner Score Comparison and Adjustment (VESCA) under various operational parameters.

Conclusion

Score adjustment through VESCA can significantly improve score accuracy in some scenarios while potentially worsening it in others.

Supporting Evidence

  • Score adjustment improved accuracy by up to 71% when baseline differences were significant.
  • 93% of students had more accurate scores in the 20% baseline-difference condition.
  • Without baseline differences, score adjustment often worsened accuracy.

Takeaway

This study looks at how to make exam scores fairer by comparing how different teachers grade students using videos. Sometimes it helps, but sometimes it can make things worse.

Methodology

The study used statistical simulation to analyze how various factors like examiner participation and baseline differences in stringency affect score accuracy.

Potential Biases

Potential biases may arise from the assumptions made in the simulation regarding examiner stringency and student performance.

Limitations

The study's simulations were limited to specific parameters and may not account for all possible influences on OSCE scores.

Participant Demographics

Participants were Year 3 medical students from Keele University.

Digital Object Identifier (DOI)

10.1186/s12909-024-06462-3

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication