Using ChatGPT for Orthopaedic Exam Prep
Author Information
Author(s): Dhruv Mendiratta, Isabel Herzog, Rohan Singh, Ashok Para, Tej Joshi, Michael Vosbikian, Neil Kaushal
Primary Institution: Department of Orthopaedic Surgery Rutgers New Jersey Medical School Newark New Jersey USA
Hypothesis
This study assesses ChatGPT's performance on the Orthopaedic In‐Training Exam (OITE) as a potential study resource for residents.
Conclusion
ChatGPT demonstrates logical, informational, and explicit fallacies which may lead to misinformation and hinder resident education.
Supporting Evidence
- ChatGPT had a success rate of 48.3% on the OITE.
- It demonstrated logical reasoning in 67.6% of the questions.
- The model utilized internal information in 68.1% of the questions.
- Informational fallacy was the most common shortcoming in ChatGPT's responses.
Takeaway
ChatGPT can answer about half of the questions on an orthopaedic exam, but it sometimes gets things wrong, which can confuse students.
Methodology
ChatGPT was tested on 207 questions from the 2022 OITE, with responses evaluated for logical reasoning and information utilization.
Potential Biases
Potential bias from researchers influencing findings, as ChatGPT could have used previous question inputs.
Limitations
The study used only one OITE practice exam, which may limit external validity.
Participant Demographics
Orthopaedic surgery residents preparing for the OITE.
Statistical Information
P-Value
p<0.001
Statistical Significance
p<0.05
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website