Evaluating ChatGPT's Performance on Oculoplastic Surgery Exam Questions
Author Information
Author(s): Muacevic Alexander, Adler John R, Gill Gurnoor S, Blair Jacob, Litinsky Steven
Primary Institution: Florida Atlantic University Charles E. Schmidt College of Medicine
Hypothesis
This study aims to evaluate the performance of ChatGPT versions 3.5 and 4.0 on image-based and text-only questions in the oculoplastic subspecialty.
Conclusion
ChatGPT-4.0 performs better than ChatGPT-3.5 on oculoplastic surgery questions, but still struggles with image-based questions.
Supporting Evidence
- ChatGPT-3.5 answered 56.85% of text-only questions correctly, while ChatGPT-4.0 achieved 73.46%.
- The biserial correlation between ChatGPT-3.5 and human performance was 0.198.
- ChatGPT-4.0's accuracy on image-only questions was 56.94%, significantly lower than on text-only questions.
Takeaway
This study shows that ChatGPT-4.0 is better at answering questions about eye surgery than the older version, but it still has trouble with questions that include pictures.
Methodology
The study analyzed 343 text-only questions and 216 image-based questions from StatPearls and OphthoQuestions, comparing the performance of ChatGPT-3.5 and ChatGPT-4.0.
Limitations
The study relied solely on StatPearls data, which may not be comprehensive, and ChatGPT-4.0's performance on image-based questions was still limited.
Statistical Information
P-Value
0.0004
Statistical Significance
p<0.05
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website