Evaluating the Performance of ChatGPT 3.5 and 4.0 on StatPearls Oculoplastic Surgery Text- and Image-Based Exam Questions
2024

Evaluating ChatGPT's Performance on Oculoplastic Surgery Exam Questions

Sample size: 559 publication Evidence: moderate

Author Information

Author(s): Muacevic Alexander, Adler John R, Gill Gurnoor S, Blair Jacob, Litinsky Steven

Primary Institution: Florida Atlantic University Charles E. Schmidt College of Medicine

Hypothesis

This study aims to evaluate the performance of ChatGPT versions 3.5 and 4.0 on image-based and text-only questions in the oculoplastic subspecialty.

Conclusion

ChatGPT-4.0 performs better than ChatGPT-3.5 on oculoplastic surgery questions, but still struggles with image-based questions.

Supporting Evidence

  • ChatGPT-3.5 answered 56.85% of text-only questions correctly, while ChatGPT-4.0 achieved 73.46%.
  • The biserial correlation between ChatGPT-3.5 and human performance was 0.198.
  • ChatGPT-4.0's accuracy on image-only questions was 56.94%, significantly lower than on text-only questions.

Takeaway

This study shows that ChatGPT-4.0 is better at answering questions about eye surgery than the older version, but it still has trouble with questions that include pictures.

Methodology

The study analyzed 343 text-only questions and 216 image-based questions from StatPearls and OphthoQuestions, comparing the performance of ChatGPT-3.5 and ChatGPT-4.0.

Limitations

The study relied solely on StatPearls data, which may not be comprehensive, and ChatGPT-4.0's performance on image-based questions was still limited.

Statistical Information

P-Value

0.0004

Statistical Significance

p<0.05

Digital Object Identifier (DOI)

10.7759/cureus.73812

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication