Effectiveness of a large language model for clinical information retrieval regarding shoulder arthroplasty
2024

Effectiveness of ChatGPT-4 for Shoulder Surgery Information

publication Evidence: moderate

Author Information

Author(s): Jacob F. Oeding, Amy Z. Lu, Michael Mazzucco, Michael C. Fu, David M. Dines, Russell F. Warren, Lawrence V. Gulotta, Joshua S. Dines, Kyle N. Kunze

Primary Institution: Department of Orthopaedics, Institute of Clinical Sciences, The Sahlgrenska Academy University of Gothenburg

Hypothesis

Medical information retrieved by ChatGPT-4 would be accurate and trustworthy compared to Google.

Conclusion

ChatGPT-4 provided trustworthy academic sources for medical information retrieval concerning total shoulder arthroplasty, while Google sources were heterogeneous.

Supporting Evidence

  • ChatGPT-4 used 100% academic sources for answers.
  • 80% of FAQs with numeric responses had identical answers between ChatGPT-4 and Google.
  • Accuracy of information was not significantly different between ChatGPT-4 and Google (p=0.32).
  • ChatGPT-4 provided a higher proportion of trustworthy academic sources compared to Google.

Takeaway

ChatGPT-4 can help answer questions about shoulder surgery using reliable academic sources, while Google gives mixed results.

Methodology

A patient-replicated query for 'total shoulder replacement' was performed using both Google and ChatGPT-4, and the accuracy of the information was graded by two blinded orthopaedic surgeons.

Potential Biases

Potential bias from the sources used by Google and the inherent training data of ChatGPT-4.

Limitations

The study only compared ChatGPT-4 with Google and did not include other platforms, limiting the generalizability of the findings.

Statistical Information

P-Value

p=0.001

Statistical Significance

p<0.05

Digital Object Identifier (DOI)

10.1002/jeo2.70114

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication