Effectiveness of ChatGPT-4 for Shoulder Surgery Information
Author Information
Author(s): Jacob F. Oeding, Amy Z. Lu, Michael Mazzucco, Michael C. Fu, David M. Dines, Russell F. Warren, Lawrence V. Gulotta, Joshua S. Dines, Kyle N. Kunze
Primary Institution: Department of Orthopaedics, Institute of Clinical Sciences, The Sahlgrenska Academy University of Gothenburg
Hypothesis
Medical information retrieved by ChatGPT-4 would be accurate and trustworthy compared to Google.
Conclusion
ChatGPT-4 provided trustworthy academic sources for medical information retrieval concerning total shoulder arthroplasty, while Google sources were heterogeneous.
Supporting Evidence
- ChatGPT-4 used 100% academic sources for answers.
- 80% of FAQs with numeric responses had identical answers between ChatGPT-4 and Google.
- Accuracy of information was not significantly different between ChatGPT-4 and Google (p=0.32).
- ChatGPT-4 provided a higher proportion of trustworthy academic sources compared to Google.
Takeaway
ChatGPT-4 can help answer questions about shoulder surgery using reliable academic sources, while Google gives mixed results.
Methodology
A patient-replicated query for 'total shoulder replacement' was performed using both Google and ChatGPT-4, and the accuracy of the information was graded by two blinded orthopaedic surgeons.
Potential Biases
Potential bias from the sources used by Google and the inherent training data of ChatGPT-4.
Limitations
The study only compared ChatGPT-4 with Google and did not include other platforms, limiting the generalizability of the findings.
Statistical Information
P-Value
p=0.001
Statistical Significance
p<0.05
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website