The Accuracy of AI in Health Care Exams
Author Information
Author(s): de Azevedo Cardoso Taiane, Eysenbach Gunther, Bortkiewicz Alicja, Domingues Nuno, Yao Zonghai, Waldock William J, Zhang Joe, Guni Ahmad, Nabeel Ahmad, Darzi Ara, Ashrafian Hutan
Primary Institution: Imperial College London
Hypothesis
How accurate are large language models (LLMs) when applied to health care examinations compared to human performance standards?
Conclusion
LLMs show promise in providing accurate information for health care decision-making, but there are significant concerns regarding their reliability and potential biases.
Supporting Evidence
- LLMs achieved an overall medical examination accuracy of 0.61.
- ChatGPT performed with an overall medical examination accuracy of 0.64.
- USMLE accuracy for LLMs was found to be 0.51.
Takeaway
This study looked at how well AI can answer medical exam questions. It found that while AI can do a decent job, there are still many things it needs to improve on.
Methodology
A systematic review and meta-analysis were conducted to evaluate the accuracy of LLMs in health care examinations.
Potential Biases
High risk of bias was found in many studies, with concerns about the accuracy of AI responses.
Limitations
The studies reviewed did not adequately address bias, hallucinations, and the variability in reporting standards.
Statistical Information
P-Value
0.51
Confidence Interval
0.58-0.64
Statistical Significance
p<0.05
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website