The Accuracy and Capability of Artificial Intelligence Solutions in Health Care Examinations and Certificates: Systematic Review and Meta-Analysis
2024

The Accuracy of AI in Health Care Exams

Sample size: 32 publication 10 minutes Evidence: moderate

Author Information

Author(s): de Azevedo Cardoso Taiane, Eysenbach Gunther, Bortkiewicz Alicja, Domingues Nuno, Yao Zonghai, Waldock William J, Zhang Joe, Guni Ahmad, Nabeel Ahmad, Darzi Ara, Ashrafian Hutan

Primary Institution: Imperial College London

Hypothesis

How accurate are large language models (LLMs) when applied to health care examinations compared to human performance standards?

Conclusion

LLMs show promise in providing accurate information for health care decision-making, but there are significant concerns regarding their reliability and potential biases.

Supporting Evidence

  • LLMs achieved an overall medical examination accuracy of 0.61.
  • ChatGPT performed with an overall medical examination accuracy of 0.64.
  • USMLE accuracy for LLMs was found to be 0.51.

Takeaway

This study looked at how well AI can answer medical exam questions. It found that while AI can do a decent job, there are still many things it needs to improve on.

Methodology

A systematic review and meta-analysis were conducted to evaluate the accuracy of LLMs in health care examinations.

Potential Biases

High risk of bias was found in many studies, with concerns about the accuracy of AI responses.

Limitations

The studies reviewed did not adequately address bias, hallucinations, and the variability in reporting standards.

Statistical Information

P-Value

0.51

Confidence Interval

0.58-0.64

Statistical Significance

p<0.05

Digital Object Identifier (DOI)

10.2196/56532

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication