Comparing Large Language Models to Doctors in Triage and Diagnosis
Author Information
Author(s): Mavragani Amaryllis, Williams Christopher, Shi Qiming, Sorich Michael Joseph, Mangoni Arduino Aleksander, Bacchi Stephen, Menz Bradley Douglas, Hopkins Ashley Mark
Primary Institution: College of Medicine and Public Health Flinders University
Hypothesis
Are contemporary large language models (LLMs) as accurate as physicians in triage and diagnostic tasks?
Conclusion
Contemporary frontier LLMs have significantly improved in diagnostic and triage accuracy compared to earlier models and perform similarly to physicians.
Supporting Evidence
- The correct diagnosis was among the top three proposed diagnoses for 98.6% of cases evaluated by frontier LLMs.
- The most likely diagnosis prediction was correct for 86.8% of cases.
- Triage was correct for 92.4% of cases evaluated by frontier LLMs.
Takeaway
This study shows that new AI models can help doctors make better decisions about patient care, almost as well as real doctors do.
Methodology
The study evaluated the performance of three large language models on 48 clinical case vignettes, comparing their triage and diagnostic accuracy to that of physicians.
Limitations
The study had a relatively small sample size of cases evaluated.
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website