Evaluating Language Models for Chinese Medical Exams
Author Information
Author(s): Zhang Siwen, Chu Qi, Li Yujun, Liu Jialu, Wang Jiayi, Yan Chi, Liu Wenxi, Wang Yizhen, Zhao Chengcheng, Zhang Xinyue, Chen Yuwen
Primary Institution: Shenyang Pharmaceutical University
Hypothesis
How do different large language models perform in a Chinese medical examination context?
Conclusion
General LLMs like GPT-4 and ERNIE Bot can effectively understand Chinese medical knowledge, but they still have limitations in specialized knowledge and output stability.
Supporting Evidence
- GPT-4 achieved an accuracy of 65.2% in the CMLE.
- ERNIE Bot scored 61.7%, also meeting CMLE standards.
- DIST-MedLLM only achieved 25.3%, failing to meet the pass standard.
Takeaway
This study looked at how well different computer programs that understand language can answer medical questions in Chinese. Some did really well, but they still have some problems.
Methodology
The study used APIs from three LLMs to assess their performance on the 2023 Chinese Medical Licensing Examination (CMLE) using a dataset of 600 questions.
Potential Biases
The models may avoid high-risk medical suggestions, reflecting a cautious approach.
Limitations
The models showed deficiencies in mastering specialized knowledge and maintaining output stability.
Participant Demographics
The study focused on the performance of LLMs in a Chinese medical context.
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website