Evaluation of large language models under different training background in Chinese medical examination: a comparative study
2024

Evaluating Language Models for Chinese Medical Exams

Sample size: 600 publication Evidence: moderate

Author Information

Author(s): Zhang Siwen, Chu Qi, Li Yujun, Liu Jialu, Wang Jiayi, Yan Chi, Liu Wenxi, Wang Yizhen, Zhao Chengcheng, Zhang Xinyue, Chen Yuwen

Primary Institution: Shenyang Pharmaceutical University

Hypothesis

How do different large language models perform in a Chinese medical examination context?

Conclusion

General LLMs like GPT-4 and ERNIE Bot can effectively understand Chinese medical knowledge, but they still have limitations in specialized knowledge and output stability.

Supporting Evidence

  • GPT-4 achieved an accuracy of 65.2% in the CMLE.
  • ERNIE Bot scored 61.7%, also meeting CMLE standards.
  • DIST-MedLLM only achieved 25.3%, failing to meet the pass standard.

Takeaway

This study looked at how well different computer programs that understand language can answer medical questions in Chinese. Some did really well, but they still have some problems.

Methodology

The study used APIs from three LLMs to assess their performance on the 2023 Chinese Medical Licensing Examination (CMLE) using a dataset of 600 questions.

Potential Biases

The models may avoid high-risk medical suggestions, reflecting a cautious approach.

Limitations

The models showed deficiencies in mastering specialized knowledge and maintaining output stability.

Participant Demographics

The study focused on the performance of LLMs in a Chinese medical context.

Digital Object Identifier (DOI)

10.3389/frai.2024.1442975

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication