Evaluating GPT-4o and o1-preview in Japanese Radiography Certification Exam
Author Information
Author(s): Muacevic Alexander, Adler John R, Goto Hiroki, Shiraishi Yoshioki, Okada Seiji
Primary Institution: Kumamoto University
Hypothesis
Can large language models accurately respond to questions about X-ray safety management and regulations?
Conclusion
GPT-4o performed well in most subjects but failed in relevant laws and regulations, while o1-preview passed all subjects.
Supporting Evidence
- GPT-4o achieved passing accuracy rates in most subjects except for relevant laws and regulations.
- o1-preview met the passing criteria across all subjects.
- The accuracy of o1-preview was significantly higher than that of GPT-4o in all questions.
- Both models struggled with questions related to relevant laws and regulations.
Takeaway
This study tested two AI models to see how well they could answer questions about X-ray safety. One model did well, but the other struggled with laws.
Methodology
The study involved evaluating the performance of GPT-4o and o1-preview on questions from the Japanese certification exam for radiography.
Potential Biases
Potential bias in training data and limitations in understanding Japanese text may affect performance.
Limitations
The AI models may not fully reflect the specific content of the certification exam, and their training data may not include the latest legal updates.
Participant Demographics
The study involved AI models rather than human participants.
Statistical Information
P-Value
0.03
Statistical Significance
p = 0.03
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website