Evaluating Vaccination Information from ChatGPT and CDC
Author Information
Author(s): Mavragani Amaryllis, Hong Lingzi, Verma Gaurav, Joshi Saubhagya, Ha Eunbin, Amaya Andee, Mendoza Melissa, Rivera Yonaira, Singh Vivek K
Primary Institution: School of Communication & Information Rutgers University
Hypothesis
How does the quality and language equity of vaccination information from ChatGPT compare to that from the CDC in English and Spanish?
Conclusion
ChatGPT can provide accurate health information, but it needs to improve its readability and cultural sensitivity for better equity.
Supporting Evidence
- Both ChatGPT and CDC provided mostly accurate and understandable responses.
- ChatGPT's English responses often exceeded the recommended readability levels.
- CDC responses outperformed ChatGPT in readability across both languages.
- Some Spanish responses from ChatGPT were direct translations from English, leading to unnatural phrasing.
Takeaway
This study looked at how well ChatGPT and the CDC give vaccination information in English and Spanish. Both are mostly accurate, but ChatGPT's answers can be hard to read.
Methodology
A comparative analysis of responses to vaccination-related questions from ChatGPT and CDC in English and Spanish, using both quantitative and qualitative assessments.
Potential Biases
Responses in Spanish sometimes appeared to be direct translations from English, which could hinder understanding.
Limitations
The study focused on a single set of FAQs from one agency and evaluated only one LLM technology.
Participant Demographics
Responses evaluated included both English and Spanish speakers.
Statistical Information
P-Value
p<0.05
Statistical Significance
p<0.05
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website