Ensuring Accuracy and Equity in Vaccination Information From ChatGPT and CDC: Mixed-Methods Cross-Language Evaluation
2024

Evaluating Vaccination Information from ChatGPT and CDC

Sample size: 128 publication 10 minutes Evidence: moderate

Author Information

Author(s): Mavragani Amaryllis, Hong Lingzi, Verma Gaurav, Joshi Saubhagya, Ha Eunbin, Amaya Andee, Mendoza Melissa, Rivera Yonaira, Singh Vivek K

Primary Institution: School of Communication & Information Rutgers University

Hypothesis

How does the quality and language equity of vaccination information from ChatGPT compare to that from the CDC in English and Spanish?

Conclusion

ChatGPT can provide accurate health information, but it needs to improve its readability and cultural sensitivity for better equity.

Supporting Evidence

  • Both ChatGPT and CDC provided mostly accurate and understandable responses.
  • ChatGPT's English responses often exceeded the recommended readability levels.
  • CDC responses outperformed ChatGPT in readability across both languages.
  • Some Spanish responses from ChatGPT were direct translations from English, leading to unnatural phrasing.

Takeaway

This study looked at how well ChatGPT and the CDC give vaccination information in English and Spanish. Both are mostly accurate, but ChatGPT's answers can be hard to read.

Methodology

A comparative analysis of responses to vaccination-related questions from ChatGPT and CDC in English and Spanish, using both quantitative and qualitative assessments.

Potential Biases

Responses in Spanish sometimes appeared to be direct translations from English, which could hinder understanding.

Limitations

The study focused on a single set of FAQs from one agency and evaluated only one LLM technology.

Participant Demographics

Responses evaluated included both English and Spanish speakers.

Statistical Information

P-Value

p<0.05

Statistical Significance

p<0.05

Digital Object Identifier (DOI)

10.2196/60939

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication