Evaluating AI-Generated Images of Human Anatomy
Author Information
Author(s): Muacevic Alexander, Adler John R, Muhr Paula, Pan Yating, Tumescheit Charlotte, Kübler Ann-Kathrin, Parmaksiz Hatice Kübra, Chen Cheng, Bolaños Orozco Pablo Sebastián, Lienkamp Soeren S, Hastings Janna
Primary Institution: University of Zurich
Hypothesis
Can a systematic approach be developed to evaluate and compare the outputs of different text-to-image generative models in medical applications?
Conclusion
The study presents a method for systematically comparing AI-generated photorealistic images of humans, which can help improve these models for medical applications.
Supporting Evidence
- The error classification system identifies five error types and three severities across five anatomical regions.
- DALL-E 3 performed better than Stable Diffusion in generating images.
- Inter-rater agreement was relatively poor, indicating subjectivity in error classification.
- Prompts involving groups of people led to higher error scores.
- Ethnic diversity was more present in Stable Diffusion models compared to DALL-E 3.
- Configuration errors were the most common across all models.
- Images with groups of people were more challenging for all models than individuals or pairs.
- The study provides a framework for evaluating and comparing errors in generated images.
Takeaway
This study looks at how well AI can create realistic pictures of people based on text descriptions and finds ways to measure how good those pictures are.
Methodology
An error classification system was developed to annotate errors in AI-generated images, applied to a dataset of 240 images from three models using 10 prompts.
Potential Biases
The study observed implicit biases in the generated images, including gender and ethnic representation issues.
Limitations
Subjectivity in error identification and the impracticality of counting errors in images with many individuals were noted as limitations.
Participant Demographics
The dataset aimed for gender balance and included a range of age groups and ethnicities, though most images represented Caucasian individuals.
Statistical Information
P-Value
3.72E-10
Statistical Significance
p<0.05
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website