Comparing AI Models for Detecting Suicidal Thoughts
Author Information
Author(s): Adonias Caetano de Oliveira, Renato Freitas Bessa, Ariel Soares Teles
Primary Institution: Escola Nacional de Saúde Pública Sergio Arouca, Fundação Oswaldo Cruz
Hypothesis
This study aims to compare the performance of three variations of BERT models and LLMs for identifying suicidal ideation from nonclinical texts written in Brazilian Portuguese.
Conclusion
Bing/GPT-4 performed best in identifying suicidal ideation, followed closely by BERTimbau models, suggesting their potential for clinical use.
Supporting Evidence
- Bing/GPT-4 achieved 98% accuracy in identifying suicidal ideation.
- BERTimbau-Large and BERTimbau-Base models showed accuracy rates of 96% and 94%, respectively.
- ChatGPT-3.5 achieved 81% accuracy, while Google Bard performed the worst with 62% accuracy.
Takeaway
Researchers tested different AI models to see which one is best at finding signs of suicidal thoughts in text. They found that one model, Bing/GPT-4, was the best at this job.
Methodology
The study involved collecting and annotating tweets, followed by training and testing various AI models to classify the presence of suicidal ideation.
Potential Biases
The dataset was labeled by psychologists, but the study did not evaluate different prompting strategies for generative models.
Limitations
The models have not been validated in a clinical setting, and the study only used binary labels for suicidal ideation.
Participant Demographics
The dataset consisted of tweets in Brazilian Portuguese, labeled by psychologists.
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website