Comparing AI Models for Detecting Suicidal Thoughts
Author Information
Author(s): Adonias Caetano de Oliveira, Renato Freitas Bessa, Ariel Soares Teles
Primary Institution: Escola Nacional de Saúde Pública Sergio Arouca, Fundação Oswaldo Cruz
Hypothesis
This study aims to compare the performance of three variations of BERT models and LLMs for identifying suicidal ideation from nonclinical texts written in Brazilian Portuguese.
Conclusion
Bing/GPT-4 achieved the best performance in detecting suicidal ideation, followed closely by fine-tuned BERT models.
Supporting Evidence
- Bing/GPT-4 achieved 98% accuracy across all metrics.
- BERTimbau-Large performed with 96% accuracy, followed by BERTimbau-Base with 94%.
- Bard had the lowest performance with 62% accuracy.
Takeaway
This study looked at how well different AI models can find signs of suicidal thoughts in text. The best model was Bing/GPT-4, which was really good at this job.
Methodology
The study involved collecting and annotating tweets, applying data preprocessing, and evaluating the performance of various AI models in detecting suicidal ideation.
Potential Biases
The study did not evaluate different prompting strategies for generative models, which could affect performance.
Limitations
The models have not been validated in a clinical setting, and the study only used binary labels for suicidal ideation.
Participant Demographics
The dataset consisted of tweets in Brazilian Portuguese, labeled by psychologists.
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website