Comparative analysis of BERT-based and generative large language models for detecting suicidal ideation: a performance evaluation study
2024

Comparing AI Models for Detecting Suicidal Thoughts

Sample size: 100 publication Evidence: high

Author Information

Author(s): Adonias Caetano de Oliveira, Renato Freitas Bessa, Ariel Soares Teles

Primary Institution: Escola Nacional de Saúde Pública Sergio Arouca, Fundação Oswaldo Cruz

Hypothesis

This study aims to compare the performance of three variations of BERT models and LLMs for identifying suicidal ideation from nonclinical texts written in Brazilian Portuguese.

Conclusion

Bing/GPT-4 performed best in identifying suicidal ideation, followed closely by BERTimbau models, suggesting their potential for clinical use.

Supporting Evidence

  • Bing/GPT-4 achieved 98% accuracy in identifying suicidal ideation.
  • BERTimbau-Large and BERTimbau-Base models showed accuracy rates of 96% and 94%, respectively.
  • ChatGPT-3.5 achieved 81% accuracy, while Google Bard performed the worst with 62% accuracy.

Takeaway

Researchers tested different AI models to see which one is best at finding signs of suicidal thoughts in text. They found that one model, Bing/GPT-4, was the best at this job.

Methodology

The study involved collecting and annotating tweets, followed by training and testing various AI models to classify the presence of suicidal ideation.

Potential Biases

The dataset was labeled by psychologists, but the study did not evaluate different prompting strategies for generative models.

Limitations

The models have not been validated in a clinical setting, and the study only used binary labels for suicidal ideation.

Participant Demographics

The dataset consisted of tweets in Brazilian Portuguese, labeled by psychologists.

Digital Object Identifier (DOI)

10.1590/0102-311XEN028824

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication