Comparative analysis of BERT-based and generative large language models for detecting suicidal ideation: a performance evaluation study
2024

Comparing AI Models for Detecting Suicidal Thoughts

Sample size: 100 publication Evidence: high

Author Information

Author(s): Adonias Caetano de Oliveira, Renato Freitas Bessa, Ariel Soares Teles

Primary Institution: Escola Nacional de Saúde Pública Sergio Arouca, Fundação Oswaldo Cruz

Hypothesis

This study aims to compare the performance of three variations of BERT models and LLMs for identifying suicidal ideation from nonclinical texts written in Brazilian Portuguese.

Conclusion

Bing/GPT-4 achieved the best performance in detecting suicidal ideation, followed closely by fine-tuned BERT models.

Supporting Evidence

  • Bing/GPT-4 achieved 98% accuracy across all metrics.
  • BERTimbau-Large performed with 96% accuracy, followed by BERTimbau-Base with 94%.
  • Bard had the lowest performance with 62% accuracy.

Takeaway

This study looked at how well different AI models can find signs of suicidal thoughts in text. The best model was Bing/GPT-4, which was really good at this job.

Methodology

The study involved collecting and annotating tweets, applying data preprocessing, and evaluating the performance of various AI models in detecting suicidal ideation.

Potential Biases

The study did not evaluate different prompting strategies for generative models, which could affect performance.

Limitations

The models have not been validated in a clinical setting, and the study only used binary labels for suicidal ideation.

Participant Demographics

The dataset consisted of tweets in Brazilian Portuguese, labeled by psychologists.

Digital Object Identifier (DOI)

10.1590/0102-311XEN028824

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication