Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning
2025

Using Open-Source LLMs for Text Annotation

Sample size: 9777 publication 10 minutes Evidence: moderate

Author Information

Author(s): Alizadeh Meysam, Kubli Maƫl, Samei Zeynab, Dehghani Shirin, Zahedivafa Mohammadmasiha, Bermeo Juan D., Korobeynikova Maria, Gilardi Fabrizio

Primary Institution: University of Zurich

Hypothesis

Can open-source Large Language Models (LLMs) effectively perform text annotation tasks in political science research?

Conclusion

Fine-tuning open-source LLMs significantly improves their performance in text annotation tasks, making them competitive with proprietary models.

Supporting Evidence

  • Fine-tuning improves the performance of open-source LLMs, allowing them to match or surpass zero-shot GPT-3.5 and GPT-4.
  • Fine-tuned open-source LLMs can be effectively deployed in a broad spectrum of text annotation applications.
  • Open-source LLMs offer significant data protection benefits by not sharing data with third parties.
  • Lower temperature settings in LLMs enhance consistency and accuracy in text annotation tasks.

Takeaway

This study shows that open-source models can help researchers label text without needing to pay for expensive services, and they can even do a good job at it.

Methodology

The study compares the performance of various open-source LLMs against proprietary models using multiple datasets and annotation tasks.

Potential Biases

Fine-tuning may introduce biases present in the training datasets, requiring careful selection of data.

Limitations

The findings may not generalize across all text annotation tasks or datasets, and the specific characteristics of the tasks and datasets used may influence performance.

Participant Demographics

The study involved trained political science students and crowd workers from Amazon's Mechanical Turk.

Digital Object Identifier (DOI)

10.1007/s42001-024-00345-9

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication