Using Open-Source LLMs for Text Annotation
Author Information
Author(s): Alizadeh Meysam, Kubli Maƫl, Samei Zeynab, Dehghani Shirin, Zahedivafa Mohammadmasiha, Bermeo Juan D., Korobeynikova Maria, Gilardi Fabrizio
Primary Institution: University of Zurich
Hypothesis
Can open-source Large Language Models (LLMs) effectively perform text annotation tasks in political science research?
Conclusion
Fine-tuning open-source LLMs significantly improves their performance in text annotation tasks, making them competitive with proprietary models.
Supporting Evidence
- Fine-tuning improves the performance of open-source LLMs, allowing them to match or surpass zero-shot GPT-3.5 and GPT-4.
- Fine-tuned open-source LLMs can be effectively deployed in a broad spectrum of text annotation applications.
- Open-source LLMs offer significant data protection benefits by not sharing data with third parties.
- Lower temperature settings in LLMs enhance consistency and accuracy in text annotation tasks.
Takeaway
This study shows that open-source models can help researchers label text without needing to pay for expensive services, and they can even do a good job at it.
Methodology
The study compares the performance of various open-source LLMs against proprietary models using multiple datasets and annotation tasks.
Potential Biases
Fine-tuning may introduce biases present in the training datasets, requiring careful selection of data.
Limitations
The findings may not generalize across all text annotation tasks or datasets, and the specific characteristics of the tasks and datasets used may influence performance.
Participant Demographics
The study involved trained political science students and crowd workers from Amazon's Mechanical Turk.
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website