Using Drift-Diffusion Models in Crowdsourcing
Author Information
Author(s): Lalvani Shamal, Katsaggelos Aggelos
Primary Institution: Northwestern University
Hypothesis
Can a drift-diffusion model improve the estimation of annotator reliability in crowdsourcing tasks?
Conclusion
The study shows that using a drift-diffusion model for annotator behavior in crowdsourcing can yield accurate predictions and insights into annotator reliability.
Supporting Evidence
- The drift-diffusion model allows for the recovery of annotator parameters with small mean-squared errors.
- Results showed similar accuracies to a state-of-the-art technique in predicting crowdsourced data labels.
- Using neuroscientifically validated models can provide insights into annotator behavior.
Takeaway
This study found a way to better understand how people label data by using a model that explains how they make decisions, which can help improve the quality of crowdsourced data.
Methodology
Two experiments were conducted using synthetic data to test the drift-diffusion model's effectiveness in predicting annotator labels.
Potential Biases
The model assumes independence of annotator decisions, which may not be true in practice.
Limitations
The study relied on synthetic data and assumptions about the decision-making process, which may not hold in real-world scenarios.
Participant Demographics
Ten hypothetical annotators were used in the experiments, with varying drift and diffusion parameters.
Digital Object Identifier (DOI)
Want to read the original?
Access the complete publication on the publisher's website