Crowdsourcing with the drift diffusion model of decision making
2024

Using Drift-Diffusion Models in Crowdsourcing

Sample size: 200 publication 10 minutes Evidence: moderate

Author Information

Author(s): Lalvani Shamal, Katsaggelos Aggelos

Primary Institution: Northwestern University

Hypothesis

Can a drift-diffusion model improve the estimation of annotator reliability in crowdsourcing tasks?

Conclusion

The study shows that using a drift-diffusion model for annotator behavior in crowdsourcing can yield accurate predictions and insights into annotator reliability.

Supporting Evidence

  • The drift-diffusion model allows for the recovery of annotator parameters with small mean-squared errors.
  • Results showed similar accuracies to a state-of-the-art technique in predicting crowdsourced data labels.
  • Using neuroscientifically validated models can provide insights into annotator behavior.

Takeaway

This study found a way to better understand how people label data by using a model that explains how they make decisions, which can help improve the quality of crowdsourced data.

Methodology

Two experiments were conducted using synthetic data to test the drift-diffusion model's effectiveness in predicting annotator labels.

Potential Biases

The model assumes independence of annotator decisions, which may not be true in practice.

Limitations

The study relied on synthetic data and assumptions about the decision-making process, which may not hold in real-world scenarios.

Participant Demographics

Ten hypothetical annotators were used in the experiments, with varying drift and diffusion parameters.

Digital Object Identifier (DOI)

10.1038/s41598-024-61687-y

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication