[TMP-008] A tale of two consensus. Building consensus in collaborative and self-interested scenarios

Contact person: Jesus Cerquides (j.cerquides@csic.es)

Internal Partners:

1. CSIC Consejo Superior de Investigaciones Científicas, Jesus Cerquides
2. CNR Consiglio Nazionale delle Ricerche, Daniele Vilone   

Many citizen science projects have a crowdsourcing component where several different citizen scientists are requested to fulfill a micro task (such as tagging an image as either relevant or irrelevant for the evaluation of damage in a natural disaster, or identifying a specimen into its taxonomy). How do we create a consensus between the different opinions/votes? Currently, most of the time, simple majority voting is used. We argue that alternative voting schemes (taking into account the errors performed by each annotator) could severely reduce the number of citizen scientists required. This is a clear example of continuous human-in-the-loop machine learning with the machine creating a model of the humans that it has to interact with. We propose to study consensus building under two different hypotheses: truthful annotators (as a model for most voluntary citizen science projects) and self-interested annotators (as a model for paid crowdsourcing projects).

Results Summary

We have contributed to the implementation of several different probabilistic consensus models in the Crowdnalysis library which has been resealed as a Python package.

We have proposed a generic mathematical framework for the definition of probabilistic consensus algorithms, and for performing prospective analysis. This has been published in a journal paper.

We have used the library and the mathematical framework for the analysis of images from the Albanian earthquake scenario.

We exploited Monte Carlo simulations to understand which can be the best way to assess group decisions in evaluating the correct level of damage in natural catastrophes. The results suggest that Majority rule is the best option as long as all the agents are competent enough to address the task. Otherwise, when the number of unqualified agents is no longer negligible, smarter procedures must be found out.

Tangible Outcomes

1. Program/code: Crowdnalysis Python package – Jesus Cerquides (cerquide@iiia.csic.es) https://pypi.org/project/crowdnalysis/