Rating crowdsourced annotations: evaluating contributions of variable quality and completeness

摘要: Crowdsourcing has become a popular means to acquire data about the Earth and its environment inexpensively, but data-sets obtained are typically imperfect of unknown quality. Two common imperfections with crowdsourced contributions from cheats or spammers missing cases. The effect latter two on method evaluate accuracy via latent class model was explored. Using simulated real data-sets, it shown that is able derive useful information even when degree imperfection very high. practical potential this ability obtain within geospatial sciences realm Digital applications indicated reference an evaluation building damage maps produced by multiple bodies after 2010 earthquake in Haiti. Critically, allowed be ranked approximately correct order could help ensure most appropriate used.