QASCA: A Quality-Aware Task Assignment System for Crowdsourcing Applications

摘要: A crowdsourcing system, such as the Amazon Mechanical Turk (AMT), provides a platform for large number of questions to be answered by Internet workers. Such systems have been shown useful solve problems that are difficult computers, including entity resolution, sentiment analysis, and image recognition. In this paper, we investigate online task assignment problem: Given pool n questions, which k should assigned worker? poor may not only waste time money, but also hurt quality application depends on workers' answers. We propose consider measures (also known evaluation metrics) relevant an during process. Particularly, explore how Accuracy F-score, two widely-used metrics applications, can facilitate assignment. Since these assume ground truth question is known, study their variants make use probability distributions derived from further strategies, enables optimal assignments. algorithms expensive, solutions attain high in linear time. develop system called Quality-Aware Task Assignment System Crowdsourcing Applications (QASCA) top AMT. evaluate our approaches five real applications. find QASCA efficient, attains better result (of more than 8% improvement) compared with existing methods.