Agreement-based credibility assessment and task replication in human computation systems
作者： Lesandro Ponciano , Francisco Brasileiro
DOI: 10.1016/J.FUTURE.2018.05.028
关键词: Artificial intelligence 、 Task (project management) 、 Measure (data warehouse) 、 Degree (graph theory) 、 Replication (computing) 、 Conceptual framework 、 Focus (computing) 、 Computer science 、 Machine learning 、 Quality (business) 、 Credibility
摘要: Abstract Human computation systems harness the cognitive power of a crowd humans to solve computational tasks for which there are so far no satisfactory fully automated solutions. To obtain quality in results, system usually puts into practice task replication strategy, i.e. same is executed multiple times by different humans. In this study we investigate how improve considering information about credibility score participants. We focus on automatically measure participants while they execute system, and such assessment can be used define, at execution time, suitable degree each task. Based conceptual framework, propose (i) four alternative metrics according agreement among them; (ii) an adaptive credibility-based algorithm that defines, evaluate proposed diversity configurations using data thousands hundreds collected from two real human projects. Results show effective optimising replication, without compromising accuracy obtained answers. doing so, it improves ability properly use provided
sci-hub.se 参考文章(56)
Benjamin Satzger, Harald Psaier, Daniel Schall, Schahram Dustdar, Stimulating skill evolution in market-based crowdsourcing business process management. pp. 66- 82 ,(2011) , 10.1007/978-3-642-23059-2_9
Francisco Vilar Brasileiro, Lesandro Ponciano, Guilherme Gadelha, Task Redundancy Strategy Based on Volunteers’ Credibility for Volunteer Thinking Projects national conference on artificial intelligence. ,(2013)
The Measurement of Interrater Agreement John Wiley & Sons, Inc.. pp. 598- 626 ,(2004) , 10.1002/0471445428.CH18
David B. Fogel, Zbigniew Michalewicz, How to Solve It: Modern Heuristics ,(2004)
Aashish Sheshadri, Matthew Lease, None, SQUARE: A Benchmark for Research on Computing Crowd Consensus national conference on artificial intelligence. ,(2013)
C. Eickhoff, A.P. deVries, How Crowdsourcable is Your Task ,(2011)
Henry Sauermann, Chiara Franzoni, Crowd science user contribution patterns and their implications Proceedings of the National Academy of Sciences of the United States of America. ,vol. 112, pp. 679- 684 ,(2015) , 10.1073/PNAS.1408907112
Walfredo Cirne, Francisco Brasileiro, Daniel Paranhos, Luís Fabrício W. Góes, William Voorsluys, On the efficacy, efficiency and emergent behavior of task replication in large distributed systems parallel computing. ,vol. 33, pp. 213- 234 ,(2007) , 10.1016/J.PARCO.2007.01.002
David Martin, Benjamin V. Hanrahan, Jacki O'Neill, Neha Gupta, Being a turker conference on computer supported cooperative work. pp. 224- 235 ,(2014) , 10.1145/2531602.2531663