摘要: Crowdsourcing has become a powerful paradigm for accomplishing work quickly and at scale, but involves significant challenges in quality control. Researchers have developed algorithmic control approaches based on either worker outputs (such as gold standards or agreement) behavior task fingerprinting), each approach serious limitations, especially complex creative work. Human evaluation addresses these limitations does not scale well with increasing numbers of workers. We present CrowdScape, system that supports the human crowd through interactive visualization mixed initiative machine learning. The combines information about outputs, helping users to better understand harness crowd. describe discuss its utility grounded case studies. explore other contexts where CrowdScape's visualizations might be useful, such user
acm.org
sci-hub.se 参考文章(19)
Ohad Shamir, Ofer Dekel, Vox Populi: Collecting High-Quality Labels from a Crowd conference on learning theory. ,(2009)
Rion Snow, Brendan O'Connor, Daniel Jurafsky, Andrew Y. Ng, Cheap and fast---but is it good? Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08. pp. 254- 263 ,(2008) , 10.3115/1613715.1613751
Salman Ahmad, Alexis Battle, Zahan Malkani, Sepander Kamvar, The jabberwocky programming environment for structured social computing Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11. pp. 53- 64 ,(2011) , 10.1145/2047196.2047203
Alfred Inselberg, Bernard Dimsdale, Parallel coordinates: a tool for visualizing multi-dimensional geometry ieee visualization. pp. 361- 378 ,(1990) , 10.5555/949531.949588
Aniket Kittur, Susheel Khamkar, Paul André, Robert Kraut, CrowdWeaver: visually managing complex crowd work conference on computer supported cooperative work. pp. 1033- 1036 ,(2012) , 10.1145/2145204.2145357
Aniket Kittur, Boris Smus, Susheel Khamkar, Robert E. Kraut, CrowdForge Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11. pp. 43- 52 ,(2011) , 10.1145/2047196.2047202
Panagiotis G. Ipeirotis, Foster Provost, Jing Wang, Quality management on Amazon Mechanical Turk knowledge discovery and data mining. pp. 64- 67 ,(2010) , 10.1145/1837885.1837906
Julie S. Downs, Mandy B. Holbrook, Steve Sheng, Lorrie Faith Cranor, Are your participants gaming the system? Proceedings of the 28th international conference on Human factors in computing systems - CHI '10. pp. 2399- 2402 ,(2010) , 10.1145/1753326.1753688
Jeffrey M. Rzeszotarski, Aniket Kittur, Instrumenting the crowd Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11. pp. 13- 22 ,(2011) , 10.1145/2047196.2047199
Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, Katrina Panovich, Soylent Proceedings of the 23nd annual ACM symposium on User interface software and technology - UIST '10. pp. 313- 322 ,(2010) , 10.1145/1866029.1866078