A crowdsourced system for user studies in information extraction
作者： Zohreh Khojasteh Ghamari
DOI: 10.1504/IJKESDP.2017.10010607
关键词: User studies 、 Entity linking 、 Information retrieval 、 World Wide Web 、 Subsequence 、 Meaning (linguistics) 、 Word (computer architecture) 、 Information extraction 、 Crowdsourcing 、 Engineering 、 Set (abstract data type)
摘要: In this paper, from an entity linking (EL) system, we take a set of tweets, where some subsequence words is annotated with possible meaning/entities and these entities are linked several Wikipedia pages. We propose model using crowdsourcing to disambiguate decide about the accurate page that must be definite word/spot. discuss importance compare different systems at end, introduce crowdflower. crowdflower features in particular. Finally, analyse output reports present novel approach select reliable results. summary, our observations show results have confidence rate over 0.5.
sci-hub.st 参考文章(7)
Gabriella Kazai, In Search of Quality in Crowdsourcing for Search Engine Evaluation Lecture Notes in Computer Science. pp. 165- 176 ,(2011) , 10.1007/978-3-642-20161-5_17
Gabriella Kazai, Jaap Kamps, Marijn Koolen, Natasa Milic-Frayling, Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking international acm sigir conference on research and development in information retrieval. pp. 205- 214 ,(2011) , 10.1145/2009916.2009947
Julie S. Downs, Mandy B. Holbrook, Steve Sheng, Lorrie Faith Cranor, Are your participants gaming the system? Proceedings of the 28th international conference on Human factors in computing systems - CHI '10. pp. 2399- 2402 ,(2010) , 10.1145/1753326.1753688
Omar Alonso, Daniel E. Rose, Benjamin Stewart, Crowdsourcing for relevance evaluation ACM SIGIR Forum. ,vol. 42, pp. 9- 15 ,(2008) , 10.1145/1480506.1480508
Mark Dredze, Tim Finin, Justin Martineau, Anand Karandikar, Nicholas Keller, William Murnane, Annotating Named Entities in Twitter Data with Crowdsourcing north american chapter of the association for computational linguistics. pp. 80- 88 ,(2010)
Aniket Kittur, Ed H. Chi, Bongwon Suh, Crowdsourcing user studies with Mechanical Turk Proceeding of the twenty-sixth annual CHI conference on Human factors in computing systems - CHI '08. pp. 453- 456 ,(2008) , 10.1145/1357054.1357127
Gabriella Kazai, Natasa Milic-Frayling, On the Evaluation of the Quality of Relevance Assessments Collected through Crowdsourcing ,(2009)