Design and Implementation of Relevance Assessments Using Crowdsourcing

摘要: In the last years crowdsourcing has emerged as a viable platform for conducting relevance assessments. The main reason behind this trend is that makes possible to conduct experiments extremely fast, with good results and at low cost. However, like in any experiment, there are several details would make an experiment work or fail. To gather useful results, user interface guidelines, inter-agreement metrics, justification analysis important aspects of successful experiment. we explore design execution judgments using Amazon Mechanical Turk platform, introducing methodology assessments series TREC 8 fixed budget. Our findings indicate workers experts, even providing detailed feedback certain query-document pairs. We also importance document presentation when performing assessment tasks. Finally, show our examples interesting their own.