Human-Powered Top-k Lists

摘要: We propose an algorithm that obtains the top-k list of items out a larger itemset, using human workers (e.g., through crowdsourcing) to perform comparisons among items. An example application is finding best photographs in large collection by asking humans evaluate different photos. Our has address several challenges: obtaining worker input high latency; may disagree on their judgments for same items; some provide wrong purpose; and, there varying difficulty comparing experimental evidence good performance algorithm, extensive simulations and actual experiments with from Amazon’s Mechanical Turk.