Unbiased Ranking Evaluation on a Budget

摘要: We address the problem of assessing quality a ranking system (e.g., search engine, recommender system, review ranker) given fixed budget for collecting expert judgments. In particular, we propose method that selects which items to judge in order optimize accuracy estimate. Our is not only efficient, but also provides estimates are unbiased --- unlike common approaches tend underestimate performance or have bias against new systems evaluated re-using previous relevance scores.