An Active Learning Approach to Ecien tly Ranking Retrieval Engines

摘要: Evaluating retrieval systems, such as those submitted to the annual TREC competition, usually requires a large number of documents be read and judged for relevance query topics. Test collections are far too big exhaustively judged, so only subset is selected form judgment \pool." The selection method that uses produces pools still quite large. Research has indicated it possible rank systems correctly using substantially smaller pools. This paper introduces an active learning algorithm whose goal reach correct rankings smallest judgments. It adds one document pool at time, always trying select with highest information gain. Several variants this described, each improvements on before. Results from experiments included comparison traditional pooling method. best version reliably outperforms method, although its degree improvement varies.