Reliable information retrieval evaluation with incomplete and biased judgements

摘要: Information retrieval evaluation based on the pooling method is inherently biased against systems that did not contribute to pool of judged documents. This may distort results obtained about relative quality evaluated and thus lead incorrect conclusions performance a particular ranking technique.We examine magnitude this effect explore how it can be countered by automatically building an unbiased set judgements from original, through pooling. We compare with other approaches problem incomplete judgements, such as bpref, show proposed leads higher accuracy, especially if manual rich in documents, but highly some systems.