An Approach to Information Retrieval Based on Statistical Model Selection

摘要: Abstract Building on previous work in the field of language modeling information retrieval (IR), this paper proposes a novel approach to document ranking based statistical model selection. The proposed offers two main contributions. First, we posit notion document’s “null model,” that conditions our assessment model’s significance with respect query. Second, introduce an information-theoretic complexity penalty into ranking. We rank documents penalized log-likelihood ratio comparing probability each generated query versus likelihood corresponding “null” it. Each is assessed by Akaike criterion (AIC), expected Kullback-Leibler divergence between observed (null or non-null) and underlying data. report experimental results where selection improvement over traditional LM retrieval.