Variance Estimation of Nominal-Scale Inter-Rater Reliability with Random Selection of Raters

摘要: Most inter-rater reliability studies using nominal scales suggest the existence of two populations inference: population subjects (collection objects or persons to be rated) and that raters. Consequently, sampling variance coefficient can seen as a result combined effect However, all estimators proposed in literature only account for subject variability, ignoring extra due raters, even though latter may biggest components. Such make statistical inference possible universe. This paper proposes will it infer both universes The consistency these is proved well their validity confidence interval construction. These results are applicable fully crossed designs where each rater must rate subject. A small Monte Carlo simulation study presented demonstrate accuracy large-sample approximations on reasonably samples.