Diversity of decision-making models and the measurement of interrater agreement.

摘要: Several papers have appeared criticizing the kappa coefficient because of its tendency to fluctuate with sample base rates. The importance these criticisms is difficult evaluate they are presented regards a highly specific model diagnostic decision making. In this article, making viewed as special case signal detection theory. Each process characterized by function that relates probability receiving positive diagnosis severity or salience symptoms. shape diagnosability curve greatly affects value obtained in study interrater reliability, how it changes response variation rates, and closely corresponds validity decisions. common practice evaluating procedure, when criterion diagnoses for comparison unavailable, on basis magnitude observed reliability questionable. New methods measuring agreement necessary, possible directions research area discussed.