ERIC Number: EJ689993
Record Type: Journal
Publication Date: 2004-Feb
Pages: 9
Abstractor: Author
ISBN: N/A
ISSN: ISSN-0013-1644
EISSN: N/A
A Measure of Agreement for Interval or Nominal Multivariate Observations by Different Sets of Judges
Janson, Harald; Olsson, Ulf
Educational and Psychological Measurement, v64 n1 p62-70 Feb 2004
This article addresses the problem of accounting overall multivariate chance-corrected interobserver agreement when targets have been rated by different sets of judges (not necessarily equal in number). The proposed approach builds on Janson and Olsson's multivariate generalization of Cohen's kappa but incorporates weighting for number of judges and applies an expression for expected disagreement suitable for the case with different judges. The authors suggest that the attractiveness of this approach to multivariate agreement measurement lies in the interpretability of the terms of expected and observed disagreement as average distances between observations, and that addressing agreement without regard to the covariance structure among variables has advantages in simplicity and interpretability. Correspondences to earlier approaches are noted, and application of the proposed measure is exemplified using hypothetical data sets.
Descriptors: Interrater Reliability, Multivariate Analysis, Evaluation Methods, Measurement Techniques, Rating Scales
Sage Publications, 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243 (Toll Free); Fax: 800-583-2665 (Toll Free).
Publication Type: Journal Articles; Reports - Descriptive
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A