ERIC Number: EJ1106355
Record Type: Journal
Publication Date: 2016-Aug
Abstractor: As Provided
Reference Count: 22
Testing the Difference of Correlated Agreement Coefficients for Statistical Significance
Gwet, Kilem L.
Educational and Psychological Measurement, v76 n4 p609-637 Aug 2016
This article addresses the problem of testing the difference between two correlated agreement coefficients for statistical significance. A number of authors have proposed methods for testing the difference between two correlated kappa coefficients, which require either the use of resampling methods or the use of advanced statistical modeling techniques. In this article, we propose a technique similar to the classical pairwise "t" test for means, which is based on a large-sample linear approximation of the agreement coefficient. We illustrate the use of this technique with several known agreement coefficients including Cohen's kappa, Gwet's AC[subscript 1], Fleiss's generalized kappa, Conger's generalized kappa, Krippendorff's alpha, and the Brenann-Prediger coefficient. The proposed method is very flexible, can accommodate several types of correlation structures between coefficients, and requires neither advanced statistical modeling skills nor considerable computer programming experience. The validity of this method is tested with a Monte Carlo simulation.
Descriptors: Differences, Correlation, Statistical Significance, Statistical Analysis, Monte Carlo Methods
SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: email@example.com; Web site: http://sagepub.com
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Authoring Institution: N/A