ERIC Number: ED334209
Record Type: RIE
Publication Date: 1991-Apr
Reference Count: N/A
Judges' Agreement and Disagreement Patterns When Encoding Verbal Protocols.
Schael, Jocelyne; Dionne, Jean-Paul
The basis of agreement or disagreement among judges/evaluators when applying a coding scheme to concurrent verbal protocols was studied. The sample included 20 university graduates, from varied backgrounds; 10 subjects had and 10 subjects did not have experience in protocol analysis. The total sample was divided into four balanced groups according to gender and experience in protocol analysis. Subjects were asked to think aloud while encoding two protocols--extracted from an ill-designed task--with a grid of categories provided to them. Subjects' verbalizations and non-verbal behaviors and manipulations were recorded. The level of agreement on the final decisions was analyzed under several criteria. The overall agreement among subjects on the selected codes was 61.4%, but the agreement on the reasons for choosing these codes was 85.1%. This marked difference indicates that the codes selected by the evaluators were not always supported by the reasons stated or that the grid categories are ambiguous. The results show that experience with the task presented (i.e., planning a three-course meal) seemed to be more important for agreement than experience with the technique of protocol analysis. Criteria for improving the quality of a grid and the reliability of coders are suggested. Three data tables and two bar graphs are included. The two study protocols and the coding grid are presented. (Author/RLC)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A
Identifiers: Verbal Routines
Note: Paper presented at the Annual Meeting of the American Educational Research Association (Chicago, IL, April 3-7, 1991).