ERIC Number: ED365091
Record Type: RIE
Publication Date: 1993
Reference Count: N/A
Rater Characteristics and Rater Bias: Implications for Training.
Lumley, Tom; McNamara, T. F.
Recent developments in multi-faceted Rasch measurement (Linacre, 1989) have made possible new kinds of investigations of aspects of performance assessments. Bias analysis, interactions between elements of any facet, can also be analyzed, which permits investigation of the way a particular aspect of the test situation may elicit a consistently biased pattern of responses from a rater. This research investigates the use of these analytical techniques in rater training for the speaking sub-test of the Occupational English Test (OET) administered in Australia, a specific purpose English-as-Second-Language performance test for health professionals. Data are presented from two rater training sessions (30 raters, 10 candidates) separated by a 6-month interval and an intervening operational test administration session (12 of the above raters, 100 candidates). The analysis is used to establish: (1) consistency of rater characteristics over the two or three occasions; and (2) rater bias in relation to role play materials and/or candidate type. The use of feedback to raters of the results of this analysis is also reported. (Contains 23 references.) (Author)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A
Identifiers: Rasch Model; Rater Effects
Note: Paper presented at the Language Testing Research Colloquium (15th, Cambridge, England, United Kingdom, August 1993).