ERIC Number: EJ842307
Record Type: Journal
Publication Date: 2009
Abstractor: As Provided
Reference Count: 31
Is Teaching Experience Necessary for Reliable Scoring of Extended English Questions?
Royal-Dawson, Lucy; Baird, Jo-Anne
Educational Measurement: Issues and Practice, v28 n2 p2-8 Sum 2009
Hundreds of thousands of raters are recruited internationally to score examinations, but little research has been conducted on the selection criteria for these raters. Many countries insist upon teaching experience as a selection criterion and this has frequently become embedded in the cultural expectations surrounding the tests. Shortages in raters for some of England's national examinations has led to non-teachers being hired to score a small minority of items and changes in technology have fostered this approach. For a National Curriculum test in English taken at age 14, this study investigated whether teaching experience was a necessary selection criterion for all aspects of the examination. Fifty-seven raters with different backgrounds were trained in the normal manner and scored the same 97 students' work. Accuracy was investigated using a cross-classified multilevel model of absolute score differences with accuracy measures at level 1 and raters crossed with candidates at level 2. By comparing the scoring accuracy of graduates with a degree in English, teacher trainees, experienced teachers and experienced raters, this study found that teaching experience was not a necessary selection criterion. A rudimentary model for allocation of raters to different question types is proposed and further research to investigate the limits of necessary qualifications for scoring is suggested.
Descriptors: National Curriculum, Scoring, Foreign Countries, Teaching Experience, Evaluation Criteria, National Competency Tests, English, Evaluation Research, Interrater Reliability, Essay Tests, Secondary School Students
Blackwell Publishing. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8599; Fax: 781-388-8232; e-mail: email@example.com; Web site: http://www.blackwellpublishing.com/jnl_default.asp
Publication Type: Journal Articles; Reports - Research
Education Level: Elementary Secondary Education; Secondary Education
Authoring Institution: N/A
Identifiers - Location: United Kingdom (England)