NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: EJ1056261
Record Type: Journal
Publication Date: 2014
Pages: 27
Abstractor: As Provided
ISSN: ISSN-1302-597X
Crossed Random-Effect Modeling: Examining the Effects of Teacher Experience and Rubric Use in Performance Assessments
Kan, Adnan; Bulut, Okan
Eurasian Journal of Educational Research, n57 p1-27 2014
Problem Statement: Performance assessments have emerged as an alternative method to measure what a student knows and can do. One of the shortcomings of performance assessments is the subjectivity and inconsistency of raters in scoring. A common criticism of performance assessments is the subjective nature of scoring procedures. The effectiveness of the performance assessment procedure depends highly on the quality and coordination of teacher and rubric. To gain a better understanding of the interaction between teachers and performance assessments, it is crucial to examine the effects of teacher-related factors and how teachers interact with scoring rubrics when grading performance assessments. One of these factors is teachers' work and scoring experience. When grading performance assessments, the experienced teachers may be expected to grade student performances more objectively through their experience in instruction and evaluation than the teachers with less teaching and scoring experience. Purpose of the study: This study investigates the impact of rubric use and teaching experience on teachers' scoring behaviors in performance assessments. The effects of teaching experience and rubric use on the consistency of scores assigned by teachers is examined through an empirical study. Methods: Crossed random-effects modeling was used to estimate rater effects, consistency among the teachers, and the effect of teaching experience. Findings and Results: Results indicated that lack of a scoring guide may cause the teachers to establish their performance criteria and score tasks inconsistently. When teachers used a rubric, inter-rater reliability substantially increased. Experienced teachers and teachers with little teaching experience exhibited different severity patterns in scoring. Conclusion and Recommendations: Based upon the results of this study, it appears that teachers who have more teaching experience tend to score performance tasks more leniently than teachers who do not have long years of teaching experience. The differences in the teachers' scoring due to their teaching experience became negligible when all teachers used a scoring rubric. In addition to teaching experience, the potential effects of other external factors should also be considered to make the use of rubrics more effective in performance assessments. This study illustrated an alternative methodology to estimate variance components and the effects of fixed factors within the same analysis. A big advantage of this modeling approach over generalizability theory is that it allows for the separation of random and fixed effects from each other. Although the findings of this study enrich the limited knowledge about the effects of rubric use and teaching experience on teachers' scoring behaviors, further research is needed to understand the reasons why these factors are influential.
ANI Publishing Company. Kizilirmak Sokak 10/A Bakanliklar, Ankara, Turkey 06440. Tel: +90-312-425-8150; Fax: +90-312-425-8111; e-mail:; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: Grade 8; Junior High Schools; Middle Schools; Elementary Education; Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A