This study adopted a mixed-methods cross-sectional approach to identify similarities and differences in the English as a second language (ESL) essay holistic scores and evaluation criteria of raters with different levels of experience. Each of 31 experienced and 29 novice raters rated a sample of ESL essays holistically and analytically and provided written explanations for each holistic score they assigned. Score and qualitative data analyses were conducted to identify the criteria that the raters employed to rate the essays holistically. The findings indicated that both groups gave more importance to the communicative quality of the essays than to other aspects of writing. However, the novice raters tended to be more lenient and to give more importance to argumentation than the experienced raters did. The experienced raters tended to be more severe, to give more importance to linguistic accuracy, and to refer to evaluation criteria other than those listed in the rating scale more frequently than the novices did. The article concludes with a call for longitudinal research to investigate to what extent, how, and why rater evaluation criteria change over time and across contexts. (Contains 5 tables and 4 footnotes.)