Although raters can be trained to evaluate the lexical qualities of student essays, the question remains as to what extent raters follow the "lexis" scale descriptors in the rating scale when evaluating or rate according to their own criteria. The current study examines the extent to which 27 trained university EFL raters take various lexical qualities into account while using an analytic rating scale to assess timed essays. In this experiment, the lexical content of 27 essays was manipulated before rating. This was done in order to determine if raters were sensitive to range, accuracy or sophistication when rating writing for lexis. Using a between-subjects ANOVA design, it was found that raters were sensitive to accuracy, but not range or sophistication, when rating essays for lexis. The implications for rater training and using rating scales are discussed. (Contains 4 tables.)