Although there are studies on the pragmatic assessment, to date no research has been done on native English raters' criteria for the assessment of EFL learners' pragmatic competence. Focusing on this topic, this study pursued two purposes. The first one was to find the criteria for rating the speech act of apology in L2 by native English teachers. The second was to investigate whether there was rater bias in native English teachers' rating of apology. To this end, 51 native English teachers rated six different pragmatic situations for an apology discourse completion task (DCT) which were accompanied by an L2 learner's response to each situation. Besides rating, the raters were asked to describe the way they rated the response to each DCT situation. The content analysis of raters' descriptions revealed five criteria they mostly applied in their rating: expression of apology, situation explanation, repair offer, promise for future, and politeness. FACETS was used to find the rater bias. Results indicated that raters showed different degrees of severity and leniency in their ratings, which contradicts the myth of native speakers being a benchmark in language assessment, including pragmatic assessment.