The effectiveness of individualized feedback on rater behavior has been investigated in several previous studies (e.g. Elder, Knoch, Barkhuizen, & von Randow, 2005; Lunt, Morton, & Wigglesworth, 1994; O'Sullivan & Rignall, 2007; Wigglesworth, 1993). The findings of these studies are somewhat inconclusive. However, all of these studies investigated the value of such feedback on a one-off basis, and it is thus not clear how raters use this type of feedback over several administrations of a test. Furthermore, previous research has focused only on one language skill at a time, and we do not know if raters can incorporate the feedback in a similar way when rating speaking and writing. This study tracks the rating behavior of 19 raters assessing a large-scale English for specific purposes (ESP) assessment for the health professions over eight administrations. After each administration, raters received detailed performance profiles of their rating behavior which were generated using many-faceted Rasch measurement. Raters also completed a questionnaire and a subset of raters were interviewed to ascertain their views on the effectiveness of this feedback. The findings showed that the raters were rating no better when receiving the feedback than when they had not received feedback and neither speaking nor writing raters were able to incorporate the feedback more successfully. Although the raters were generally positive about the success of the feedback, there was no relationship between their perceptions of the feedback and the success of the feedback. (Contains 6 figures, 12 tables and 1 note.)