Direct behavior rating (DBR) offers users a flexible, feasible method for the collection of behavioral data. Previous research has supported the validity of using DBR to rate three target behaviors: academic engagement, disruptive behavior, and compliance. However, the effect of the base rate of behavior on rater accuracy has not been established. In addition, although teachers report frequent use of DBR-like tools, little is known about what type of training is necessary to ensure accurate usage. The present study sought to determine (a) whether training with practice and feedback improves rater accuracy when rating academic engagement, disruptive behavior, and compliance and (b) whether rater accuracy varies at low, medium, and high levels of these behaviors. Participants ("N" = 67) were randomly assigned to one of three training conditions: brief familiarization, brief training with practice and feedback, or extensive training with practice and feedback. Following training, participants watched videos of elementary students and rated behavior with DBRs. Results indicated that practice with feedback improved rater accuracy for disruptive behavior but not for the behaviors of academic engagement and compliance. In addition, rater accuracy was found to vary significantly at different levels of behavior. Academic engagement was rated most accurately when high rates of the behavior were displayed. Disruptive behavior and compliance were rated most accurately at low or high base rates, while raters had more difficulty rating medium levels of behaviors. Limitations, implications, and future directions for research are discussed.