The category of coherence in rating scales has often been criticized for being vague. Typical descriptors might describe students' writing as having 'a clear progression of ideas' or "lacking logical sequencing." These descriptors inevitably require subjective interpretation on the side of the raters. A number of researchers (Connor & Farmer, 1990; Intaraprawat & Steffensen, 1995) have attempted to measure coherence more objectively. However, these efforts have not thus far been reflected in rating scale descriptors. For the purpose of this study, the results of an adaptation of topical structure analysis (Connor and Farmer, 1990; Schneider and Connor, 1990), which proved successful in distinguishing different degrees of coherence in 602 academic writing scripts was used to formulate a new rating scale. The study investigates whether such an empirically grounded scale can be used to assess coherence in students' writing more reliably and with greater discrimination than the more traditional measure. The validation process involves a multi-faceted Rasch analysis of scores derived from multiple ratings of 100 scripts using the old and new rating descriptors as well as a qualitative analysis of questionnaires canvassed from the raters. The findings are discussed in terms of their implications for rating scale development. (Contains 9 tables and 8 figures.)