This study examined the use of a college course evaluation instrument in an effort to better understand and improve the assessment data derived from the instrument. The purpose of the analysis was to examine the dimensionality and reliability of the instrument but, more importantly, to understand what the data really tells us and whether assessment literacy would improve data usability. Analysis of the results suggests that the instrument tended to produce internally consistent data. However, the instrument measured predominately one aspect of course quality. In addition, based on results of the assessment literacy exercise, respondents seem to use very different criterion for rating; the constructs being measured generally did not match what was intended; and the questions being asked did not always match the scale being used. As a result the usefulness of any data interpretation and subsequent decisions was deemed suspect. The results of this study also suggest that simple assessment literacy interventions by themselves do not seem to drastically change the ability of raters to score items reliably. A much more comprehensive effort would be needed to produce results that would be beneficial for long term, data-driven decision making. (Contains 2 tables.)