This article examines how the use of automated scoring procedures for short-answer reading tasks can affect the constructs being assessed. In particular, it highlights ways in which the development of scoring algorithms intended to apply the criteria used by human raters can lead test developers to reexamine and even refine the constructs they wish to assess. The article also points out examples of ways in which attempting to replicate human rating behavior while avoiding incidental construct alteration can pinpoint areas of the automated scoring process requiring further development. The examples discussed in this article illustrate the central point that construct definitions should always guide the development of scoring algorithms while the process of developing and refining such algorithms requires more rigorous construct definitions and can potentially push us to refine our constructs. (Contains 1 figure and 3 footnotes.)