The disparity between the results of state accountability tests in reading and the fourth-grade National Assessment of Educational Progress (NAEP) has left many educators dismayed and concerned. In this article the authors describe a comparison of assessment approaches evident in NAEP to the approaches taken in a sample of state tests. The authors found sizeable differences in the use of open-ended items, in the proportion of items that demand higher levels of thinking, and in the use of distractors that undercut the type of thinking that many items were designed to demand of the reader. The results of this study suggest that not all tests of reading are created equal and that NAEP is likely to be able to distinguish among advanced, proficient, and basic readers far more effectively than the state tests in their sample. (Contains 4 tables and 1 figure.)