This paper compares the performance of students at a high-performing U.S. public school (n = 64) on the advanced placement (AP) chemistry exam to their performance on the ChemQuery assessment system. The AP chemistry exam was chosen because, as the National Research Council acknowledges, it is the "perceived standard of excellence and school quality". In contrast to the nationally recognized AP chemistry exam, the ChemQuery assessment system is a research tool that uses item-response theory to map student progress on a scale of conceptual understanding in chemistry. Our findings indicate that the two types of assessments, traditional problem-solving skills and conceptual understanding, are highly correlated as measured here. However, student performance on the ChemQuery assessment is disappointingly low. On the basis of the data analysis, this paper discusses the implications of the findings with a focus on the current efforts to redesign the AP chemistry exam.