This article reports the development of the Test Takers' Metacognitive Awareness Reading Questionnaire (TMARQ) which measures test takers' metacognition in reading comprehension tests. The TMARQ comprises seven subscales: planning strategies, evaluating strategies, monitoring strategies, strategies for identifying important information, inference-making strategies, integrating strategies, and supporting strategies. In this article, a validity argument is laid out for the questionnaire by presenting content-referenced, substantive, and structural evidence of validity, which is primarily yielded through Rasch measurement and structural equation modeling.