Rate and comprehension are two components related to broad reading abilities (i.e., phonemic awareness, decoding, vocabulary, comprehension). The purpose of this study was to examine the unique contribution of three curriculum-based measures (CBM)-comprehension assessments compared to a CBM-oral reading rate assessment through diagnostic consistency and incremental validity analyses. An extant sample of fall screening data in a national assessment system were used for this investigation. The CBM-comprehension assessments measured oral recall, synthesis of main ideas, and free-response question-answering. Scores from the CBM-comprehension measures were associated with weak criterion-related validity to a measure of broad reading. In addition, diagnostic consistency analyses revealed poor overlap between CBM-comprehension and CBM-oral reading rate assessment score classifications based on at-risk benchmarks. Incremental validity evidence replicated previous findings, demonstrating that the CBM-comprehension measures explain unique variation in broad reading scores even after controlling for rate and accuracy. Implications for the usage of CBM-R and CBM-comprehension for screening is addressed.