In this study, we developed a Reading for Meaning special education teacher observation rubric that details the elements of evidence-based comprehension instruction and tested its psychometric properties using many-faceted Rasch measurement (MFRM). Video observations of classroom instruction from 10 special education teachers across three states during the 2015-16 school year were collected. External raters (n=4) were trained to observe and evaluate instruction using the rubric, and assign scores of 'implemented', 'partially implemented' or 'not implemented' for each of the items. Analyses showed that the item, teacher, lesson and rater facets achieved high psychometric quality for the instrument. Teacher performance was consistent with what has been reported in the literature. Implications for research and practice are discussed. [This paper was published in "Reading & Writing Quarterly" 2019 (EJ1216107).]