This mixed-methods study reports psychometric properties of the 34-item Reasoning about P-values and Statistical Significance (RPASS) scale. RPASS is being designed as a research tool to assess effects of teaching methods on students' inferential reasoning. During development (Phase I), two graphical scenarios and 12 items were added to the scale, field tested, and evaluated by three content raters. During Phase II, reliability and validity evidence were gathered in three college statistics courses. Score reliability was sufficient to conduct group research (alpha = 0.76, n = 105). RPASS scores were correlated with college entrance scores and GPAs as evidence of construct-related validity. Further validity evidence was obtained by analyzing consistency between students' reasoning and answers for eight items. Future development and research are discussed. (Contains 10 tables and 2 figures.)