The Research Methods Skills Assessment (RMSA) was created to measure psychology majors' statistics knowledge and skills. The American Psychological Association's Guidelines for the Undergraduate Major in Psychology (APA, 2007, 2013) served as a framework for development. Results from a Rasch analysis with data from n = 330 undergraduates showed good fit for 16 of the 21 items created. Validity analysis showed that using the RMSA to measure students' statistics knowledge was an improvement over the assessment recommended by the APA Guidelines and was less time consuming. Together the findings provide a preliminary test that can be further developed to provide a tool for instructors and departments to use when assessing psychology students' statistics knowledge and skills. Future research should expand the RMSA to include items that measure knowledge with confidence intervals as the proposed items were removed due to poor fit. Future studies should also aim to replicate the current findings with additional samples of psychology students.