The ability to purposefully access, reflect, and use evidence from educational research (Educational Research Literacy) is expected of future professionals in educational practice. Based on the presented conceptual framework, a test instrument was developed to assess the different competency aspects: Information Literacy, Statistical Literacy, and Evidence-based Reasoning. This paper addresses the question of whether this test allows monitoring students' development during courses on research methods in educational science. In winter semester 2013/14, two courses in Early Education and Teacher Training degree programs were accompanied. Both cross-sectional (multidimensional latent regression models) and longitudinal (paired t tests of plausible values) comparisons are reported. The results indicate no significant differences in competencies between the degree programs at start of the semester. For the end of semester, only data of the Early Education course was available. These course participants showed higher proficiency in all aspects of educational research literacy. In conclusion, the results are discussed with regard to the goals of the assessment of competencies in higher education, particularly the use of test feedback for course development, and desiderata for future research (e.g., ability to transfer research into practice) are stated.