ERIC Number: ED528882
Record Type: Non-Journal
Publication Date: 2011
Rigorous Measures of Implementation: A Methodological Framework for Evaluating Innovative STEM Programs
Cassata-Widera, Amy; Century, Jeanne; Kim, Dae Y.
Society for Research on Educational Effectiveness
The practical need for multidimensional measures of fidelity of implementation (FOI) of reform-based science, technology, engineering, and mathematics (STEM) instructional materials, combined with a theoretical need in the field for a shared conceptual framework that could support accumulating knowledge on specific enacted program elements across evaluations of reform-based STEM innovations, has informed CEMSE's approach to fidelity of implementation measurement. Beginning in 2007, CEMSE received support from the National Science Foundation for its "Applied Research on Science Materials Implementation: Bringing Measurement of Fidelity of Implementation to Scale" project (Award ESI-0628052). Through this project, the CEMSE team developed, pilot tested, field tested, and revised a suite of eight instruments for measuring the fidelity of implementation (FOI) of reform-based K-8 science and mathematics instructional materials programs. These instruments, which provide a variety of data collection approaches, are unique among those that measure enactment of programs in that they focus on clearly and specifically describing the nature of program implementation, using constructs representing the essential elements of reform-based mathematics and science instructional materials programs organized into a conceptual framework that facilitates their application across multiple programs. By explicitly measuring the different elements of educational interventions organized according to this framework (the "FOI Framework"), the authors' approach provides an improved way to understand and measure program implementation that reflects the complex, multidimensional reality of interventions enacted in classroom settings. Measuring the presence and level of enactment of structural and interactional critical components enables them to distinguish between the quality of the treatment as well as the degree of the treatment. In addition to providing a more detailed and nuanced method for measuring enactment of multiple aspects of interventions, using a critical components approach to conceptualize program implementation also addresses the existing problems in the field related to accounting for fidelity of implementation within experimental studies. The tools provide a way to open up the "black box" of what happens in the treatment group to explore more nuanced relationships between interventions and outcomes. Appended are: (1) References; and (2) Tables and Figures. (Contains 1 footnote.
Descriptors: Program Implementation, Science Materials, Instructional Materials, STEM Education, Measurement, Tests, Elementary Secondary Education, Kindergarten, Grade 8, Data Collection, Models, Evaluation Methods
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail: firstname.lastname@example.org; Web site: http://www.sree.org
Publication Type: Reports - Evaluative
Education Level: Elementary Education; Elementary Secondary Education; Grade 1; Grade 2; Grade 3; Grade 4; Grade 5; Grade 6; Grade 7; Grade 8
Authoring Institution: Society for Research on Educational Effectiveness (SREE)