ERIC Number: ED498071
Record Type: Non-Journal
Publication Date: 2003-May-9
The Effectiveness of Educational Technology: Issues and Recommendations for the National Study
Agodini, Roberto; Dynarski, Mark; Honey, Margaret; Levin, Douglas
Mathematica Policy Research, Inc.
The No Child Left Behind Act (P.L. 107-110) called for the U.S. Department of Education to carry out a national study of the effectiveness of educational technology. With computers becoming ubiquitous in American schools, and purchases of hardware and software now substantial expenses for school districts, whether funding is supporting effective uses of technology and whether spending can be more effective have become concerns. The legislation's mandate called for the study to use rigorous methods to provide evidence of effectiveness. In October 2002, the U.S. Department of Education began working with Mathematica Policy Research, Inc. and its partners, the American Institutes for Research and the Education Development Center, to identify issues confronting a national study of technology effectiveness and to develop designs for the study. A key part of the design effort was to engage a panel of outside experts on educational technology, educational policy, and research methodology, to help identify important questions to be addressed in the study and to suggest possible approaches for answering them. The design team worked with the advisory panel and with ED staff to arrive at nine recommendations for how the national study could focus its attention (see box, next page). The panel played an important role in suggesting issues and approaches, and in discussing the strengths and weaknesses of various approaches, but did not formally make recommendations. The key broad question to be addressed by the evaluation is "Is educational technology effective in improving student academic achievement?" The design team recognized that, stated in this way, no single study could answer the question. In effect, many questions are implied, related to alternative definitions of education technology, effectiveness, and improving student achievement. The team needed to define what is meant by "educational technology," "effective," and so on. The design team's recommendations refine the study, so it can have the potential to contribute substantially to what is known about the effectiveness of educational technology. The recommendations focus attention on technology applications that support instruction in reading or math in low-income schools serving the K-12 grade levels. The study would use experimental designs (with random assignment of students, classrooms, or schools, depending on the type of technology application) to ensure that measured effects can be attributed to the technology applications. The key outcome would be scores from a commonly used standardized test, supported by other academic outcomes collected from extant data. The report provides rationales for the recommendations and discusses conceptual frameworks and statistical issues related to measuring effects and determining sample sizes. Appended are: (1) Technical Working Group Members; and (2) Estimates of Intra-Cluster Correlation Coefficients for Schools and Classrooms. (Contains 6 figures and 5 tables.)
Descriptors: Teaching Methods, Program Effectiveness, Federal Legislation, Research Methodology, Standardized Tests, Correlation, Educational Technology, Educational Policy, Research Design, Academic Achievement, Elementary Secondary Education, Disadvantaged Schools, Control Groups, Effect Size, Sample Size, Scores
Mathematica Policy Research, Inc. P.O. Box 2393, Princeton, NJ 08543-2393. Tel: 609-799-3535; Fax: 609-799-0005; e-mail: firstname.lastname@example.org; Web site: http://www.mathematica-mpr.com/publications/
Publication Type: Reports - Descriptive
Education Level: Elementary Secondary Education
Authoring Institution: Mathematica Policy Research, Princeton, NJ.; Institute of Education Sciences (ED), Washington, DC.
Identifiers - Laws, Policies, & Programs: No Child Left Behind Act 2001
IES Cited: ED511782; ED503448