NotesFAQContact Us
Search Tips
ERIC Number: ED546575
Record Type: Non-Journal
Publication Date: 2012
Pages: 105
Abstractor: As Provided
ISBN: 978-1-2675-9271-2
Using Hierarchical Rater Model to Adjust Effect Size
Luo, Long
ProQuest LLC, Ph.D. Dissertation, Southern Methodist University
The topic of this research is inference for effect size of a curriculum intervention, which is an important research topic in education. The linear relationship between the outcomes of an intervention and teachers' fidelity, the extent to which the intervention was actually delivered by teachers as intended, is an important component for effect size. In many intervention studies, the assessment of teachers' fidelity is based on the ordinal scores assigned by observers to a subset of lessons of a teacher based on a certain coding rubric. Teachers' fidelity is not perfectly measured in this way, because of the effects of coding rubrics, observer's imperfect evaluation skills and the sampling errors. The imperfect measurements within fidelity, if ignored, will bias the estimation of a linear model and hence the inference for effect size. Many researchers in education, however, ignore the existence of the imperfect measurements and treat ordinal scores as interval values. They use the average score of the lessons of a teacher as an estimate for this teacher's fidelity and then use the estimates for all the teachers in an intervention study as the values of an explanatory variable of a linear model. In this research, this approach is called the traditional method. In this research, a hierarchical rater model is proposed to estimate teachers' fidelity and then an effective experimental design for intervention studies is recommended based on the proposed model. A new calibration method for the explanatory variables of a linear model is then proposed based on the rater model; this approach is called the rater model based calibration (RMC) method, which uses the estimates for teachers' fidelity by the rater model to calibrate an explanatory variable of a linear model. A series of simulation studies are conducted to compare the RMC method and the traditional method in estimating the effect size of an interventional study. Then some suggestions on the use of two methods based on the simulation studies are made when researchers have a set of ordinal scores from an experiment. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page:]
ProQuest LLC. 789 East Eisenhower Parkway, P.O. Box 1346, Ann Arbor, MI 48106. Tel: 800-521-0600; Web site:
Publication Type: Dissertations/Theses - Doctoral Dissertations
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A