ERIC Number: EJ1049507
Record Type: Journal
Publication Date: 2014-Nov
Abstractor: As Provided
Reference Count: 48
Exploring a Three-Level Model of Calibration Accuracy
Schraw, Gregory; Kuch, Fred; Gutierrez, Antonio P.; Richmond, Aaron S.
Journal of Educational Psychology, v106 n4 p1192-1202 Nov 2014
We compared 5 different statistics (i.e., G index, gamma, "d'", sensitivity, specificity) used in the social sciences and medical diagnosis literatures to assess calibration accuracy in order to examine the relationship among them and to explore whether one statistic provided a best fitting general measure of accuracy. College undergraduates completed separate 15-item vocabulary, probability, and paper folding tests by answering a test item and indicating whether or not the item was answered correctly. We computed scores for each of the 5 calibration statistics using the same raw scores for each test and compared 3 theoretical models, including 1-, 2-, and 3-factor confirmatory factor analysis solutions. Results supported the 3-factor model over the 1-factor and 2-factor models with respect to goodness-of-fit indices and least number of estimated parameters. The 3-factor solution was consistent with the hypothesis that the 5 individual calibration scores are related to 2 different types of 2nd-order processes (i.e., accuracy of judgments about correct and incorrect performance), as measured by sensitivity and specificity that are subsumed under a general 3rd-order discrimination process as measured by d'. Implications for a theory of calibration accuracy and measurement practice were discussed.
Descriptors: Statistics, Statistical Analysis, Correlation, Accuracy, Undergraduate Students, Vocabulary, Probability, Manipulative Materials, Scores, Raw Scores, Models, Comparative Analysis, Factor Analysis, Goodness of Fit, Hypothesis Testing, Metacognition, Knowledge Level, Computation
American Psychological Association. Journals Department, 750 First Street NE, Washington, DC 20002. Tel: 800-374-2721; Tel: 202-336-5510; Fax: 202-336-5502; e-mail: firstname.lastname@example.org; Web site: http://www.apa.org
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Authoring Institution: N/A