NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ1153731
Record Type: Journal
Publication Date: 2017
Pages: 17
Abstractor: As Provided
ISSN: ISSN-0895-7347
A Multilevel Factor Analysis of Third-Party Evaluations of Noncognitive Constructs Used in Admissions Decision Making
Oliveri, Maria; McCaffrey, Daniel; Ezzo, Chelsea; Holtzman, Steven
Applied Measurement in Education, v30 n4 p297-313 2017
The assessment of noncognitive traits is challenging due to possible response biases, "subjectivity" and "faking." Standardized third-party evaluations where an external evaluator rates an applicant on their strengths and weaknesses on various noncognitive traits are a promising alternative. However, accurate score-based inferences from third-party evaluations requires disentangling score variance due to raters versus applicants by utilizing a multilevel factor analysis (MFA). To date, MFA is highly underutilized in the measurement field. In this study, we apply the MFA to analyze third-party evaluations using data from the Personal Potential Index (PPI). The PPI is a third-party measure used to evaluate graduate school applicants noncognitive traits to help inform admissions decisions. We analyzed 12,693 ratings of 6,249 applicants divided into two randomly selected subgroups. We conducted multilevel exploratory factor analysis with one subgroup and tested the hypothesized structure with the other subgroup. This work illustrates the advantages and challenges of using MFA approach to support the meaningful and valid interpretation of scores from third-party evaluations.
Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A