NotesFAQContact Us
Search Tips
ERIC Number: ED531723
Record Type: Non-Journal
Publication Date: 2011-May-24
Pages: 51
Abstractor: As Provided
Reference Count: 16
What Do We Measure? Methodological versus Institutional Validity in Student Surveys
Johnson, Jeffrey Alan
Association for Institutional Research (NJ1), Paper presented at the Annual Forum of the Association for Institutional Research (51st, Toronto, Ontario, May 21-25, 2011)
This paper examines the tension in the process of designing student surveys between the methodological requirements of good survey design and the institutional needs for survey data. Building on the commonly used argumentative approach to construct validity, I build an interpretive argument for student opinion surveys that allows assessment of the inferences and assumptions underlying the claim that an opinion survey item is a valid measure of a construct. I then evaluate the content validity of measures of program satisfaction and academic growth in Utah Valley University's 2010 Graduated Alumni Survey and 2009 Graduating Student Survey, surveys designed to maximize conformity with existing and legacy institutional priorities and demands. Using analytical assessment and empirical tests including response change across surveys, inter-item correlation, and factor analysis I show that UVU's surveys--and by implication, all surveys developed with primarily institutional validity in mind--are subject to grave challenges to the construct validity. I conclude by suggesting that effective operationalization of institutional priorities can bring together construct and institutional validity. AIR Forum Powerpoint Presentation is appended. (Contains 9 tables and 4 footnotes.)
Association for Institutional Research. 1435 East Piedmont Drive Suite 211, Tallahassee, FL 32308. Tel: 850-385-4155; Fax: 850-383-5180; e-mail:; Web site:
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: Utah