ERIC Number: EJ1159255
Record Type: Journal
Publication Date: 2017-Dec
Abstractor: As Provided
On the Validity of Repeated Assessments in the UMAT, a High-Stakes Admissions Test
Andrich, David; Styles, Irene; Mercer, Annette; Puddey, Ian B.
Advances in Health Sciences Education, v22 n5 p1245-1262 Dec 2017
The possibility that the validity of assessment is compromised by repeated sittings of highly competitive and high profile selection tests has been documented and is of concern to stake-holders. An illustrative example is the Undergraduate Medicine and Health Sciences Admission Test (UMAT) used by some medical and dental courses in Australia and New Zealand. The proficiencies of all applicants who sat the UMAT from one to four sittings between 2006 and 2012 were estimated on the same metric using the probabilistic Rasch model. A fit index characterising each profile's degree of conformity to the model was also calculated. Confirming expectations, mean proficiencies increased with repeated sittings on all three UMAT scales with the greatest difference (which was nevertheless relatively small) between the first two sittings. The fit index showed that the increases in proficiency estimates arose from additional easier items being answered correctly on repeated sittings rather than additional more difficult ones, suggesting that improvements are not on the substantive construct of the variable of assessment but in skills in answering the questions. Although strategies for dealing with the increase in proficiency estimates on repeated sittings could be canvassed, these results suggest that the validity of results on repeated sittings was not compromised. Accordingly, it might be concluded that although particular individuals might improve substantially between sittings, any validity is not likely to be compromised with the possibility that for some applicants, the second sitting might be the most valid.
Descriptors: High Stakes Tests, College Entrance Examinations, Undergraduate Study, Selection Criteria, Medical Education, Health Sciences, Item Response Theory, Probability, Goodness of Fit, Test Validity, Foreign Countries
Springer. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: firstname.lastname@example.org; Web site: http://www.springerlink.com
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education
Authoring Institution: N/A
Identifiers - Location: Australia; New Zealand
Grant or Contract Numbers: N/A