ERIC Number: EJ812872
Record Type: Journal
Publication Date: 2008-May
Abstractor: As Provided
Reference Count: N/A
mini-PAT (Peer Assessment Tool): A Valid Component of a National Assessment Programme in the UK?
Archer, Julian; Norcini, John; Southgate, Lesley; Heard, Shelley; Davies, Helena
Advances in Health Sciences Education, v13 n2 p181-192 May 2008
Purpose: To design, implement and evaluate a multisource feedback instrument to assess Foundation trainees across the UK. Methods: mini-PAT (Peer Assessment Tool) was modified from SPRAT (Sheffield Peer Review Assessment Tool), an established multisource feedback (360 [degree]) instrument to assess more senior doctors, as part of a blueprinting exercise of instruments suitable for assessment in Foundation programmes (first 2 years postgraduation). mini-PAT's content validity was assured by a mapping exercise against the Foundation Curriculum. Trainees' clinical performance was then assessed using 16 questions rated against a six-point scale on two occasions in the pilot period. Responses were analysed to determine internal structure, potential sources of bias and measurement characteristics. Results: Six hundred and ninety-three mini-PAT assessments were undertaken for 553 trainees across 12 Deaneries in England, Wales and Northern Ireland. Two hundred and nineteen trainees were F1s or PRHOs and 334 were F2s. Trainees identified 5544 assessors of whom 67% responded. The mean score for F2 trainees was 4.61 (SD = 0.43) and for F1s was 4.44 (SD = 0.56). An independent t test showed that the mean scores of these 2 groups were significantly different (t = -4.59, df 390, p less than 0.001). 43 F1s (19.6%) and 19 F2s (5.6%) were assessed as being below expectations for F2 completion. The factor analysis produced 2 main factors, one concerned clinical performance, the other humanistic qualities. Seventy-four percent of F2 trainees could have been assessed by as few as 8 assessors (95% CI [plus or minus] 0.6) as they either scored an overall mean of 4.4 or above or 3.6 and below. Fifty-three percent of F1 trainees could have been assessed by as few as 8 assessors (95% CI [plus or minus] 0.5) as they scored an overall mean of 4.5 or above or 3.5 and below. The hierarchical regression when controlling for the grade of trainee showed that bias related to the length of the working relationship, occupation of the assessor and the working environment explained 7% of the variation in mean scores when controlling for the year of the Foundation Programme (R squared change = 0.06, F change = 8.5, significant F change less than 0.001). Conclusions As part of an assessment programme, mini-PAT appears to provide a valid way of collating colleague opinions to help reliably assess Foundation trainees.
Descriptors: Feedback (Response), Evaluation Methods, Content Validity, Tests, Factor Analysis, Foreign Countries, Clinical Experience, Peer Evaluation, Physicians, Scores, Curriculum
Springer. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: email@example.com; Web site: http://www.springerlink.com
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education
Authoring Institution: N/A
Identifiers - Location: United Kingdom (England); United Kingdom (Northern Ireland); United Kingdom (Wales)