ERIC Number: ED410267
Record Type: Non-Journal
Publication Date: 1996-May
Reference Count: N/A
Sampling Variability and Generalizability of Work Keys Listening and Writing Scores. ACT Research Report Series 96-1.
The use of the Work Keys Listening and Writing Assessment, part of an assessment system of the generic employability skills of individuals, needs to be accompanied by systematic evaluation of its technical qualities. This study examined sampling variability and generalizability of Listening and Writing scores when multiple forms, raters, and prompts (tasks) were used. Different types of scoring, current level scores, and mean scores (unrounded and rounded) were also compared in terms of generalizability. One sample consisted of 167 examinees and the other, of 89 examinees. The results indicate that: (1) examinees' scores vary from one test form to another due to large task-sampling variability; (2) the rank orderings of prompt difficulty differ for the various examinees; (3) measurement errors are mainly introduced by task-sampling variability, and not by rater-sampling variability; (4) the Writing scores are more generalizable than the Listening scores; and (5) current level scores are less generalizable than mean scores (both rounded and unrounded). The use of six prompts and two raters in the Work Keys assessment leads to a greater generalizability, especially in Writing, than other performance assessments with smaller numbers of tasks or raters. In addition, these analyses also caused test developers to explore alternative level scores with higher reliability than the current Work Keys scores. (Contains 6 figures, 7 tables, and 12 references.) (Author/SLD)
Descriptors: Adults, Difficulty Level, Generalization, Job Skills, Listening Comprehension Tests, Performance Based Assessment, Sampling, Scores, Scoring, Tables (Data), Test Construction, Writing Tests
ACT Research Report Series, P.O. Box 168, Iowa City, IA 52243-0168.
Publication Type: Numerical/Quantitative Data; Reports - Research
Education Level: N/A
Authoring Institution: American Coll. Testing Program, Iowa City, IA.
Identifiers - Assessments and Surveys: Work Keys (ACT)