ERIC Number: ED401305
Record Type: Non-Journal
Publication Date: 1996-Apr
Reference Count: N/A
Weighting Tasks from Job Analysis Data To Derive Test Specifications for Licensure Examinations: Some Methodological and Statistical Considerations.
Thomas, Leslie; Kalohn, John C.
Test specifications dictate the kind of content that should be included on each form of an examination, and the relative weight that each content domain should contribute to the determination of examinees' test scores by specifying the proportion of items to be included in each content area. This paper addresses a step in the development of specifications: combining job analysis rating data to determine an index of importance for each task. The Kane procedure (Kane et al, 1989), a multiplicative model, was compared with a new procedure based on a modified version of the Mahalanobis Distance (MD) measure. A conceptual model was also proposed to evaluate the results when comparing such procedures. A small data set of 5 task statements and 10 respondents was used as an example of data to be analyzed by both procedures. The Kane weights produced orderings of task statements that did not meet the expected ordering based on the model, except when criticality was weighted by a factor of 10. The MD weights approached a limit as the criticality weight was increased. The conceptual model proposed gives a logical rank ordering of tasks, but does not provide a means to translate this ordinal information into reasonable task weights. An ideal conceptual model would include a rationale for evaluating a weighting scheme. Preliminary analyses of the actual job data with each approach demonstrated that both methods failed to produce weights that were compatible with the model presented. Research on a more comprehensive conceptual model is being planned. (Contains 5 figures, 12 tables, and 11 references.) (SLD)
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A