ERIC Number: ED474275
Record Type: RIE
Publication Date: 2001-Apr
How To Create Complex Measurement Models: A Case Study of Principled Assessment Design.
Bauer, Malcolm; Williamson, David M.; Steinberg, Linda S.; Mislevy, Robert J.; Behrens, John T.
In computer-based simulations, students must bring a wide range of relevant knowledge, skills, and abilities to bear jointly as they solve meaningful problems in a learning domain. To function effectively as an assessment, a simulation system must additionally be able to evoke and interpret observable evidence about targeted knowledge in a manner that is principled, defensible, and suited to the purpose at hand (e.g., licensure, achievement testing, coached practice). This study focused on the grounding for a simulation-based assessment of design and troubleshooting in the domain of computer networks. The application was designed as a prototype for assessing these skills in an instructional program, as interim practice tests and as chapter or end-of-course assessments. An evidence-centered assessment design framework was used to guide the work. An important part of this work is a cognitive task analysis, designed to tap the knowledge network engineers and students use when they design and troubleshoot networks and elicit behaviors that manifest this knowledge. There were 24 participants at 3 levels of ability in this early phase of the research. After summarizing the results of the analysis, the paper discusses implications for designing psychometric models, automated scoring algorithms, and task frameworks, and for the capabilities required for the simulation environment itself. (Contains 6 figures and 24 references.) (Author/SLD)
Publication Type: Reports - Research
Education Level: N/A
Authoring Institution: N/A