ERIC Number: ED477927
Record Type: Non-Journal
Publication Date: 2003-Apr
Reference Count: N/A
Creating a Complex Measurement Model Using Evidence Centered Design.
Williamson, David M.; Bauer, Malcom; Steinberg, Linda S.; Mislevy, Robert J.; Behrens, John T.
In computer-based simulations meant to support learning, students must bring a wide range of relevant knowledge, skills, and abilities to bear jointly as they solve meaningful problems in a learning domain. To function efficiently as an assessment, a simulation system must also be able to evoke and interpret observable evidence about targeted knowledge in a manner that is principled, defensible, and suited to the purpose at hand (e.g., licensure, achievement testing, coached practice). This paper concerns the grounding for the design of a simulation-based assessment of design and troubleshooting in the domain of computer networking. The application is a prototype for assessing these skills as part of an instructional program, as interim practice tests, and as chapter or end-of-course assessments. An evidence-centered assessment design (ECD) framework was used to guide the work. A total of 24 students in a computer networking class (high school and community college students) participated. An important part of this work was a cognitive task analysis designed to: (1) tap the knowledge computer network specialists and students use when they design and troubleshoot networks; and (2) elicit behaviors that manifest this knowledge. After summarizing the results, the paper discusses implications of this analysis, as well as information gathered from other methods of domain analysis, for designing psychometric models, automated scoring algorithms, and task frameworks, and for the capabilities required for the delivery of this example of a complex simulation-based assessment. (Contains 2 figures, 2 tables, and 27 references.) (Author/SLD)
Publication Type: Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Authoring Institution: N/A