ERIC Number: ED545460
Record Type: Non-Journal
Publication Date: 2013-Apr
Abstractor: As Provided
Reference Count: 17
An Application of Reverse Engineering to Automatic Item Generation: A Proof of Concept Using Automatically Generated Figures
Lorié, William A.
A reverse engineering approach to automatic item generation (AIG) was applied to a figure-based publicly released test item from the Organisation for Economic Cooperation and Development (OECD) Programme for International Student Assessment (PISA) mathematical literacy cognitive instrument as part of a proof of concept. The author created an item template from which three items were randomly generated from within each of six types defined by a feature deemed to be most likely to affect item difficulty, for a total of eighteen distinct items. To assess their equivalence, these items were embedded in otherwise identical test forms and administered to human intelligence task workers on the Amazon Mechanical Turk system. One level of the type-defining feature appeared to affect item difficulty systematically. The author provides a task requirement rationale for removing this level. Implications for AIG theory and practice are discussed. An appendix presents a sample HTT.
Descriptors: Numeracy, Mathematical Concepts, Mathematical Logic, Difficulty Level, Test Items, Task Analysis, Engineering Technology, Human Factors Engineering, Item Sampling, Cognitive Measurement, Equated Scores, Construct Validity, Intelligence, Mathematical Applications, Mathematical Models, Statistical Analysis
Publication Type: Reports - Research
Education Level: N/A
Authoring Institution: N/A
Identifiers - Assessments and Surveys: Program for International Student Assessment