ERIC Number: EJ1166127
Record Type: Journal
Publication Date: 2017
Pages: 27
Abstractor: As Provided
ISBN: N/A
ISSN: EISSN-2157-2100
EISSN: N/A
WordBytes: Exploring an Intermediate Constraint Format for Rapid Classification of Student Answers on Constructed Response Assessments
Kim, Kerry J.; Meir, Eli; Pope, Denise S.; Wendel, Daniel
Journal of Educational Data Mining, v9 n2 p45-71 2017
Computerized classification of student answers offers the possibility of instant feedback and improved learning. Open response (OR) questions provide greater insight into student thinking and understanding than more constrained multiple choice (MC) questions, but development of automated classifiers is more difficult, often requiring training a machine learning system with many human-classified answers. Here we explore a novel intermediate constraint question format called WordBytes (WB) where students assemble one-sentence answers to two different college evolutionary biology questions by choosing, then ordering, fixed tiles containing words and phrases. We found WB allowed students to construct hundreds to thousands of different answers (= 20 tiles), with multiple ways to express correct and incorrect answers with different misconceptions. We found humans could specify rules for an automated WB grader that could accurately classify answers as correct/incorrect with Cohen's kappa = 0.88, near the measured intra-rater reliability of two human graders and the performance of machine classification of OR answers (Nehm et al., 2012). Finer-grained classification to identify the specific misconception had lower accuracy (Cohen's kappa < 0.75), which could be improved either by using a machine learner or revising the rules, but both would require considerably more development effort. Our results indicate that WB may allow rapid development of automated correct/incorrect answer classification without collecting and hand-grading hundreds of student answers.
Descriptors: Classification, Computer Assisted Testing, Multiple Choice Tests, Test Format, College Students, Evolution, Biology, Science Tests, Phrase Structure, Misconceptions, Accuracy, Interrater Reliability, Computer Software
International Educational Data Mining. e-mail: jedm.editor@gmail.com; Web site: http://jedm.educationaldatamining.org/index.php/JEDM
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: National Science Foundation (NSF)
Authoring Institution: N/A
Grant or Contract Numbers: 1227245