NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ962667
Record Type: Journal
Publication Date: 2011-Dec
Pages: 15
Abstractor: As Provided
Reference Count: 53
ISSN: ISSN-1931-7913
Applying Computerized-Scoring Models of Written Biological Explanations across Courses and Colleges: Prospects and Limitations
Ha, Minsu; Nehm, Ross H.; Urban-Lurain, Mark; Merrill, John E.
CBE - Life Sciences Education, v10 n4 p379-393 Dec 2011
Our study explored the prospects and limitations of using machine-learning software to score introductory biology students' written explanations of evolutionary change. We investigated three research questions: 1) Do scoring models built using student responses at one university function effectively at another university? 2) How many human-scored student responses are needed to build scoring models suitable for cross-institutional application? 3) What factors limit computer-scoring efficacy, and how can these factors be mitigated? To answer these questions, two biology experts scored a corpus of 2556 short-answer explanations (from biology majors and nonmajors) at two universities for the presence or absence of five key concepts of evolution. Human- and computer-generated scores were compared using kappa agreement statistics. We found that machine-learning software was capable in most cases of accurately evaluating the degree of scientific sophistication in undergraduate majors' and nonmajors' written explanations of evolutionary change. In cases in which the software did not perform at the benchmark of "near-perfect" agreement (kappa greater than 0.80), we located the causes of poor performance and identified a series of strategies for their mitigation. Machine-learning software holds promise as an assessment tool for use in undergraduate biology education, but like most assessment tools, it is also characterized by limitations. (Contains 4 tables and 4 figures.)
American Society for Cell Biology. 8120 Woodmont Avenue Suite 750, Bethesda, MD 20814-2762. Tel: 301-347-9300; Fax: 301-347-9310; e-mail:; Website:
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: Ohio