NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: EJ1149168
Record Type: Journal
Publication Date: 2017
Pages: 22
Abstractor: As Provided
ISSN: EISSN-1929-7750
How Flexible Is Your Data? A Comparative Analysis of Scoring Methodologies across Learning Platforms in the Context of Group Differentiation
Ostrow, Korinn S.; Wang, Yan; Heffernan, Neil T.
Journal of Learning Analytics, v4 n2 p91-112 2017
Data is flexible in that it is molded by not only the features and variables available to a researcher for analysis and interpretation, but also by how those features and variables are recorded and processed prior to evaluation. "Big Data" from online learning platforms and intelligent tutoring systems is no different. The work presented herein questions the quality and flexibility of data from two popular learning platforms, comparing binary measures of problem-level accuracy, the scoring method typically used to inform learner analytics, with partial credit scoring, a more robust, real-world methodology. This work extends previous research by examining how the manipulation of scoring methodology has the potential to alter outcomes when testing hypotheses, or specifically, when looking for significant differences between groups of students. Datasets from ASSISTments and Cognitive Tutor are used to assess the implications of data availability and manipulation within twelve mathematics skills. A resampling approach is used to determine the size of equivalent samples of high- and low-performing students required to reliably differentiate performance when considering each scoring methodology. Results suggest that in eleven out of twelve observed skills, partial credit offers more efficient group differentiation, increasing analytic power and reducing Type II error. Alternative applications of this approach and implications for the Learning Analytics community are discussed. [Federal funding was received from the Graduate Assistance in Areas of National Need (GAANN) program, Student Service, Higher Education Programs, Office of Postsecondary Education.]
Society for Learning Analytics Research. 121 Pointe Marsan, Beaumont, AB T4X 0A2, Canada. Tel: +61-429-920-838; e-mail:; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED); National Science Foundation (NSF); Office of Naval Research (ONR); Office of Postsecondary Education (ED), Higher Education Programs
Authoring Institution: N/A
IES Funded: Yes
Grant or Contract Numbers: R305A120125; R305C100024; ACI1440753; DRL1252297; DRL1109483; DRL1316736; DRL1031398