NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: EJ1187513
Record Type: Journal
Publication Date: 2018
Pages: 19
Abstractor: As Provided
ISSN: EISSN-1929-7750
Metrics for Discrete Student Models: Chance Levels, Comparisons, and Use Cases
Bosch, Nigel; Paquette, Luc
Journal of Learning Analytics, v5 n2 p86-104 2018
Metrics including Cohen's kappa, precision, recall, and F[subscript 1] are common measures of performance for models of discrete student states, such as a student's affect or behaviour. This study examined discrete model metrics for previously published student model examples to identify situations where metrics provided differing perspectives on model performance. Simulated models also systematically showed the effects of imbalanced class distributions in both data and predictions, in terms of the values of metrics and the chance levels (values obtained by making random predictions) for those metrics. Random chance level for F[subscript 1] was also established and evaluated. Results for example student models showed that over-prediction of the class of interest (positive class) was relatively common. Chance-level F[subscript 1] was inflated by over-prediction; conversely, maximum possible values for F[subscript 1] and kappa were negatively impacted by over-prediction of the positive class. Additionally, normalization methods for F[subscript 1] relative to chance are discussed and compared to kappa, demonstrating an equivalence between kappa and normalized F[subscript 1]. Finally, implications of results for choice of metrics are discussed in the context of common student modelling goals, such as avoiding false negatives for student states that are negatively related to learning.
Society for Learning Analytics Research. 121 Pointe Marsan, Beaumont, AB T4X 0A2, Canada. Tel: +61-429-920-838; e-mail:; Web site:
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A