NotesFAQContact Us
Search Tips
ERIC Number: EJ1126255
Record Type: Journal
Publication Date: 2017
Pages: 7
Abstractor: As Provided
ISSN: ISSN-0013-1962
Human Expert Labeling Process (HELP): Towards a Reliable Higher-Order User State Labeling Process and Tool to Assess Student Engagement
Aslan, Sinem; Mete, Sinem Emine; Okur, Eda; Oktay, Ece; Alyuz, Nese; Genc, Utku Ergin; Stanhill, David; Esme, Asli Arslan
Educational Technology, v57 n1 p53-59 Jan-Feb 2017
In a series of longitudinal research studies, researchers at Intel Corporation in Turkey have been working towards an adaptive learning system automatically detecting student engagement as a higher-order user state in real-time. The labeled data necessary for supervised learning can be obtained through labeling conducted by human experts. Using multiple labelers to label collected data and obtaining agreement among different labelers on the same samples of data, it is critical to train all to use the engagement model accurately. Addressing these challenges, the researchers developed a rigorous human expert labeling process (HELP) specific to the educational context, with multi-faceted labels and multiple expert labelers. HELP has three distinct stages: (1) "Pre-Labeling", including planning, labeler recruitment, training, and evaluation steps; (2) "Labeling", involving actual labeling conducted by multiple labelers, and related steps for formative assessment of their performance; and (3) "Post-Labeling", generating final labels and agreement measures through processing multiple decisions. In this article, the researchers outline proposed methods in HELP and describe the developed labeling tool.
Educational Technology Publications. 700 Palisade Avenue, Englewood Cliffs, NJ 07632-0564. Tel: 800-952-2665; Web site:
Publication Type: Journal Articles; Reports - Descriptive
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: Turkey
Grant or Contract Numbers: N/A