ERIC Number: EJ1148519
Record Type: Journal
Publication Date: 2017-Sep
Abstractor: As Provided
Automated Assessment of the Quality of Peer Reviews Using Natural Language Processing Techniques
Ramachandran, Lakshmi; Gehringer, Edward F.; Yadav, Ravi K.
International Journal of Artificial Intelligence in Education, v27 n3 p534-581 Sep 2017
A "review" is textual feedback provided by a reviewer to the author of a submitted version. Peer reviews are used in academic publishing and in education to assess student work. While reviews are important to e-commerce sites like Amazon and e-bay, which use them to assess the quality of products and services, our work focuses on academic reviewing. We seek to help reviewers improve the quality of their reviews. One way to measure review quality is through "metareview" or review of reviews. We develop an automated metareview software that provides rapid feedback to reviewers on their assessment of authors' submissions. To measure review quality, we employ metrics such as: review content type, review relevance, review's coverage of a submission, review tone, review volume and review plagiarism (from the submission or from other reviews). We use natural language processing and machine-learning techniques to calculate these metrics. We summarize results from experiments to evaluate our review quality metrics: review content, relevance and coverage, and a study to analyze user perceptions of importance and usefulness of these metrics. Our approaches were evaluated on data from Expertiza and the Scaffolded Writing and Rewriting in the Discipline (SWoRD) project, which are two collaborative web-based learning applications.
Descriptors: Natural Language Processing, Peer Evaluation, Educational Quality, Meta Analysis, Automation, Computer Software, Feedback (Response), Authors, Writing for Publication, Plagiarism, Evaluation Methods
Springer. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: email@example.com; Web site: http://www.springerlink.com
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Sponsor: National Science Foundation (NSF)
Authoring Institution: N/A
Grant or Contract Numbers: 0942279