NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ1419734
Record Type: Journal
Publication Date: 2024
Pages: 32
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: EISSN-1946-6226
Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment
Mike Richards; Kevin Waugh; Mark A Slaymaker; Marian Petre; John Woodthorpe; Daniel Gooch
ACM Transactions on Computing Education, v24 n1 Article 5 2024
Cheating has been a long-standing issue in university assessments. However, the release of ChatGPT and other free-to-use generative AI tools has provided a new and distinct method for cheating. Students can run many assessment questions through the tool and generate a superficially compelling answer, which may or may not be accurate. We ran a dual-anonymous "quality assurance" marking exercise across four end-of-module assessments across a distance university computer science (CS) curriculum. Each marker received five ChatGPT-generated scripts alongside 10 student scripts. A total of 90 scripts were marked; every ChatGPT generated script for the undergraduate modules received at least a passing grade (>40%), with all of the introductory module CS1 scripts receiving a distinction (>85%). None of the ChatGPT-taught postgraduate scripts received a passing grade (>50%). We also present the results of interviewing the markers and of running our sample scripts through a GPT-2 detector and the TurnItIn AI detector, which both identified every ChatGPT-generated script but differed in the number of false positives. As such, we contribute a baseline understanding of how the public release of generative AI is likely to significantly impact quality assurance processes. Our analysis demonstrates that in most cases, across a range of question formats, topics, and study levels, ChatGPT is at least capable of producing adequate answers for undergraduate assessment.
Association for Computing Machinery. 2 Penn Plaza Suite 701, New York, NY 10121. Tel: 800-342-6626; Tel: 212-626-0500; Fax: 212-944-1318; e-mail: acmhelp@acm.org; Web site: http://toce.acm.org/
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: United Kingdom
Grant or Contract Numbers: N/A