ERIC Number: EJ1011669
Record Type: Journal
Publication Date: 2013-Jan
Pages: 28
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1093-023X
EISSN: N/A
Web-Based Portfolio Assessment: Validation of an Open Source Platform
Collins, Regina; Elliot, Norbert; Klobucar, Andrew; Deek, Fadi P.
Journal of Interactive Learning Research, v24 n1 p5-32 Jan 2013
Assessment of educational outcomes through purchased tests is commonplace in the evaluation of individual student ability and of educational programs. Focusing on the assessment of writing performance in a longitudinal study of first-time, full-time students (n = 598), this research describes the design, use, and assessment of an open-source scoring platform. Augmenting usability testing, the research design relies on a framework of inter-reader agreement, inter-reader reliability, and coefficients of determination. The open-source, web-based portfolio assessment system yielded rates of agreement, reliability, and determination superior to the traditional paper-based portfolio assessment method. In addition, the system appears to be ideally suited to assess EPortfolios created to showcase student ability in digital environments: agreement range = 70% to 85%; reliability range = "kappa" = 0.67 ("p" less than 0.01) to "kappa" = 0.85 ("p" less than 0.01); coefficient of determination = "R[superscript 2]" = 0.95, "F"(5, 34) = 118.59 ("p" less than 0.01). This novel and innovative application of an open source platform for outcomes assessment yields the foundation for a sound validity argument, the control of human error, and complete system transparency and flexibility. Future research directions point to the need for the design and assessment of an open-source system designed to capture complex constructs as they emerge in digital environments. (Contains 3 tables.)
Descriptors: Portfolio Assessment, Open Source Technology, Longitudinal Studies, Writing Evaluation, Student Evaluation, Interrater Reliability, Scoring, College Students, Case Studies, Educational Technology, Writing Skills, Computer Uses in Education, Reliability, Competence
Association for the Advancement of Computing in Education. P.O. Box 1545, Chesapeake, VA 23327. Tel: 757-366-5606; Fax: 703-997-8760; e-mail: info@aace.org; Web site: http://www.aace.org
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A