NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: EJ1122833
Record Type: Journal
Publication Date: 2015
Pages: 13
Abstractor: As Provided
ISSN: ISSN-1042-0541
Peer Review in Agricultural Education: Interrater Reliability of Manuscript Reviews for the 2014 National Agricultural Education Research Conference
Shoulders, Catherine W.; Johnson, Donald M.; Flowers, Jim
Journal of Agricultural Education, v56 n3 p121-133 2015
This study analyzed 336 peer reviews of 112 manuscripts submitted for possible presentation at the 2014 National Agricultural Education Research Conference (NAERC). There were scoring errors on 6.8% of the reviews; the most frequent errors were failure to record a score or assigning a score above the range of points possible for one or more of the review criteria. The coefficient of variation (CV) for same-paper ratings of manuscript quality ranged from 3.94% to 96.43% with a mean of 19.65% (SD = 15.22%). The interrater reliability for same-paper evaluations of manuscript quality was 0.15. The CVs for reviewer same-paper reject-accept recommendations ranged from 0.00% to 98.97% with a mean of 38.80% (SD = 22.72%). The interrater reliability for same-paper reject-accept recommendations was .09. On 82 (73.21%) manuscripts all three reviewers agreed on the manuscript's relevance to agricultural education. Mean manuscript quality scores explained 68.3% of the variance in mean reject-accept recommendations leaving 31.7% of the variance unexplained. There were no significant (p > 0.05) differences by academic rank in ratings of manuscript quality, reject-accept recommendations, or assessments of relevance to agricultural education. The authors offered recommendations for improvement of the NAERC manuscript review process.
American Association for Agricultural Education. P.O. Box 7607, Department of Agricultural and Extension Education, Raleigh, NC 27695. Web site:
Publication Type: Journal Articles; Reports - Research; Information Analyses
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A