NotesFAQContact Us
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ1109041
Record Type: Journal
Publication Date: 2014-Sep
Pages: 12
Abstractor: As Provided
ISSN: ISSN-1759-2879
Practicalities of Using a Modified Version of the Cochrane Collaboration Risk of Bias Tool for Randomised and Non-Randomised Study Designs Applied in a Health Technology Assessment Setting
Robertson, Clare; Ramsay, Craig; Gurung, Tara; Mowatt, Graham; Pickard, Robert; Sharma, Pawana
Research Synthesis Methods, v5 n3 p200-211 Sep 2014
We describe our experience of using a modified version of the Cochrane risk of bias (RoB) tool for randomised and non-randomised comparative studies. Objectives: (1) To assess time to complete RoB assessment; (2) To assess inter-rater agreement; and (3) To explore the association between RoB and treatment effect size. Methods: Cochrane risk of bias assessment was performed on a sample of full text primary reports included in a systematic review comparing operative techniques for radical prostatectomy. Inter-rater agreement was assessed using the kappa statistic. Results: Twenty-four studies were judged as high overall RoB, 13 were judged as low RoB and 11 were unclear. The weighted Kappa value was 0.35 indicating fair agreement. The median (range) time taken to rate each study was 30 min (10-49). The effect estimate for all studies was 0.61 (95% credible interval (CrI) 0.46-0.83) and 0.73 (95% CrI 0.29-1.75) for low risk studies. Conclusions: Although the process was time consuming, using a modified version of the RoB tool proved useful for demonstrating conservative effect estimates. That we only achieved a fair agreement between reviewers demonstrates the urgent need for further validation to improve inter-rater agreement. We suggest additional RoB levels could improve inter-rater reliability. [This article was written with the assistance of the UK Robotic Laparoscopic Prostatectomy HTA Study Group.]
Wiley-Blackwell. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail:; Web site:
Publication Type: Journal Articles; Reports - Research; Information Analyses
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A