NotesFAQContact Us
Collection
Advanced
Search Tips
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: EJ1256850
Record Type: Journal
Publication Date: 2017-Sep
Pages: 21
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1759-2879
EISSN: N/A
An Exploration of Crowdsourcing Citation Screening for Systematic Reviews
Mortensen, Michael L.; Adam, Gaelen P.; Trikalinos, Thomas A.; Kraska, Tim; Wallace, Byron C.
Research Synthesis Methods, v8 n3 p366-386 Sep 2017
Systematic reviews are increasingly used to inform health care decisions, but are expensive to produce. We explore the use of "crowdsourcing" (distributing tasks to untrained workers via the web) to reduce the cost of screening citations. We used Amazon Mechanical Turk as our platform and 4 previously conducted systematic reviews as examples. For each citation, workers answered 4 or 5 questions that were equivalent to the eligibility criteria. We aggregated responses from multiple workers into an overall decision to include or exclude the citation using 1 of 9 algorithms and compared the performance of these algorithms to the corresponding decisions of trained experts. The most inclusive algorithm (designating a citation as relevant if "any" worker did) identified 95% to 99% of the citations that were ultimately included in the reviews while excluding 68% to 82% of irrelevant citations. Other algorithms increased the fraction of irrelevant articles excluded at some cost to the inclusion of relevant studies. Crowdworkers completed screening in 4 to 17 days, costing $460 to $2220, a cost reduction of up to 88% compared to trained experts. Crowdsourcing may represent a useful approach to reducing the cost of identifying literature for systematic reviews.
Wiley-Blackwell. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: cs-journals@wiley.com; Web site: http://www.wiley.com/WileyCDA
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Agency for Healthcare Research and Quality (AHRQ) (DHHS/PHS); National Institutes of Health (DHHS)
Authoring Institution: N/A
Grant or Contract Numbers: R03HS025024; UH2CA203711