An exploration of crowdsourcing citation screening for systematic reviews.

Res Synth Methods 2017 Sep 4;8(3):366-386. Epub 2017 Jul 4.

College of Computer and Information Science, Northeastern University, Boston, MA, USA.

Systematic reviews are increasingly used to inform health care decisions, but are expensive to produce. We explore the use of crowdsourcing (distributing tasks to untrained workers via the web) to reduce the cost of screening citations. We used Amazon Mechanical Turk as our platform and 4 previously conducted systematic reviews as examples. For each citation, workers answered 4 or 5 questions that were equivalent to the eligibility criteria. We aggregated responses from multiple workers into an overall decision to include or exclude the citation using 1 of 9 algorithms and compared the performance of these algorithms to the corresponding decisions of trained experts. The most inclusive algorithm (designating a citation as relevant if any worker did) identified 95% to 99% of the citations that were ultimately included in the reviews while excluding 68% to 82% of irrelevant citations. Other algorithms increased the fraction of irrelevant articles excluded at some cost to the inclusion of relevant studies. Crowdworkers completed screening in 4 to 17 days, costing $460 to $2220, a cost reduction of up to 88% compared to trained experts. Crowdsourcing may represent a useful approach to reducing the cost of identifying literature for systematic reviews.

Download full-text PDF

Source
http://dx.doi.org/10.1002/jrsm.1252DOI Listing
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5589498PMC
September 2017
42 Reads

Publication Analysis

Top Keywords

systematic reviews
16
trained experts
8
reviews
5
citation algorithms
4
exclude citation
4
completed screening
4
decision include
4
include exclude
4
algorithms compared
4
corresponding decisions
4
decisions trained
4
algorithms corresponding
4
performance algorithms
4
crowdworkers completed
4
compared performance
4
workers decision
4
responses multiple
4
$460 $2220
4
increased fraction
4
answered questions
4

Similar Publications