Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis.

PLoS One 2019 12;14(4):e0215052. Epub 2019 Apr 12.

Department of Methodology and Statistics, Tilburg University, Tilburg, the Netherlands.

Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias exists, how strongly it affects different scientific literatures is currently less well-known. We examined evidence of publication bias in a large-scale data set of primary studies that were included in 83 meta-analyses published in Psychological Bulletin (representing meta-analyses from psychology) and 499 systematic reviews from the Cochrane Database of Systematic Reviews (CDSR; representing meta-analyses from medicine). Publication bias was assessed on all homogeneous subsets (3.8% of all subsets of meta-analyses published in Psychological Bulletin) of primary studies included in meta-analyses, because publication bias methods do not have good statistical properties if the true effect size is heterogeneous. Publication bias tests did not reveal evidence for bias in the homogeneous subsets. Overestimation was minimal but statistically significant, providing evidence of publication bias that appeared to be similar in both fields. However, a Monte-Carlo simulation study revealed that the creation of homogeneous subsets resulted in challenging conditions for publication bias methods since the number of effect sizes in a subset was rather small (median number of effect sizes equaled 6). Our findings are in line with, in its most extreme case, publication bias ranging from no bias until only 5% statistically nonsignificant effect sizes being published. These and other findings, in combination with the small percentages of statistically significant primary effect sizes (28.9% and 18.9% for subsets published in Psychological Bulletin and CDSR), led to the conclusion that evidence for publication bias in the studied homogeneous subsets is weak, but suggestive of mild publication bias in both psychology and medicine.

Download full-text PDF

Source
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0215052PLOS
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC6461282PMC
April 2019
3 Reads

Article Mentions


Provided by Crossref Event Data
twitter
Twitter:
April 18, 2019, 10:03 am EST
twitter
Twitter:
April 18, 2019, 9:59 am EST
twitter
Twitter:
April 18, 2019, 8:51 am EST
twitter
Twitter:
April 18, 2019, 8:15 am EST
twitter
Twitter:
April 17, 2019, 1:48 pm EST
twitter
Twitter:
April 17, 2019, 1:47 pm EST
twitter
Twitter:
April 17, 2019, 1:46 pm EST
twitter
Twitter:
April 17, 2019, 1:43 pm EST
twitter
Twitter:
April 17, 2019, 1:42 pm EST
twitter
Twitter:
April 16, 2019, 6:59 pm EST
twitter
Twitter:
April 16, 2019, 11:36 am EST
twitter
Twitter:
April 16, 2019, 11:35 am EST
twitter
Twitter:
April 16, 2019, 10:26 am EST
twitter
Twitter:
April 16, 2019, 8:01 am EST
twitter
Twitter:
April 16, 2019, 7:59 am EST
twitter
Twitter:
April 16, 2019, 5:39 am EST
twitter
Twitter:
April 16, 2019, 3:33 am EST
twitter
Twitter:
April 16, 2019, 3:16 am EST
twitter
Twitter:
April 16, 2019, 3:11 am EST
twitter
Twitter:
April 16, 2019, 2:14 am EST
twitter
Twitter:
April 16, 2019, 12:46 am EST
twitter
Twitter:
April 16, 2019, 12:44 am EST
twitter
Twitter:
April 16, 2019, 12:41 am EST
twitter
Twitter:
April 16, 2019, 12:38 am EST
twitter
Twitter:
April 15, 2019, 8:32 pm EST
twitter
Twitter:
April 15, 2019, 1:19 pm EST
twitter
Twitter:
April 15, 2019, 12:34 pm EST
twitter
Twitter:
April 15, 2019, 12:04 pm EST
twitter
Twitter:
April 15, 2019, 11:12 am EST
twitter
Twitter:
April 15, 2019, 9:13 am EST
twitter
Twitter:
April 15, 2019, 7:07 am EST
twitter
Twitter:
April 15, 2019, 6:57 am EST
twitter
Twitter:
April 15, 2019, 6:38 am EST
twitter
Twitter:
April 15, 2019, 5:31 am EST
twitter
Twitter:
April 15, 2019, 5:24 am EST
twitter
Twitter:
April 15, 2019, 5:05 am EST
twitter
Twitter:
April 15, 2019, 4:25 am EST
twitter
Twitter:
April 15, 2019, 4:02 am EST
twitter
Twitter:
April 15, 2019, 3:57 am EST
twitter
Twitter:
April 15, 2019, 3:42 am EST
twitter
Twitter:
April 15, 2019, 3:41 am EST
twitter
Twitter:
April 15, 2019, 3:41 am EST
twitter
Twitter:
April 15, 2019, 3:40 am EST
twitter
Twitter:
April 15, 2019, 3:33 am EST
twitter
Twitter:
April 15, 2019, 3:31 am EST
twitter
Twitter:
April 15, 2019, 3:27 am EST
twitter
Twitter:
April 15, 2019, 3:27 am EST
twitter
Twitter:
April 15, 2019, 3:26 am EST
twitter
Twitter:
April 15, 2019, 3:24 am EST
twitter
Twitter:
April 15, 2019, 3:20 am EST

Publication Analysis

Top Keywords

publication bias
48
homogeneous subsets
16
bias
13
psychological bulletin
12
publication
12
published psychological
12
evidence publication
12
meta-analyses published
8
representing meta-analyses
8
bias methods
8
primary studies
8
meta-analyses psychology
8
studies included
8
included meta-analyses
8
systematic reviews
8
number sizes
8
psychology medicine
8
meta-analyses
7
subsets
6
statistically primary
4

Altmetric Statistics

References

(Supplied by CrossRef)
Primary, secondary, and meta-analysis of research
GV Glass et al.
Educational Researcher 1976
Estimating effect size: Bias resulting from the significance criterion in editorial decisions.
DM Lane et al.
British Journal of Mathematical & Statistical Psychology 1978
Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa
TD Sterling et al.
The American Statistician 1995
The rules of the game called psychological science
M Bakker et al.
Perspectives on Psychological Science 2012
Things I have learned (so far).
J. Cohen et al.
American Psychologist 1990

Similar Publications