Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2011;6(12):e28422.
doi: 10.1371/journal.pone.0028422. Epub 2011 Dec 2.

Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews

Georgina Imberger  1 , Alexandra Damgaard Vejlby, Sara Bohnstedt Hansen, Ann M Møller, Jørn Wetterslev
Affiliations

Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews

Georgina Imberger et al. PLoS One. 2011.

Abstract

Background: Systematic reviews with meta-analyses often contain many statistical tests. This multiplicity may increase the risk of type I error. Few attempts have been made to address the problem of statistical multiplicity in systematic reviews. Before the implications are properly considered, the size of the issue deserves clarification. Because of the emphasis on bias evaluation and because of the editorial processes involved, Cochrane reviews may contain more multiplicity than their non-Cochrane counterparts. This study measured the quantity of statistical multiplicity present in a population of systematic reviews and aimed to assess whether this quantity is different in Cochrane and non-Cochrane reviews.

Methods/principal findings: We selected all the systematic reviews published by the Cochrane Anaesthesia Review Group containing a meta-analysis and matched them with comparable non-Cochrane reviews. We counted the number of statistical tests done in each systematic review. The median number of tests overall was 10 (interquartile range (IQR) 6 to 18). The median was 12 in Cochrane and 8 in non-Cochrane reviews (difference in medians 4 (95% confidence interval (CI) 2.0-19.0). The proportion that used an assessment of risk of bias as a reason for doing extra analyses was 42% in Cochrane and 28% in non-Cochrane reviews (difference in proportions 14% (95% CI -8 to 36). The issue of multiplicity was addressed in 6% of all the reviews.

Conclusion/significance: Statistical multiplicity in systematic reviews requires attention. We found more multiplicity in Cochrane reviews than in non-Cochrane reviews. Many of the reasons for the increase in multiplicity may well represent improved methodological approaches and greater transparency, but multiplicity may also cause an increased risk of spurious conclusions. Few systematic reviews, whether Cochrane or non-Cochrane, address the issue of multiplicity.

PubMed Disclaimer

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

Figures

Figure 1
Figure 1. Selection of non-Cochrane reviews.
Figure 2
Figure 2. Box plot showing the median, interquartile range and range of the distribution of the number of statistical tests in our population of systematic reviews (excluding Carlisle 2006).
Figure 3
Figure 3. Box plot showing the median, interquartile range and range of the distribution of the number of statistical tests in the Cochrane and non-Cochrane reviews (excluding the Carlisle 2006).
See this image and copyright information in PMC

References

    1. Oxman AD, Guyatt GH. The science of reviewing research. Ann N Y Acad Sci. 1993;703:125–133. - PubMed
    1. Petty WC, Kremer M, Biddle C. A synthesis of the Australian Patient Safety Foundation Anesthesia Incident Monitoring Study, the American Society of Anesthesiologists Closed Claims Project, and the American Association of Nurse Anesthetists Closed Claims Study. AANA J. 2002;70:193–202. - PubMed
    1. Wetterslev J, Møller AM, Lundstrøm LH, Jensen PF. 2009. Danish Anaesthesia Database Annual Report 2008.
    1. Sharp SJ. Analysing the relationship between treatment benefit and underlying risk: precaution and recommendations. In: Egger M, Smith GD, Altman DG, editors. Systematic Reviews in Health Care. London: BMJ Publishing Group; 2006.
    1. Sterne JA, Egger M, Smith GD. Investigating and dealing with publication and other biases. In: Egger M, Smith GD, Altman DG, editors. Systematic Reviews in Health Care. London: BMJ Publishing Group; 2006. - PMC - PubMed

Publication types