Evaluating meta-analyses in the general surgical literature: a critical appraisal

Abstract

Objective: To assess the methodologic quality of meta-analyses of general surgery topics published in peer-reviewed journals.

Summary background data: Systematic reviews and meta-analysis are used to seek, summarize, and interpret primary studies on a given topic. Accordingly, systematic reviews and meta-analyses of high-quality primary studies may be the highest level of evidence for issues of prevention and treatment in evidence-based medicine. However, not all published meta-analyses are rigorously performed.

Methods: We searched MEDLINE (from January 1, 1997, to September 1, 2002) and reference lists and solicited general surgery specialists to identify relevant meta-analyses. Inclusion criteria were use of meta-analytic methods to pool the results of primary studies in general surgery on issues of diagnosis, causation, prognosis, or treatment. Our search strategies identified 487 potentially relevant articles. After excluding articles based on a priori criteria, 51 meta-analyses fulfilled eligibility criteria. In duplicate and independently, 2 reviewers assessed the quality of these meta-analyses using a 10-item index called the Overview Quality Assessment Questionnaire.

Results: Overall concordance between 2 independent reviewers was good (interobserver agreement 81%, and a kappa of 0.62 (95% CI 0.55-0.69). Of 51 relevant articles, 38 were published in surgical journals. Most studies had major methodologic flaws (median score of 3.3, scale of 1-7). Factors associated with low overall scientific quality included the absence of any prior meta-analyses publications by authors and meta-analyses produced by surgical department members without external collaboration.

Conclusions: This critical appraisal of meta-analyses published in the general surgery literature demonstrates frequent methodologic flaws. The quality of these reports limits the validity of the findings and the inferences that can be made about the primary studies reviewed. To improve the quality of future meta-analyses, we recommend following guidelines for the optimal conduct and reporting of meta-analyses in general surgery.

FIGURE 1. Literature search strategy.

FIGURE 2. Characteristics of meta-analysis by year of publication. White line represents mean OSQ score by year (left vertical axis). Black line represents mean impact factors by year (right vertical axis). Horizontal axis represents year of publication, with n equal to number of articles published for a given year.